In-Service Performance Evaluation: Guidelines for the Assembly and Analysis of Data (2022)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

44 The preparation of a comprehensive ISPE report is of paramount importance to successful completion of the ISPE, communicating your results, and collaborating with other jurisdictions to facilitate improved roadside safety. The ISPE report should be prepared as a formal tech- nical report. This chapter documents the minimum requirements; a jurisdiction may include additional details at its own discretion. There are always uncertainties within the data available for evaluating the in-service performance of a safety feature; however, these uncertainties are insufficient reason not to make the best possible evaluation of the in-service performance of that safety feature. Communicate any known uncertainties in the data and the steps to provide the detail necessary to allow your colleagues to verify your findings. While few studies will be veri- fied, the act of providing the necessary details inspires confidence in your findings. ISPE studies can be both verified and validated. Verification occurs when a different analyst takes the same data, performs the same analysis, and replicates the results. The second analyst may verify that the first analyst’s evaluations were done correctly and in accordance with the procedures described in this report by replicating the analysis. A study is validated when another analyst using another set of data produces essentially the same results. If two ISPE studies in two different states determine that the ES and point estimates for a particular evaluation mea- sure are essentially the same, the ES and point estimates can be said to be validated. One of the primary purposes of documenting ISPEs and making the results available on the ISPE Resource Hub when it becomes available is to provide opportunities to validate results by using many different highway agencies. 7.1 General Reporting Recommendations The following elements should be included in the front matter of each ISPE report: • Title page, • Technical Report Documentation page, • Disclosure statement, • Acknowledgments (if applicable), • Table of contents, • List of figures, and • List of tables. The title page should include the title of the report, the names(s) and affiliations(s) of the authors and analyst, the name and address of the analyst’s organization, the sponsoring agency, and the date of the report. The federal Technical Report Documentation Page should be com- pleted and included as part of the report whenever federal funds were used. A disclosure state- ment should be provided that documents sources of financial or organizational conflict of C H A P T E R 7 ISPE Documentation

ISPE Documentation 45   interest (e.g., ownership of the analyst’s organization, other organizational relationships, pro- prietary rights and interests) and explains how each was accounted for in the ISPE. If there are no issues related to objectivity, this should be stated. The acknowledgments section is optional. 7.2 ISPE Report Contents The ISPE report should include, at a minimum, the following chapters or sections: • Introduction. The introductory chapter should include the following: – Identification of the SFUE. – A brief (one sentence) description summarizing the study area and data collection period for quick reference—for example, “All data were collected on state-maintained roads in the State of Maine from 2009 through 2013.” • System Details. Standard drawings, construction specifications, and any other appropriate means that the highway agency uses to identify roadside hardware systems (e.g., inventory identification manual, state DOT QPL/APL) should be used to describe the SFUE. When the NAME field is included and multiple standard drawings and/or specifications are avail- able, these should all be included. When available, summary sheets from crash test report(s) should be provided as an appendix to the report. The test level and a reference to the crash test report(s) should also be provided. When the crash test report summary sheet and original crash test report are not available for reference, this section should include a statement to indicate such. • Study Period and Region. A statement should be made as to whether the ISPE report rep- resents a routine or investigative ISPE. For both routine and investigative ISPEs, the specific start and end dates for data collection should be indicated, and a text description of the study area should be provided. The description of the study area is often accompanied by a map. Neither the study period nor the study area should change mid-ISPE. Every effort has been made to maintain flexibility in the application of the guidelines presented in this report, so as to allow for application by the greatest number of jurisdictions. Authors are reminded to be mindful of how the safety features being evaluated are used in their jurisdiction and to convey that information to readers. • Available Data. The assemblage and processing of the study sample are documented in this chapter. The origin of the crash database, electronic or otherwise, should be provided here. The origin of the electronic inventory, paper copies of maintenance and construction records, and other documents used to assemble the study sample should be noted. The materials, including sources, used to identify the SFUE and values of NAME should be documented. If manual identification or manual processes were employed, documentation that describes these manual methods should be provided. This chapter should include the ISPE Data Set Assemblage and Case Reduction Sheet (see Section 7.4.1). • Evaluation Measures. This chapter summarizes which performance outcomes were evalu- ated (i.e., structural adequacy, occupant risk, vehicle trajectory and orientation) and which evaluation measures were used (i.e., A, B, C, D, F, H, J, K, L, M). A summary of which condi- tions were met for each evaluation measure considered should be provided. The jurisdiction’s threshold values should be given, as well as any deviation of the recommended confidence value. A summary of the results by performance assessment level should be provided for each evaluation measure considered. • Summary and Conclusions. Conclusions, including interpretation of the results and sug- gested implementation of the results, are addressed in this chapter. This chapter presents a conclusion about the crashworthiness of the SFUE and its continued use and any conclu- sions about the need for improvements. • References. A list of all references used or cited in the report should be provided. This will include the documentation for the installation and maintenance of the safety features studied,

46 In-Service Performance Evaluation: Guidelines for the Assembly and Analysis of Data as well as information regarding the crash test documentation for the safety feature studied. Other references used should also be included here. • Appendices. Appendix A contains the Crash Test Report Summary Sheet for each system included in the ISPE. If the Crash Test Report Summary Sheet is not available, a statement to that fact should be made. Appendix B contains the ISPE Summary of Data and Results Sheets (see Section 7.4.3) presenting each evaluation measure and system considered. The ISPE Data Set and Analysis Template generates these sheets automatically. Additional appendices are optional and should contain information the authors wish to keep with the ISPE report but do not wish to include within the body of the report. 7.3 Meta-Analysis of ISPE Reports A meta-analysis of ISPE reports should include, as a minimum, the following chapters or sections: • Introduction. The introductory chapter should include the following: – Identification of the SFUE. – Discussion of the performance outcomes and evaluation measures: The performance out- comes evaluated (i.e., structural adequacy, occupant risk, vehicle trajectory and orienta- tion) and the measures used to evaluate them (i.e., A, B, C, D, F, H, J, K, L, M) should be summarized. – ISPE Reports included: The ISPE reports that form the basis of the meta-analysis should be documented and described. – ISPE Reports excluded: The ISPE reports that were judged to be relevant but were not included in the meta-analysis should also be documented with explicit statements explain- ing why the reports were excluded. • Summary of ISPEs Included. This chapter should provide the ISPE Summary of Results Sheets from each component ISPE for each evaluation measure considered in the meta-analysis. Include on each Summary of Results Sheet a footnote identifying the original source. A con- cise description of how the literature search was performed and a description of each compo- nent ISPE should be included. • Evaluation Measures. An individual tabulation of the ISPE results used in the meta-analysis for each evaluation measure for each performance assessment level should be provided in this chapter. • Summary of Results and Conclusions. This chapter of the meta-analysis should include tabulations of the mathematical computations discussed in Chapter 6 of the present report. Conclusions justifying the crashworthiness of the safety feature evaluated and its continued use and any conclusions about the need for improvements should be noted in this chapter. The threshold values used should be provided as well as any deviation of the recommended CI value. • References. A list of all references used or cited should be provided. • Appendices. Appendices are optional and should contain information the authors wish to keep with the ISPE report but do not wish to include within the body of the report. 7.4 Presentation of Data and Findings 7.4.1 ISPE Data Set Case Reduction Sheet The means and methods used to assemble the data set from the original data through each case reduction or filter should be documented to allow for validation of the ISPE data set. The order of case reduction and the number of cases reduced should be captured and documented.

ISPE Documentation 47   Figure 2 provides the ISPE Data Set Assemblage and Case Reduction Sheet, which is the recom- mended format for documenting the assembly of the data set and the case reduction process. Figure 3 is a completed example of the same. 7.4.2 Preparation of ISPE Data Set The preparation of the ISPE data set includes assembling data elements from each of the various data sources. Equivalences may be needed between jurisdiction codes and codes used in the ISPE data set. Section 3.2 notes that these equivalences should be documented to promote con- sistent interpretation. Tables 10 through 17 provide templates and examples of equivalence tables in which the standardized definitions of the variables in the ISPE data set are mapped to jurisdictional values. Table 10 provides the recommended format for the MAX_SEV equiva- lence, and Table 11 is a completed example of the same. The values on the right side of Table 11, for example, would likely come from a particular jurisdiction’s police crash report. Table 12 provides the recommended format for the VEH_TYPE equivalence table, and Table 13 is an Crash Information Data Reduction Year Cases Intent/Codes Removed Data Years Cases Remaining 1 2 Safety Feature Identification Method(s) KEY ISPE Data Set Data Reduction Step Data Files At discretion of Analyst 2* * 1* Figure 2. ISPE Data Set Assemblage and Case Reduction Sheet.

48 In-Service Performance Evaluation: Guidelines for the Assembly and Analysis of Data Crash Information Data Reduction Year Cases Intent/Codes Removed Data Years Cases Remaining 2010 215,813 1 Identified Cases with Safety Feature(s): 2010 11,117 2011 221,483 2011 12,029 2012 220,190 2012 11,480 2013 218,794 Retain cases having values of 25 and 26 in harmful events field 1- 3. 2013 11,854 2014 214,886 2014 11,854 --- --- --- --- 2 Identify cases within study area 2010 9,557 Safety Feature Identification Method(s) 2011 10,405 2012 9,704 Retain cases occurring in XDOT Maintenance Districts 1, 2 and 3. 2013 10,149 2014 10,223 Asset Inventory --- --- KEY ISPE Data Set CRASH VEHICLE PERSON ROADWAY* 2 1 Data Reduction Step Data Files At discretion of Analyst* Figure 3. Example of completed ISPE Data Set Assemblage and Case Reduction Sheet. MAX_SEV Jurisdictional Equivalence K A B C O U Table 10. Recommended format for equivalence table for ISPE data set MAX_SEV.

ISPE Documentation 49   MAX_SEV Jurisdictional Equivalence K 1 – Killed A 2 – Major injury B 3 – Moderate injury C 4 – Minor injury O 0 – Not injured U 8 – Injury/unknown severity 9 – Unknown Table 11. Example of recommended equivalence table for ISPE data set MAX_SEV. VEH_TYPE Jurisdictional Equivalence MC PC PU SUT BUS TT OTR 99 Table 12. Recommended format for equivalence table for ISPE data set VEH_TYPE. VEH_TYPE Jurisdictional Equivalence MC 02 – Motorcycle PC 01 – Automobile PU 06 – SUV 04 – Small Truck 07 – Van SUT 12 – Construction Equipment 11 – Farm Equipment BUS 03 – Bus TT 05 – Large Truck OTR 13 – ATV 10 – Snowmobile 21 – Other Pedalcycle 22 – Horse and Buggy 23 – Horse and Rider 24 – Train 18 – Other Special Vehicle 20 – Unicycle; Bicycle or Tricycle 99 19 – Unknown Type of Special Vehicle Table 13. Example of recommended equivalence table for ISPE data set VEH_TYPE.

50 In-Service Performance Evaluation: Guidelines for the Assembly and Analysis of Data example of the completed equivalence. Table 14 is the recommended format for documenting the PostHE equivalence, and Table 15 provides an example of the documentation of the PostHE equivalence. Equivalence tables such as these should be provided when such decisions are made. These tables convey the information necessary for the data set to be validated. The NAME field in Table 16 uses lowercase variables in the left column to represent any number of subgroups of a safety feature. The equivalence of each lowercase letter to a particular safety feature should be documented in the right-hand column of Table 16, as illustrated in Table 17 for an example longitudinal barrier. The source material for each field within the ISPE data set should also be documented. Many times, this documentation will be as simple as referencing a previously constructed data docu- mentation table. Table 18 provides the recommended format and Table 19 a completed example. 7.4.3 ISPE Summary of Data and Results Sheets It is recommended that each ISPE Report include an ISPE Summary of Data and Results Sheet for the applicable evaluation measures. This will facilitate interpretation of the results and collab- oration between jurisdictions. Examples of ISPE Summary of Data and Results Sheets are shown in Appendix A of this report. An example is provided for each evaluation measure; however, a typical ISPE includes only a limited number of evaluation measures. If the study considered values of NAME, then see the examples in Appendix B. 7.5 Electronic Data Requirements for assembling and analyzing the ISPE data set are provided earlier in this report. These requirements are intended to facilitate standardization and collaboration between jurisdictions. The ISPE Data Set and Analysis Template for conducting the analysis is available for downloading from the National Academies Press website (www.nap.edu) by searching for NCHRP Research Report 1010: In-Service Performance Evaluation: Guidelines for the Assembly and Analysis of Data. Submitting the assembled ISPE Data Set and Analysis Template to the ISPE Resource Hub when the Hub becomes available is not a requirement but is suggested. It is planned that the ISPE Resource Hub will serve as a repository for ISPEs of safety features that can be readily considered by the highway safety community. PostHE Jurisdictional Equivalence 00 99 RFS RSS ROLL TER VEH PED FO BA BAR TCC OTR Table 14. Recommended format for equivalence table for ISPE data set PostHE.

ISPE Documentation 51   PostHE Jurisdictional Equivalence 00 No harmful event following impact with the safety feature listed in crash sequence. 99 (99) – Unknown what was hit RFS (50) – Overturn or rollover and review of police investigation sketch RSS (50) – Overturn or rollover and review of police investigation sketch ROLL (50) – Overturn or rollover TER (28) – Impact with ditch VEH (01) – Hit Unit 01 (02) – Hit Unit 02 (03) – Hit Unit 03 (04) – Hit Unit 04 (05) – Hit Unit 05 (06) – Hit other traffic unit (11) – Struck by Unit 01 (12) – Struck by Unit 02 (13) – Struck by Unit 03 (14) – Struck by Unit 04 (15) – Struck by Unit 05 (16) – Struck by other traffic unit [If any of Unit 01–05 is listed as a motor vehicle in transit.] PED (01) – Hit Unit 01 (02) – Hit Unit 02 (03) – Hit Unit 03 (04) – Hit Unit 04 (05) – Hit Unit 05 (06) – Hit other traffic unit [If any of Unit 01–05 is listed as pedestrian/cyclist.] FO (21) – Hit tree or shrubbery (23) – Hit utility pole (24) – Hit traffic sign (30) – Hit fence or wall (31) – Hit building (32) – Hit culvert (33) – Hit bridge pier or abutment (34) – Hit parapet end (40) – Hit mailbox (48) – Hit other fixed object (49) – Hit unknown fixed object BA Review of police investigation sketch BAR (25) – Hit guard or guide rail (26) – Hit guard or guide rail end (28) – Hit concrete or longitudinal barrier (35) – Hit bridge rail (37) – Hit impact attenuator or crash cushion (43) – Hit temporary construction barrier TCC (33) – Guardrail end terminal; impact attenuator/crash cushion OTR (03) – Fire/explosion (04) – Immersion, full or partial Table 15. Example of recommended equivalence table for ISPE data set PostHE.

52 In-Service Performance Evaluation: Guidelines for the Assembly and Analysis of Data NAME Jurisdiction’s Description a b c d e f g h i j k l m n p q r s t u v w x y z 99 Note: The letter o is not used in the NAME field. Table 16. Recommended format for equivalence table for ISPE data set NAME. NAME Jurisdiction’s Description* a Strong Post Cable b Weak Post Cable c Strong Post W-Beam with Rub Rail and Offset Bracket d Strong Post W-Beam with Offset Bracket e Strong Post W-Beam f Weak Post W-Beam g Strong Post W-Beam, Double Faced h Weak Post W-Beam, Double Faced i Weak Post Box Beam j Concrete Safety Shape k IBC Barrier l Cable Safety System (CASS) m Wire Rope Safety Fence (WRSF) n Safence Cable Barrier p Gibraltar Cable Barrier q Nu-Cable Barrier r Concrete Safety Shape s Concrete Safety Shape t Concrete Safety Shape u Not used v Not used w Not used x Not used y Not used z Other 99 Unknown Note: The letter o is not used in the NAME field. *As hardware appears on the jurisdiction’s QPL/APL. Table 17. Example of recommended equivalence table for ISPE data set NAME for longitudinal barrier.

ISPE Documentation 53   Column Field Name Definition Source A SFUE Safety feature under evaluation B CRN Crash number C CRASH_DATE Date of crash D TOTAL_UNITS Number of units involved in crash E MAX_SEV Maximum severity of injuries to occupants of impacting vehicle F VEH_TYPE Body type of vehicle G SPEED_LIMIT Posted speed limit H PostHE First harmful event after impact with safety feature I MHE Impact with safety feature was most harmful event J FHE Impact with safety feature was first harmful event K AHE Impact with safety feature occurred anywhere in crash sequence of events L FOHE Impact with safety feature was first and only harmful event M BREACH Vehicle breached safety feature N BREAK Predictable breakaway of safety feature O PRS Controlled penetration, redirection, or stop P PEN Roadside hardware penetrated into occupant compartment Q ICP Initial point on impacting vehicle at which it made contact with safety feature R NAME Subgroups of safety feature S INSTALL Safety feature inspected at installation T MAINT Safety feature inspected on a routine schedule Table 18. Recommended format for ISPE data set and source material.

54 In-Service Performance Evaluation: Guidelines for the Assembly and Analysis of Data Column Field Name Definition Source A SFUE Safety feature under evaluation Crash data Harmful Event Fields 1–4, Field Codes 25 and 26. B CRN Crash number Crash data, Field Code CRN. C CRASH_DATE Date of crash Crash data, Field Code DATE. D TOTAL_UNITS Number of units involve in crash Crash data, Field Code UNITS. E MAX_SEV Maximum severity of injuries to occupants of impacting vehicle See Table 11. F VEH_TYPE Body type of vehicle See Table 13. G SPEED_LIMIT Posted speed limit Highway Inventory, Field Code SPEED_LIMIT. H PostHE First harmful event after impact with safety feature See Table 15. I MHE Impact with safety feature was most harmful event Crash data Field Code MHE_CD_UNIT. J FHE Impact with safety feature was first harmful event Crash data Field Code FHE_UNIT_NUM. K AHE Impact with safety feature occurred anywhere in crash sequence of events Crash data Field Code for all harmful events within the sequence. L FOHE Impact with safety feature was first and only harmful event As documents in Section 3.3.12 of this report. M BREACH Vehicle breached safety feature Review of scene diagrams. N BREAK Predictable breakaway of safety feature Not applicable. O PRS Controlled penetration, redirection, or stop Not applicable. P PEN Roadside hardware penetrated into occupant compartment This information was not available. An investigative ISPE was not undertaken at this time. Q ICP Initial point on impacting vehicle at which it made contact with safety feature Crash data Field Code IMPACT_POINT matches Figure 1 of this NCHRP Research Report. R NAME Subgroupings of safety feature Equivalence shown in Table 17. S INSTALL Safety feature inspected at installation DOT inspects hardware as it is installed. T MAINT Safety feature inspected on a routine schedule DOT has a maintenance inspection program. Table 19. Example of ISPE data set and source material.