Crash Modification Factors in the Highway Safety Manual: Resources for Evaluation (2023)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Appendix A Review of Procedures for Assessing CMF Quality A-1 

Table of Contents 1. Introduction ............................................................................................................................. A-3 2. Part I: CMF Quality ................................................................................................................ A-4 2.1 CMF Quality and Study Quality ....................................................................................... A-4 2.2 Overview of Procedures for Assessing CMF Quality....................................................... A-5 2.3 Factors Used to Assess CMF Quality ............................................................................. A-14 2.4 Additional Factors to Assess CMFs from Results of Multiple Studies .......................... A-19 2.5 Defining Acceptable Quality Level ................................................................................ A-22 2.6 Summary of Findings Regarding Factors Used to Describe CMF Quality and Study Quality................................................................................................................................... A-23 3. Part II: Corrections to Improve CMF Quality....................................................................... A-25 3.1 Introduction ..................................................................................................................... A-25 3.2 Background ..................................................................................................................... A-25 3.3 Correcting CMFs for Regression to the Mean ................................................................ A-25 3.4 Correcting CMFs for Change in Traffic Volume ........................................................... A-26 3.5 Correcting CMF Standard Error ..................................................................................... A-27 3.6 Summary of Findings ...................................................................................................... A-28 4. Part III: Tasks 1 and 2 Findings and Recommended Research for Improving the Quality of CMFs in the HSM and the Clearinghouse .......................................................... A-28 4.1 Findings Specific to the HSM Procedure........................................................................ A-28 4.2 Findings Specific to the CMF Clearinghouse Procedure ................................................ A-31 4.3 Recommended Research ................................................................................................. A-32 5. References ............................................................................................................................. A-34 Appendix A.1: Estimating Safety Effect of Using Results from Multiple Studies.................. A-36 A-2 

1. INTRODUCTION Highway safety practitioners were given a significant new tool in 2010 with the publication of the 1st edition of the AASHTO Highway Safety Manual (HSM). In Part D of the HSM, crash modification factors (CMFs) are provided to estimate the safety effects for a variety of treatments or countermeasures. Some of these CMFs are also presented in Part C to complement the predictive methodology in assessing the safety effects of design decisions. By the time the 2nd edition HSM is published, many more important CMFs will have been developed and additional CMFs will be needed to support enhancements to the predictive methodologies, for example, to better predict crashes by type and severity. It is critical that the next edition of the HSM incorporate these new CMFs in filling voids in order to continue to push forward this significant tool. The objectives NCHRP Project 17-72 are to: • Assess the current criteria and existing process for evaluating the quality of and identifying CMFs for appropriate use with the HSM. • Develop proposed revisions to the criteria and process, including how existing and new CMFs may be incorporated in the HSM. Provide guidance for practitioner application of the revised process. • Apply the evaluation criteria to identify and assess CMFs and develop a list of appropriate CMFs for the HSM. The primary objective of this document is to conduct a critical review of the crash modification factor (CMF) quality assessment procedures used in the development of the Highway Safety Manual (HSM) and the FHWA CMF Clearinghouse. This review was conducted for Tasks 1 and 2 of NCHRP Project 17-72. The findings from this review are intended to guide in Task 4 the development of a recommended procedure for evaluating the quality of CMFs and determining which CMFs are appropriate for use with the HSM. This review was conducted within the context of a wider review of knowledge to identify the issues and challenges associated with CMF quality assessment in general. Key components of that expanded review include the identification of factors that indicate CMF quality and the identification of criteria that have been used to determine whether a CMF is acceptable. Findings from the review are documented here along with research that is needed to address key issues. This document consists of three parts. The first part documents the findings from a review of factors used to describe CMF quality. This part initially describes the concept of CMF quality. Then, it provides an overview of alternative procedures used to assess CMF quality including and beyond those used in the HSM and the Clearinghouse. Next, it identifies the factors used to assess CMF quality. Then, it identifies additional factors used to assess the quality of CMFs derived from the results of multiple studies. Next, it describes the criteria used to define an acceptable quality level. The second part documents the findings from a review of techniques used to improve CMF quality, where the improvement takes place after the associated study is published. This part initially reviews the sources of bias in the estimate of a CMF or its standard error. Then, it identifies procedures for correcting published results that are believed to be biased due to study deficiencies. The third part summarizes the critical review findings on key quality assessment issues for the HSM and the Clearinghouse and identifies the research that could be conducted to address these issues as they pertain to the HSM. A-3 

Appendix A1 provides an overview of procedures described in the literature for estimating the safety effect of a treatment using the CMFs from multiple studies of the same treatment. The focus is on alternative procedures for determining whether the variability in CMFs is due to random or systematic sources. A CMF is defined herein as an index (i.e., a value) that describes how much crash experience is expected to change following a modification in design or traffic control. A CMF function is defined as equation used to predict a CMF as a function of application circumstances, including site characteristics. 2. PART I: CMF QUALITY This part documents the findings from a review of factors used to describe CMF quality. This part initially describes the concept of CMF quality and study quality. Then, it provides an overview of alternative procedures used to assess CMF quality, including, not only the HSM and the Clearinghouse ones, but also a procedure that resulted from NCHRP Project 17-25 and one documented by Elvik (2008). Next, it identifies the factors used to assess CMF quality. Then, it identifies additional factors used to assess the quality of CMFs derived from the results of multiple studies. Next, it describes the criteria used to define an acceptable quality level. 2.1 CMF Quality and Study Quality Many investment decisions are made on the basis of the CMFs reported in the literature. The decision is often based on whether the proposed treatment will improve safety, and if it does, whether the safety benefits are sufficient to justify the cost of treatment implementation. As a result, the correctness of the decision is dependent on the accuracy of the CMF. The quality ratings in the CMF Clearinghouse are a reminder that some CMFs are not sufficiently accurate to be the basis for important decisions. In fact, these ratings indicate that there is a wide range of CMF quality reflected in the literature. Information about CMF quality is useful when determining whether to use a CMF (e.g., by comparison with minimum acceptance criteria). When the CMF is used in a safety evaluation, information about CMF quality can be used to weigh the accuracy of the results and the confidence to be placed in decisions made using the results. CMF quality can also be used to assess the relative accuracy of multiple CMFs for the same treatment that are produced by separate studies. CMF quality denotes the extent to which (1) the associated treatment is sufficiently well described to be reproduced by others, (2) the site characteristics are sufficiently well described that intended application sites can be correctly identified by others, and (3) the reported CMF is free of bias and sufficiently precise for road infrastructure investment decisions. A related concept is study quality. Study quality denotes the extent to which (1) the study details are documented and (2) the study design is free of methodological weaknesses that may affect CMF quality. The definitions of CMF quality and study quality both refer to the study documentation. In this regard, quality is an attribute that is assessed by the reader of the document (rather than the researcher conducting the study). Thus, it is only through good documentation that a reader can assess quality (Carter et al., 2012). A-4 

It is important to note that identifying a CMF as having high quality does not necessarily imply that the associated treatment will have exactly the same result when implemented at a new location. Rather, it indicates the result that is likely to be realized collectively at a new set of very similar locations. Differences in effect at any one location may be a result of differences between the new location and the locations studied. 2.2 Overview of Procedures for Assessing CMF Quality A review of the international literature by Elvik (2008) indicated the existence of 35 procedures for assessing study quality. Each procedure included the evaluation of various factors describing quality of the report, data collection procedures, sampling procedure, control for confounding factors, and statistical analysis. Most of the procedures were focused on the review of experimental study designs used in the field of medicine. Three of the procedures were applicable to road safety design. Based on his review, he found that the assessment of the quality was highly subjective and difficult to repeat among reviewers. From this review, he concluded that “next to nothing useful can be learnt…for the purpose of developing a quality scoring system for road safety evaluation studies” (p. 27, Elvik, 2008). This section provides an overview of procedures that have been developed for assessing CMF quality. Four procedures were identified as applying to road safety studies and having been developed after the aforementioned literature review by Elvik. These procedures are identified in the following list: • NCHRP Project 17-25 Procedure • Highway Safety Manual Procedure • CMF Clearinghouse Procedure • Elvik Procedure Each procedure is briefly described in the remaining subsections. The description includes the goal of the procedure, the factors considered, and the criteria used to identify CMFs of acceptable quality (if applicable). NCHRP Project 17-25 The goal of this procedure was to identify CMFs that meet or exceed a minimum acceptable quality. CMF quality was described as high, medium-high, medium-low, or low (Harkey et al., 2008). Only CMFs that were produced by studies with a high or medium-high quality were considered to be acceptable for use. If there were multiple acceptable CMFs for a given treatment, the researchers recommended use of the CMF from the highest ranked study. Each of the quality ratings (e.g., high, medium-high, etc.) was described in terms of the characteristics of the study design. These characteristics are listed in Table 1. For example, a high rating is described as “a rigorous before-after study that incorporated the current best study design and statistical methods” (p. 10, Harkey et al., 2008). In contrast, a low rating was assigned to either (1) a study that used a simple before-after study design without control for biases, or (2) cross-sectional studies in which modeling techniques were questionable. A-5 

Table 1. CMF quality characteristics developed for NCHRP Project 17-25. Study Characteristic CMF Quality • CMF developed in a rigorous before-after study that incorporated the current best study High designs and statistical analysis methods. • Study must also have included a sufficiently large number of treatment sites, a large reference group composed of comparable sites, and enough crashes for statistical validity. • CMF developed in an empirical Bayes (EB) before-after study with limited numbers of Medium-High treatment sites or crashes, or a before-after study that incorporated sound (but not EB) statistical methods and/or may have been reviewed and “vetted” by an expert panel of researchers. • CMF produced by an expert research panel using the combination of findings from different (less controlled) before-after and cross-sectional studies. • CMF developed in a rigorous meta-analysis by a recognized meta-analysis expert. • CMF developed from a cross-sectional analysis (controlling for other factors statistically). Medium-Low • CMF developed from less-than-rigorous before-after studies, but is still judged to be of value. • CMF developed in a simple before-after study without control for biases. Low • CMF developed from cross-sectional studies in which modeling techniques are questionable. Source: Harkey et al., 2008. Highway Safety Manual The goal for Part D – Crash Modification Factors of the HSM was to document the most reliable information about the safety effect of a given safety treatment (HSM, 2010). If there were multiple CMFs found for a given treatment, they were combined to obtain an estimate of the overall average CMF and its standard error. The standard error of the overall average CMF was then evaluated to determine if the overall CMF was known with sufficient precision to be included in the HSM. A procedure was established to identify the CMFs reported in the literature, assess their quality, remove selected biases (if present), compute a best estimate of the CMF (if multiple CMFs were found), and evaluate the quality of the CMF to determine whether it was sufficiently reliable to be included in the HSM (Bahar, 2010). The procedure actually consisted of two procedures completed in sequence–a Literature Review Procedure, followed by an Inclusion Procedure. The Literature Review Procedure consisted of applying the following steps for each document found in the literature: • Step 1. Determine estimate of safety effect of treatment as documented in respective evaluation study publication. This step involved determining the CMF for a given study. In some cases, the CMF was computed by the reviewers using the information found in the study report. • Step 2. Adjust estimate of safety effect to account for potential bias from regression-to-the-mean (RTM) and changes in traffic volume. This step involved determining if the study controlled for bias from RTM and volume change. If either form of bias was present, an estimate was made of its likely magnitude and the reported CMF was adjusted accordingly. • Step 3. Determine ideal standard error of safety effect. This step involved determining the ideal standard error for a given study. In some cases, the ideal standard error was computed by the reviewers using the information found in the study report. • Step 4. Apply method correction factor (MCF) to ideal standard error, based on evaluation study characteristics. This step was used to compute the corrected standard error by multiplying the ideal standard error by a MCF. The MCF is intended to reflect the indirect effect of study quality on standard error. The study design and characteristics associated with each MCF are listed in Table 2. A-6 

• Step 5. Adjust corrected standard error to account for bias from RTM and changes in traffic volume. This step adjusts the corrected standard error from Step 4 for RTM bias, if it was identified in Step 2 as being present. The correction increases the value of the standard error. • Step 6. Combine CMFs when specific criteria are met. In a limited number of cases, multiple evaluation studies provided estimates of safety effect for the same treatment. These estimates were used to compute a weighted average value for the CMF and its standard error. The adjusted CMF from Step 2 and the corrected standard error from Step 5 were used in this calculation. Table 2. Method correction factors used for the Highway Safety Manual. Study Study Characteristic Method Design Correction Factor a Before-after All potential sources of bias were properly accounted for. 1.2 and meta- Uses crash frequency. analysis Accounts for regression to the mean bias. 1.8 (including Uses crash frequency. expert panels) Regression to the mean may not be accounted for but considered to be 2.2 minor, if any. Uses crash frequencies or crash rates. Regression to the mean not accounted for and considered to be likely. 3 Uses crash rates. Severe lack of information published regarding study data and findings. 5 Non- All potential confounding factors have been accounted for by matching 1.2 regression sites. cross- Most potential confounding factors have been accounted for by matching 2 sectional sites. Traffic volume is the only confounding factor accounted for in the study. 3 No confounding factors accounted for in the study. 5 Severe lack of information published regarding study data and findings. 7 Regression All potential confounding factors have been accounted for by variables of 1.2 cross- the regression in an appropriate functional form. sectional Most potential confounding factors have been accounted for by variables 1.5 of the regression in an appropriate functional form. Several important confounding factors were accounted for, and functional 2 form is conventional. Few variables used and functional form is questionable. 3 Severe lack of information published regarding study data study and 5 findings. Source: Bahar, G., 2010. Note: a – A method correction factor of 1.0 was reserved for a rigorous randomized trial evaluation. The RTM bias correction in Step 2 is focused on correcting for the bias incurred when sites are selected for treatment because their short-term trend in observed crash frequency indicates above average values. It is a specific type of RTM bias that is often called “site selection bias” (Chapter 3, HSM, 2010). The technique for making this correction is described in Part II of this document. A-7 

The Inclusion Procedure is intended to ensure that the CMFs presented in the HSM are sufficiently accurate to provide a sound basis for road infrastructure investment decisions. The procedure was developed to quantify the maximum acceptable adjusted standard error for an existing CMF such that future research would be unlikely to show the existing CMF to be inaccurate. Thus, an equation was developed to compute the maximum acceptable adjusted standard error based on consideration of (1) a specified allowable maximum change in the CMF, and (2) a specified value of the maximum standard error likely to be obtained from future research. Using the specified values, it was determined that a CMF would need to have an adjusted standard error of 0.1 or less to be considered for inclusion in the HSM. This criterion was relaxed for the case where a CMF for one crash category or severity (e.g., multiple-vehicle crash CMF) met the criterion but the CMF for another crash category or severity (e.g., single-vehicle crash CMF) from the same study did not meet the criterion. In this case, the CMF for the other category was also included in the HSM if its adjusted standard error was 0.3 or less. In this context, it is important to note that the assessment of this criterion was done after the adjusted standard errors higher than 0.1 were rounded to the first decimal, e.g., an adjusted standard error of 0.14 was rounded to 0.1, and the adjusted standard error of 0.18 was rounded to 0.2. Assessing CMF Quality. The elements of the Literature Review Procedure used to assess CMF quality include the standard error of the CMF from Step 3, the MCF from Step 4, and the bias correction from Step 5. The MCF uses a numeric score to describe the relative quality of the study. Lower quality studies are unlikely to account for most sources of bias, control for confounding factors, and use the appropriate functional form for regression models. The MCFs ranged from 1.2 to 7.0, with smaller values in this range used for high quality studies. The product of the standard error and the MCF is called the “corrected standard error”. The corrected standard error is adjusted for RTM bias (if present) to produce the “adjusted standard error.” It represents an estimate of the true standard error. The Inclusion Procedure uses the adjusted standard error to determine which CMFs are of acceptable quality for inclusion in the HSM. The adjusted standard error can be used to judge CMF quality, where a smaller standard error denotes higher quality. The HSM does not specifically cite quality ratings (e.g., high, medium-high, etc.) associated with specific ranges of adjusted standard error. However, as mentioned earlier, only those studies with at least one CMF with an adjusted standard error of 0.1 or lower (after rounding) are included in the HSM; in other words, the HSM considers these studies to be of higher quality than others. The HSM identifies all CMFs whose confidence interval (defined by the CMF ± two times the standard error) includes 1.0. When this condition is identified, the guidance given in the HSM is, “observed variability suggests that this treatment could result in an increase, decrease, or no change in crashes” and “these CMFs should be used with caution.” CMF Clearinghouse The CMF Clearinghouse is an online database of CMFs that is maintained by the Federal Highway Administration (Carter et al., 2012). It has been updated on a quarterly basis since its creation in 2009. The goal of the Clearinghouse is to identify and include all available CMFs, in sharp contrast to the HSM, and to provide an indication of the quality of each CMF. CMF quality is determined based on consideration of study design, sample size, standard error, potential bias, and source of the data. Each of these factors is independently considered. For each factor, a three-point scale (i.e., excellent = 2, fair = 1, poor = 0) is used to describe the level of rigor, A-8 

accommodation, or compliance. The points associated with each factor are then aggregated to compute a cumulative point-based score. This score is computed using the following equation: Equation 1 𝑆𝑆𝑞𝑞 = 2 × 𝑆𝑆𝑆𝑆 + 2 × 𝑆𝑆𝑆𝑆 + 𝑆𝑆𝑆𝑆 + 𝑃𝑃𝑃𝑃 + 𝐷𝐷𝐷𝐷 where Sq = cumulative point-based score; SD = study design points; SS = sample size points; SE = standard error points; PB = potential bias points; and DS = data source points. The two values of “2” used in Equation 1 are used to give additional weight to the study design and sample size factors. A higher score denotes a higher CMF quality. The characteristics associated with each of the five factors are described in Table 3. Table 3. CMF quality characteristics developed for the CMF Clearinghouse. Factor Factor Characteristics Points Study design Empirical Bayes before-after with reference group 2 Full Bayes before-after with reference group Before-after with comparison group without bias in site selection (i.e., area- wide implementation or random selection) Cross-sectional regression model 1 Case control Cohort Other coefficient based analysis Before-after with comparison group with some selection bias Simple before-after 0 Simple with-without cross-sectional (non-regression) Before-after with comparison group with large effect from selection bias Sample size Before-after: before crashes plus expected after crashesa for treatment 2 group are 200 or more. Cross-sectional: total crashes are 400 or more. Before-after: before crashes plus expected after crashes for treatment 1 group are 100 to 200. Cross-sectional: total crashes are 200 to 400 or more. Before-after: before crashes plus expected after crashes for treatment 0 group are fewer than 100. Cross-sectional: total crashes are fewer than 200. A-9 

Factor Factor Characteristics Points Standard error CMF is significantly different from 1.0 at the 0.05 significance level. 2 CMF is significantly different from 1.0 at the 0.1 significance level but not at 1 the 0.05 level. CMF is not significantly different from 1.0 at the 0.1 significance level. 0 Potential bias Controls for all potential bias. 2 Controls for important potential bias, such as traffic volume. 1 Minimal consideration for potential bias. 0 Data source Multiple states (or multiple countries for international). 2 One state (or one country for international), multiple jurisdictions. 1 Single jurisdiction. 0 Source: Carter, 2012. Note: aexpected after crashes are the expected number of crashes in the after period had the treatment not been implemented. The studies considered for inclusion in the Clearinghouse have to meet minimum criteria. These criteria include (Carter, 2011): • Study results are based on crash data. The results of expert panels may be included if their decisions were data-driven and based on crash-based evaluation. • Study objective was to quantify the safety effect of a road feature or characteristic. • Study focused on the safety effect of an infrastructure characteristic, feature, or modification that would fall under engineering responsibilities (not planning-level or area-wide studies; not studies of public education or law enforcement efforts). • Study explicitly presents CMFs or CMF functions. The CMFs for all studies that met the aforementioned criteria are provided in the Clearinghouse, along with a star rating. The star rating converts the cumulative point-based score to a number ranging from 1 to 5 (and shown using a number of stars). A CMF with a “5 star” rating has the highest quality. A 5 star rating is associated with a score of 14, 4 stars is associated with 11 to 13, 3 stars is associated with 7 to 10, 2 stars is associated with 3 to 6, and 1 stars is associated with 1 to 2. It is important to note that the CMF Clearinghouse rating procedure was only applied for CMFs that were published after the HSM was published. For those CMFs that were published in the HSM and/or NCHRP 17-27 (iTrans, 2004), the adjusted standard error and the information on whether the CMF was significantly different from 1.0 at the 0.05 significance level was used to assign a star rating. Table 4 shows the star ratings based on the adjusted standard error and whether the CMF was significantly different from 1.0 at the 0.05 significance level: A-10 

Table 4. Adjusted Standard Error and Star Rating Significant? Adjusted Standard Error Star Rating (0.05 sign) 0 – 0.05 Yes 5 0.05 – 0.2 Yes 4 0.2 – 0.4 Yes 3 > 0.4 Yes 2 0 – 0.4 No 3 0.4 – 0.5 No 2 > 0.5 No 1 Elvik Procedure Elvik (2008) developed a systematic approach to assessing the quality of road safety evaluation studies. His approach consists of the separate evaluation of factors in four groups. These groups are described in the list below: • Statistical association – consists of items that describe the statistical association between a road safety treatment and its effect. • Causal relationship – consists of criteria for determining the existence, and strength, of a causal connection between the treatment and its effect. • Control for confounding – consists of items that describe potential confounding influences on treatment effect and the degree to which they were controlled. • Description of study – consists of information describing the potential for generalization and application of study findings. The first, second, and fourth groups collectively include 10 evaluation factors. These factors are listed in Table 5. The factors in the third group vary by study design. They are listed Table 6. There are three study designs listed in Table 6. Elvik (2008) also provided similar lists of factors for the following study designs: experiments, case-control, and time-series. The factors for these studies are not shown herein because the objective of this subsection is to provide an overview of each procedure, which is accomplished by showing a sample of factors for the more common road safety study designs. A-11 

Table 5. Study quality characteristics developed by Elvik. Group Factor (weight) Levels Score Statistical Significant effect Detecting an effect of practical interest possible 1 association (0.06) Detecting an effect of practical interest not possible 0 (∑w= 0.12) Strength of Comparison of effect size with other effects in data possible 1 association (0.03) Comparison of effect size with other effects in data not possible 0 Consistency Consistency of association across subsets of data accessible 1 (0.03) Consistency of association across subsets of data not accessible 0 Causality Clarity of direction Causal direction can be determined and is in right direction 1 (∑w= 0.30) (0.10) Causal direction cannot be determined or is in wrong direction 0 Causal Mechanism fully specified and evaluated empirically 1 mechanism (0.06) Mechanism partly specified and evaluated empirically 0.67 Mechanism specified, but not evaluated empirically 0.33 No causal mechanism discussed or identified 0 Theoretic A well-established theory exists that may explain study findings 1 explanation (0.03) No well-established theory exists that may explain study findings 0 Dose-response Study design allows for assessing a dose-response pattern 1 (0.08) Study design does not allow assessing a dose-response pattern 0 Effect on target Study design allows for assessing specificity of effect 1 group (0.03) Study design does not allow for assessing specificity of effect 0 Confounding see Table 6 (∑w= 0.50) Description of Sampling Study of entire population 1 Study technique (0.03) Random sampling from known sampling frame 0.75 (∑w= 0.08) Non-random sampling; criteria stated 0.5 Convenience sample 0.25 Specification of Fatal, serious, slight injury, and property damage only specified 1 severity (0.05) Fatal, injury, and property damage only specified 0.75 Fatal, serious injury, and slight injury specified 0.75 Injury (including fatal) and property damage only specified 0.5 Study limited to injury crashes, severity not further specified 0.25 Severity not specified; mixing of levels probable 0 Source: Elvik, 2008. A-12 

Table 6. Confounding factors considered by Elvik procedure. Study Design Potentially Confounding Levels Score Factors (weight) Before-after Regression-to-the-mean Empirical Bayes model-based control 1 (RTM) (0.40) Control for RTM by means of simpler techniques 0.5 No control for RTM 0 Long-term trends (0.30) Controlled (comparison group or time-series) 1 Not controlled 0 Changes in traffic volume Local exogenous changes in volume controlled 1 (0.10) Local exogenous changes in volume controlled 0 Co-incident events (0.05) No co-incident events known to have occurred 1 Co-incident events occurred 0 Multiple treatments (0.10) Use of multiple treatments known and controlled 1 Use of multiple treatments not known or controlled 0 Crash migration (0.05) Migration identified and controlled 1 Migration is judged not likely to occur 0.5 Migration is judged likely, but was not controlled 0 Non-regression Self-selection of subjects to Not present; selection is random 1 cross-sectional treatment (0.20) Present, adjusted for statistically or by matching 0.67 Not positively known, but suspected 0.33 Known to occur, no adjustment for it 0 Endogeneity of treatment Assessed and found not to be present 1 (0.20) Present, adjusted for statistically 0.67 Suspected; inconclusive evidence; no adjustment 0.33 Treatment known to be endogenous, no adjustment 0 Differences in traffic Differences adjusted for by multivariate model 1 volume (0.20) Adjusted for using accident rates 0.5 Not controlled 0 Differences in traffic Differences controlled 1 composition (0.20) Differences not controlled 0 Differences in other risk Control for all factors known to affect safety 1 factors (0.20) Control for some factors known to affect safety 0.5 Control for few or no risk factors that affect safety 0 Multivariate Endogeneity of treatment Assessed and found not to be present 1 Analyses (0.40) Present, adjusted for statistically 0.67 (Regression Suspected; inconclusive evidence; no adjustment 0.33 cross-sectional) Treatment known to be endogenous, no adjustment 0 Functional form (0.10) Form explicitly chosen and shown to be best 1 Form chosen by default; by standard specification 0.5 Implausible form; strange or non-logical implications 0 Collinearity among Shown not to be a problem 1 explanatory variables Insufficient information to asses if it is a problem 0.5 (0.10) Suspected or shown to be a problem 0 Omitted variable bias (0.10) No omitted variables can be identified 1 Insufficient information to assess 0.5 Suspected or shown to be present 0 Specification of residual Reasonable specification 1 terms (0.05) Questionable specification 0 Inappropriate model form Single-state or plausible dual-state model used 1 (0.10) Theoretically implausible dual-state model used 0 Inappropriate choice of Number of accidents used 1 dependent variable (0.15) Accident rate (linear) used 0 Source: Elvik, 2008. A-13 

To assess the quality of a study, the reviewer makes a subjective judgment about each factor using the factor levels listed. The corresponding score is multiplied by the associated weight (listed in column 2 of both tables). The quality score is computed as the sum of weighted scores for all factors. It has a value in the range of 0.0 to 1.0, with a value of 1.0 representing the highest quality study. Elvik (2008) applied the procedure to several studies. Based on the results, he offered the interpretations in Table 7 as a general guide for interpreting the resulting quality scores. Table 7. Study assessment based on quality score. Quality Study Description Score Assessment Range 0.00 to 0.499 Inadequate Study findings are more likely to reflect methodological weaknesses than true effect of treatment. 0.50 to 0.599 Weak Study provides weak evidence of the effect of the treatment. Significant methodological shortcomings exist. 0.60 to 0.799 Moderately good Study findings are more likely to show the effects of the safety measure than merely the effects of methodological weaknesses. 0.80 to 1.00 Very good Study findings are clearly more likely to show the true effects of a road safety measure than the effects of methodological factors not adequately addressed by the study. 2.3 Factors Used to Assess CMF Quality This section provides a summary of the factors used to assess CMF quality. The summary focuses on the factors included in the procedures described in the previous section. Several factors are discussed in detail and observations are offered regarding their use in a quality assessment. These factors include: • Bias in CMF, • Bias in CMF standard error, • Data source, and • Basis in expert judgment. The first subsection to follow provides the aforementioned summary. The factors listed above are discussed in the subsequent subsections. The focus of this section is on factors used to assess the quality of a single CMF (and the study that produced it). The next section provides a discussion of additional factors that can be used to assess the quality of CMFs derived from the results of multiple studies of a common treatment. Study Quality Table 8 lists the CMF quality assessment factors identified in a review of the literature. A total of 36 factors are listed; however, a subset of this number is used in the evaluation of any one study because some factors are only applicable to certain study designs (as specified in column 1 of the table). A-14 

The information in the last five columns indicates the extent to which each factor is considered in one of the four procedures reviewed in the preceding section. The column attributed to Carter et al. represents the factors identified in their documentation of recommended CMF development protocols. Table 8. Summary of factors used to assess CMF quality. Group Factor Level of Factor Consideration1 Carter Elvik CMF HSM2 NCHRP et al. (2008) Clearing 17-252 (2012) -house 2 Statistical Suitability of comparison or reference groups ID -- -- -- -- association – Sample size ID -- E -- G before-after study Choice of dependent variable (desirably crash count) -- -- -- E -- design Statistical Suitability of functional form ID E -- ID -- association – Collinearity among explanatory variables ID E -- -- -- regression cross- Specification of residual (error) terms ID E -- -- -- sectional study Choice of dependent variable (desirably crash count) -- E -- -- -- design Sample size ID -- E -- G Temporal and spatial correlation in observations ID -- -- -- -- Statistical Analysis driven examination of results from studies -- -- ID G E association – using crash data expert panel Statistical Standard error ID -- -- E ID association - Statistically significant effect (confidence interval) ID E E E -- analysis Strength of effect, relative to other factors in the data -- E -- -- -- Consistency of effect in subsets of the data -- E -- -- -- Causality Clarity of direction specified -- E -- -- -- Causal mechanism specified -- E -- -- -- Theoretic explanation available -- E -- -- -- Dose-response pattern assessed -- E -- -- -- Effect on target group or crash type specified -- E -- -- -- Confounding – Regression-to-the-mean ID E G E -- before-after study Long-term trends ID E (as a G -- Changes in traffic volume ID E group) E -- Coincident events ID E G -- Implementation of other safety treatments ID E (as a -- Crash migration -- E group) -- State-to-state differences ID -- -- Confounding – Endogeneity of treatment -- E G G -- non-regression Differences of traffic volume -- E (as a ID -- cross-sectional Other differences that influence safety -- E group) G -- study Confounding – Endogeneity of independent variables ID E G G -- regression cross- Omitted variable bias ID E (as a (as a -- sectional study State-to-state differences ID -- group) group) -- Description of Technique used to sample study units -- E -- -- -- study Specification of crash type and severity ID E -- -- E Data quality addressed ID -- -- -- -- Data sources (i.e., one/many states; in/out of U.S.) -- -- E -- E Data age (old data may not reflect current conditions) -- -- -- -- E Documentation (details for application and evaluation) ID -- -- -- E Notes: 1 – E: factor is explicitly evaluated using specified levels; G: factor is evaluated in general terms (specified levels not provided); ID – factor identified as informative but no guidance is provided for its evaluation; “--”: factor not indicated. 2 - The Clearinghouse, HSM, and NCHRP Project 17-25 procedures associate a quality level with each study design but may not delve into the details of the study design (as listed in the table) that are more directly related to quality. A-15 

For any one source identified in the last five columns, the extent to which a factor was considered is indicated by one of the following four codes: E, G, ID, or --. The letter “E” is used to indicate that the factor is explicitly evaluated in the associated procedure using specified levels or criteria. The letter “G” is used to indicate that the factor is evaluated in general terms because specific levels are not provided. The letters “ID” are used to indicate that the factor is identified as informative to an assessment of CMF quality but no guidance is provided for its evaluation. The character “--” is used to indicate cases where the factor is not explicitly identified. As indicated by the information in the last four columns, none of the four procedures explicitly considers all of the factors listed. The Elvik procedure considers the most factors. The NCHRP Project 17-25 procedure considers the fewest number of factors. None of the factors is explicitly considered by all four procedures. While several factors are generally considered or identified as informative in multiple procedures, the following are explicitly considered in two or more procedures: standard error, statistical significance, regression-to-the-mean, changes in traffic volume, specification of crash type and severity, and data sources. Study design is not listed as a factor in Table 8. Rather, study design in identified in the first column and the factors that are unique to each study design are used to more specifically assess quality. This approach is used by Elvik (2008). In contrast, the Clearinghouse, HSM, and NCHRP Project 17-25 procedures associate a quality level with each study design but do not delve deeply into the related factors that affect quality. For example, the NCHRP Project 17-25 procedure indicates that a “high” quality CMF is produced by a “rigorous before-after study,” but it does not explicitly identify the extent to which confounding factors would need to be addressed in the study. The need to consider confounding is more clearly indicated in the Clearinghouse and the HSM procedures than it is for the NCHRP Project 17-25 procedure. The Clearinghouse procedures indicate the need to identify sources of potential bias (which are often caused by confounding influences), but it does not list the possible sources of bias to be considered. The HSM procedure specifically indicates a need to assess the number of confounding factors that have been accounted for in the study design or statistical analysis (but it does not list these factors). For these reasons, confounding factors are assessed in general as a group, rather than individually by these two procedures. This condition is shown in Table 8. Elvik (2008) surveyed eight road safety experts about the concept of study quality. One of his goals was to identify the factors that should be considered when assessing study quality. Based on the survey results, he concluded, “There is no consensus among leading road safety researchers about the concept of study quality” (page 36, Elvik, 2008). There was agreement that study quality is comprised of many factors; although opinions differed regarding the factors to be considered and their relative importance. Some of the more commonly cited factors include: • The uncertainty of the estimate of effect should be estimated and presented. • Appropriate statistical techniques of analysis should be used. • Assumptions made in analyzing data should be made explicit and should be plausible. • Various confounding factors should be controlled. • Interpretation of study findings should be honest and objective, and allow for the possibility that the study is inconclusive. Bias in CMF Bias is defined as the difference between the expected CMF and the true CMF. The accuracy of a CMF is a function of the amount of bias in the CMF. Accuracy improves with a decrease in bias. The A-16 

amount of bias in a given CMF is difficult (perhaps impossible) to fully quantify. However, it can be minimized by careful study design or the use of statistical analysis techniques. The confounding factors listed in Table 8 are each a source of possible bias if they are not addressed through study design or statistical analysis. Elvik (2008) reviewed several reports that used a before-after study to quantify a CMF. The purpose of his review was to obtain a rough sense of the magnitude of the bias introduced in the results when a confounding factor was not controlled. His findings are summarized in Table 9. Table 9. Typical range of bias due to various sources. Factor Changes True CMF by… Comment Average Maximum Change Change Regression-to-the-mean 6% 30% Neither the magnitude nor direction of bias can be reliably predicted. Long-term trends 4% 17% Neither the magnitude nor direction of bias can be reliably predicted. Changes in traffic volume 3% 5% -- Implementation of other 6% per -- Magnitude of bias will vary with safety treatments added number of other treatments treatment implemented. Crash migration 15% 27% -- Source: p. 65, Elvik (2008) It should be noted that Elvik (2008) also reviewed many reports that used study designs other than before-after. The sources of bias in each report were identified, as well as the likely effect on the CMF. However, he did not review a sufficient number of these reports to generalize the average and maximum change by confounding factor. The effect of confounding factors varies widely, as suggested by the trends shown in Table 9. The magnitude and direction of the bias often varies from site-to-site based on site characteristics. Elvik (2008) observed that study results may not be biased even though the researchers failed to control for one or more confounding factors. Bias in CMF Standard Error The HSM procedure described in a previous section implies that the standard error reported in a typical road safety study is likely to be biased smaller than the true standard error. The sources of bias were described in terms of study design and the proportion of confounding factors controlled. The unbiased estimate of the true standard error (i.e., adjusted standard error) is computed using the following equation (Bahar, 2010). Equation 2 𝑠𝑠𝑎𝑎 = �(𝑠𝑠𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 × 𝑀𝑀𝑀𝑀𝑀𝑀 )2 + (𝐶𝐶𝐶𝐶𝐶𝐶𝑟𝑟 × 𝑓𝑓𝑅𝑅𝑅𝑅𝑅𝑅 )2 where sa = adjusted standard error; A-17 

sideal = reported standard error; MCF = method correction factor (from Table 2); CMFr = reported CMF; and fRTM = RTM bias correction factor. The RTM bias correction factor has a value between 0.05 and 0.25 if RTM bias is thought to exist. The exact value is judged by the person reviewing the study. A value of 0.05 is used when a larger portion of the population of sites was treated and many years of before period data were included in the study. A value of 0.25 is used where only a small proportion of sites were treated, those selected for treatment had a high accident frequency, and few before years of data were included in the study. Harkey et al., (2006) reviewed the MCFs in Table 2 and documented their findings in a memorandum to the Chair of the Task Force for the Development of a Highway Safety Manual. They found that the adjusted standard error provides valid information about CMF quality. However, they argued that the MCF values should be more consistent across study designs. They noted that the higher quality before- after studies should have smaller MCFs than those based on cross-sectional data. The need for consistency is amplified by the fact that the Literature Review Procedure includes a procedure for combining CMFs and that some of these CMFs may be based on different study designs. Harkey et al. (2006) also questioned why the MCF for the highest quality study design was greater than 1.0. This trait inflates the standard error for every CMF considered for inclusion in the HSM. The HSM Inclusion Procedure limits the CMFs included in the HSM to those with a standard error less than a specified maximum value. As a result, any unjustified inflation will increase the number of CMFs that do not satisfy the inclusion criterion. Finally Harkey et al. (2006) examined Equation 2 and observed that both the MCF and the RTM adjustment include consideration of the magnitude of RTM bias. As a result, they felt that the equation was “double counting” the effect of this bias. Like Harkey et al., Elvik (2008) found the adjusted standard error to be an attractive means of obtaining a quantitative indicator of CMF quality. As a second benefit, he noted that the nature of the adjustment was to increase the standard error such that a weighted-average CMF (when combining results from multiple studies with adjusted standard error as the weight) would more reasonably represent overall study quality. On the other hand, Elvik (2008) noted that the use of MCFs introduces an element of subjectivity into the calculation of adjusted standard error. Data Source Data source describes the geographic representation in the data used to derive one or more CMFs. The quality rating procedures used for the CMF Clearinghouse recognize geographic diversity as a positive feature of a study (Carter, 2011). CMFs from geographically diverse data are given a higher quality rating. Diversity is rationalized to improve the transferability of the reported CMFs. There may be a point of diminishing returns regarding diversity. Harkey et al. (2006) suggest that CMFs for which the data are largely representative of countries outside of North America may be of lower quality. They rationalize that treatment effect may vary among countries and continents due to differences in driving behaviors, roadway design, and other factors. A-18 

Basis in Expert Judgment The Clearinghouse, HSM, and NCHRP Project 17-25 procedures acknowledge the potential value of CMFs produced by expert panels. The NCHRP 17-25 procedure provided the most detailed explanation of the rigor needed to develop a high quality CMF using expert panels. The HSM procedure included consideration of CMFs from expert panels in determining the MCF value. The Clearinghouse indicates that CMFs from expert panels can be considered for inclusion, but does not describe how the quality assessment procedure is used to evaluate expert-panel-based CMFs. It is important to note that the only expert-panel-based CMFs in the Clearinghouse were from the HSM and/or NCHRP 17-27 (iTrans, 2004). Since the expert-panel-based CMFs do not include a standard error, they are not rated in the Clearinghouse. It is notable that the level of rigor for assessing expert-panel CMFs has declined from the NCHRP Project 17-25 procedure, to the HSM procedure, to the Clearinghouse, to the Elvik procedure (which does not address them at all). This time trend likely reflects a trend in the profession over the last several years to develop more CMFs directly from data, as opposed to using judgment-based expert evaluation of safety information and implementation experience. Nevertheless, Washington et al. (2010) in a critique of the HSM expert panel process, suggest (p.10) that “with theoretical support for the CMF analytical approach and an established history of appropriate uses of expert opinions and panels, the use of expert panels is likely to continue in the near future”. They note, however, that important questions remain to be addressed, including: 1. “Are the results derived from expert panels accurate and precise? 2. Can expert panels be used to derive estimates of uncertainty? 3. Do results across expert panels differ, and if so, how? 4. Can expert panels be made to ensure repeatable and accurate results? 5. Should expert panels follow informal procedures (as they have) or more formal and structured procedures such as the Delphi method?” Another aspect of “expert” judgment is the need for subjectivity in certain aspects of both the HSM and Clearinghouse procedures. The extent to which the repeatability of the resulting assessment has been evaluated is unclear based on published information about the procedures. 2.4 Additional Factors to Assess CMFs from Results of Multiple Studies CMFs from multiple studies of the same treatment are often examined to determine if the collective set of CMFs provides greater confidence in the treatment’s effect and its transferability. In some cases, the CMFs may be combined to produce a single value that represents the best estimate of treatment effect. In other cases, the CMFs may be evaluated to determine if there is a systematic trend such that the CMF varies predictably as a function of site characteristics. In all cases, the information obtained from the evaluation of multiple CMFs can be used to obtain more reliable information about treatment effect. This section summarizes the additional factors that can be used to assess the quality of CMFs obtained from multiple studies of a common treatment. The factors used to assess the quality of a single CMF were described in the previous section. A-19 

Several factors are described in this section, and observations are offered regarding their use in a quality assessment. These factors include: • Study quality • Publication bias, and • Systematic trend due to differences in site characteristics. Study Quality Table 10 lists the CMF quality assessment factors for the case where there are several CMFs available from multiple studies. Seven factors are listed. These factors would be in addition to those used to assess the quality of an individual study or its CMF (the factors for assessing an individual study or CMF were identified in the previous section). The information in the last four columns indicates the extent to which each factor is considered in the reference documents reviewed. The column attributed to Carter et al. (2012) represents the factors identified in their documentation of CMF development guidelines. The column attributed to OECD represents the factors used to assess the transferability of CMFs using meta-analysis (Elvik 2005). Only two columns of Table 10 correspond to procedures reviewed in a preceding section (i.e., HSM and NCHRP 17-25). The Elvik procedure and the CMF Clearinghouse procedure are not included. All four procedures are reviewed in the section titled Overview of Procedures for Assessing CMF Quality. The Elvik procedure is focused on determining the quality of a single CMF study. The CMF Clearinghouse procedure presents CMFs from all studies of acceptable quality for any given treatment. Thus, it does not consider factors used to evaluate the collective quality of CMFs from multiple studies. For any one source identified in the last four columns, the extent to which a factor was considered is indicated by one of the following four codes: E, G, ID, or --. The letter “E” is used to indicate that the factor is explicitly evaluated in the associated procedure using specified levels or criteria. The letter “G” is used to indicate that the factor is evaluated in general terms because specific levels are not provided. The letters “ID” are used to indicate that the factor is identified as informative to an assessment of CMF quality but no guidance is provided for its evaluation. The character “--” is used to indicate cases where the factor is not explicitly identified. A-20 

Table 10. Summary of factors used to assess CMF quality when based on multiple studies. Group Factor Description Level of Factor Consideration1 Carter HSM NCHRP OECD et al. 17-25 (2012) (2012) Statistical Statistically Use of standard error to assess statistical -- -- E -- association - significant effect significance. analysis (confidence interval) Confounding Time trend Presence of time trend suggests that the CMFs -- -- -- G from older studies cannot be used. It should be assessed and, if present, controlled by study design. Publication bias Tendency to publish only significant effects should -- -- -- G be assessed and, if present, account for it by statistical technique or control it by study design. Differences in site Determine if systematic variation is present, and if ID -- ID G characteristics it is, account for it statistically or control for it (CMF standard through study design. deviation) Description Data source Minimum proportion of CMFs is from studies -- -- E -- of study conducted in the location of interest (e.g., United States, Midwest, Illinois). Data quality Studies from which CMFs are obtained should be -- E -- E of acceptable quality. Sample size Results represent a large number of studies -- -- -- G describing several regions (ensures transferability). Note: 1 – E: factor is explicitly evaluated using specified levels; G: factor is evaluated in general terms (specified levels not provided); ID – factor identified as informative but no guidance is provided for its evaluation; “--”: factor not indicated. Publication Bias Harkey et al. (2008) define publication bias as the tendency to only publish studies that produced favorable results for the treatment being evaluated. A more specific definition of publication bias is the tendency to not publish results whose findings are counterintuitive, difficult to interpret, or not statistically significant (OECD 2012). Techniques are available for detecting publication bias (Elvik 2011). The presence of publication bias can bias a CMF that is computed from the results of multiple studies. It can also bias the computed CMF standard error, which may inflate measures of consistency and give a false impression that study findings are transferable (OECD 2012). In other words, publication bias can affect the CMF and its standard error, both of which influence CMF quality. Systematic Trend Due to Differences in Site Characteristics The safety effect of a treatment may vary from location to location because of differences in site characteristics (e.g., road geometry, traffic characteristics, traffic control) and local conditions (e.g., driver behavior, enforcement level, weather) that influence the safety effect of a treatment. As a result, the value of a CMF will often vary from study to study. The variability found in a collective set of CMFs is an indication of the presence of the aforementioned influences. A small amount of variability suggests that the treatment effect is transferrable to other locations, regardless of their differences. A small variability also suggests that A-21 

greater confidence can be placed in road infrastructure investment decisions that are based on the CMF. For these reasons, Hauer et al. (2012) argue that CMF standard deviation is an important indicator of CMF quality, perhaps more so than the standard error of the estimate of the CMF. Appendix A1 describes alternative procedures for determining whether the variability in CMFs is due to random or systematic sources. 2.5 Defining Acceptable Quality Level This section reviews the criteria used by various sources to identify an acceptable level of CMF quality. The sources reviewed are those discussed in the previous section titled, Overview of Procedures for Assessing CMF Quality. The sources are identified in Table 11. Table 11. CMF acceptance criteria used with four quality assessment procedures. Source of Goal (for a given treatment) Factors Considered for Acceptance Criteria Used Procedure Acceptance NCHRP Include the highest quality Study design CMF has a high or medium- Project 17-25 published CMF based on an high quality. overall assessment of factors. Additional factors for meta- Additional factors for meta- analysis CMFs: analysis CMFs: • Percent of studies from • At least 20% of studies from North America North America. • Confidence interval • 95% confidence interval does not include 1.0 HSM Include the highest quality Adjusted standard error.1, 2 At least one of the CMFs from a CMF that is unbiased and study has an adjusted standard unlikely to change in value by error of 0.1 or lower (after a significant amount as new rounding) research is conducted. CMF Include all published CMFs No specific factors None in addition to that in “All Clearinghouse and provide an indication of considered (other than that procedures.” CMF quality. listed in “All procedures”). Elvik (2008) Provide an indication of study No specific factors None in addition to that in “All quality. considered (other than that procedures.” listed in “All procedures”). All procedures Dependent variable CMF based on crash data (not on surrogate measures of safety or anecdote). Notes: 1 – Reported standard error is adjusted to a larger value based on the consideration of study design and likely presence of bias (due to confounding factors) in the results of the original research. 2 – If multiple studies are published for same treatment, then adjusted standard error is based on combined CMFs. Each of the sources identified in Table 11 describe a procedure for evaluating CMF or study quality; however, the intended use of the evaluation results varied. The NCHRP 17-25 and HSM procedures had a goal of identifying the highest quality CMF. The Clearinghouse and Elvik procedures had a goal of simply assigning a quality score to all CMFs. The former goal is intended to provide analysts with the highest quality information about the safety effect of a given treatment. The latter goal is intended to provide the user with information about one or more studies of a given treatment, and an indication of study quality to help the analyst make a rational decision about whether and how to use the study results. A-22 

The criteria used by the NCHRP 17-25 and HSM procedures was noted by Harkey et al. (2006) to be based on fairly simple generalizations of study design and possible sources of bias. They pointed out that this basis reinforces the need for the generalizations in Table 1 and Table 2 to be reliable. This reliability could be measured in terms of their consistency in rating across study designs and the consistency of rating results among reviewers applying the procedure. These observations were partial motivation for Elvik to develop a more rigorous procedure. 2.6 Summary of Findings Regarding Factors Used to Describe CMF Quality and Study Quality This section summarizes the general findings from the review of the literature regarding factors used to describe CMF quality and study quality. Four procedures for assessing CMF quality or study quality were briefly described in a previous section. All procedures indicate the need to review the study documentation for the purpose of identifying the likely presence of factors that could bias the results (e.g., regression-to-the-mean). The HSM procedure is unique among the four procedures because it went beyond quality assessment. Specifically, it included a technique for combining CMFs from multiple studies of a common treatment. It also included a technique for correcting the published results to reduce any perceived bias. Appendix A1 describes techniques for combining CMFs from multiple studies. Part II of this document discusses bias corrections intended to improve CMF quality. Elvik (2008) conducted a review of the literature on the topic of study quality assessment procedures. He found that only a few procedures had been developed for the purpose of assessing the quality of road safety evaluation studies. He found that the application of these procedures was highly subjective and rarely tested in a scientifically defensible way. The literature review conducted for this document did not reveal the existence of any research on road-safety-related study or CMF quality since 2008. Three procedures (i.e., NCHRP Project 17-25, HSM, and CMF Clearinghouse) are focused on assessing the quality of a CMF. The Elvik procedure is more broadly focused on assessing study quality. The guidelines for applying the three procedures are most clear on the evaluation of a CMF, or a simple linear CMF function having a single regression coefficient. Their application to CMF functions with a complex form or multiple regression coefficients is not addressed. All of the procedures require a high level of documentation for each study whose quality is assessed. This need for information about the details of a study is inherent to the assessment of the study’s quality. Study documentation should include a description of the study, the statistical validity of the findings, and a discussion of how the researchers addressed potential confounding factors (Carter et al., 2012). There are many factors that affect the quality of results from a study. These factors are summarized in Table 8. There are some additional factors that must be considered when assessing the quality of a CMF derived from the CMFs produced by multiple studies of a common treatment. These factors are summarized in Table 10. To what extent should subjective criteria be part of the process? Elvik (2008) found that most quality assessment procedures in the scientific field are the highly subjective, which is not desirable from the standpoint of producing repeatable and reliable CMF quality assessments. He sought to develop a procedure based in science that would minimize the subjectivity of the quality assessment. However, after a relatively in-depth investigation, he concluded that his “research effort has by and large been A-23 

unsuccessful” and that “a large element of arbitrariness is unavoidable in any formal quality scoring system” (p. 80, Elvik, 2008). How should the assessment procedure be updated to reflect emerging statistical analysis methods? The procedures described in this document do not recognize the differences among analysis methods used to quantify the CMF. These differences may result when one method is more efficient or effective in improving CMF accuracy. The quality evaluation factors used in these procedures specify a desired end result (e.g., controlled for regression-to-the-mean bias: yes or no). The reviewer assesses then whether the method used controlled for a confounding factor, produced a small standard error, etc. In this manner, the procedures are sufficiently flexible that they can continue to be used as more sophisticated analysis methods emerge. No distinction would be made in the quality score unless the emerging method alters the study findings (e.g., the study results were not statistically significant when the old method was used, but they are significant when the new method is used). A second tier could be added to the quality assessment process wherein the degree to which the method used was efficient or effective at producing a higher quality result. In this manner, a study that uses a new method would likely receive a higher quality score. The challenge with this approach (in addition to the overall complexity of adding a second evaluation tier) is that it would require a quality scoring system that is not bounded in terms of the highest quality score. For example, a procedure that currently assigns 5 stars to the highest quality study would need to allow for more stars (as new methods emerge) such that 6 stars would designate the highest quality score when a new method is available. Alternatively, if the upper bound associated with highest quality is retained, then all previously evaluated studies would need to be reassessed (as new methods emerge) to reflect that the older methods no longer justify the highest score. Continuing the example, an older study may have been designated as a 5-star study when it was first published but once the new method emerges, the older study results are downgraded to 4 stars. The large amount of work suggested by this alternative suggests that it may be better to use the first option (i.e., unbounded limit on highest quality score) if a second tier is added to the quality assessment process. How strong is the correlation between study design and quality? As shown in Table 2, the method correction factors (MCF) in the HSM procedure are fairly similar across study designs, especially for those studies associated with a small MCF. As a result, a well-designed before-after study could have the same MCF as a regression cross-sectional study. However, general wisdom is that a CMF from a sound before-after study is more reliable than a CMF from a cross-sectional study (Harkey et al. 2006). Elvik (2008) applied his procedure to 17 published studies. He produced a quality score for each study. Examination of the results indicates there is very little correlation between study design and quality. The scores for three before-after studies ranged from 0.533 to 0.863 (where 1.0 represents the highest quality and 0.0 the lowest quality). In contrast, the scores for three regression cross-sectional studies ranged from 0.669 to 0.735. This finding suggests that there is little correlation between study design and quality; which is consistent with the trend noted in the previous paragraph. The trend in the MCFs in Table 2 and those reported by Elvik (2008) suggests that there is little correlation between study design and quality. In support of this trend, it was noted in the review of Table 8 that study design category is not directly related to study quality. Rather, the factors associated with a given study design have a more direct influence on study quality. A-24 

3. PART II: CORRECTIONS TO IMPROVE CMF QUALITY 3.1 Introduction This part documents the findings from a review of techniques used in the post hoc correction of published study results. The discussion in this part is focused on techniques that are used to improve the quality of the published CMF. The procedures described in the previous part are used to assess the quality of a published CMF. This part initially reviews the sources of bias in the estimate of the CMF and its standard error. Then, it identifies procedures for correcting published results that are believed to be biased due to study deficiencies. Finally, it summarizes the key issues and identifies the research that could be conducted to address these issues. 3.2 Background There are many possible factors that could bias a CMF or CMF function. Most sources of bias are due to confounding factors that are not controlled through study design or statistical analysis. A list of the typical confounding factors is provided in Table 8 and Table 10. In the context of a post hoc assessment of CMF quality, a reviewer may believe that one or more sources of bias is present in a reported CMF based on a determination of whether a confounding factor was controlled. In this case, CMF study quality can be improved by removing the bias from the CMF, its standard error, or both. Another technique for improving CMF quality can sometimes be available when there are multiple CMFs reported in the literature for a common treatment. These CMFs can be combined to obtain an estimate of the overall average CMF and its standard error. They may also be used to develop a CMF function that predicts the CMF as a function of site circumstances. Techniques are described in Appendix A1 for determining whether the CMFs should be combined or used to develop a CMF function. The remainder of this document describes techniques for removing identified biases. The focus is on the techniques described by Bahar (2010). These techniques were used to remove bias from the CMFs presented in the HSM. The techniques removed bias that was caused by lack of control for regression-to- the-mean (RTM) and for changes in traffic volume. 3.3 Correcting CMFs for Regression to the Mean The HSM procedure includes a technique for correcting published CMFs when site selection bias is believed to be present in the published results. This bias is a specific type of RTM bias that occurs when the sites are selected for treatment because their short-term trend in observed crash frequency indicates above average values. The guidance offered in the procedure indicates that RTM bias may be present when all three of the following statements are true (Bahar, 2010): • The evaluation method used is a simple before-after comparison and does not account for RTM, • Site selection bias is likely because sites were selected on the basis of poor safety record, and • Study data used in the before period includes the time period when the site had a poor safety record influencing the treatment decision. A-25 

If the statements in the list are true, then the following equation is used to compute the unbiased CMF. Equation 3 𝐶𝐶𝐶𝐶𝐶𝐶𝑢𝑢 = 𝐶𝐶𝐶𝐶𝐶𝐶𝑟𝑟 + 𝐶𝐶𝐶𝐶𝐶𝐶𝑟𝑟 × 𝑓𝑓𝑅𝑅𝑅𝑅𝑅𝑅 where CMFu = unbiased CMF; CMFr = reported CMF; and fRTM = RTM bias correction factor. The RTM bias correction factor has a value between 0.05 and 0.25. The exact value is judged by the person reviewing the study. A value of 0.05 is used when a larger portion of the population of sites was treated and many years of before period data were included in the study. A value of 0.25 is used where only a small proportion of sites were treated, those selected for treatment had a high accident frequency, and few before years of data were included in the study. The HSM procedure describes the correction technique based on a review of the data used to compute the CMF. The CMF could be computed from one or more sites. Thus, if the CMF is based on data for several sites, one representative bias correction factor is judged for the collective set of sites. The range of values used for the RTM bias correction factor is consistent with Elvik’s findings following his review of several papers on the topic of RTM bias (Elvik, 2008), as summarized in Table 9. He found that magnitude of the bias often varied from site-to-site based on site and local area characteristics. He concluded that it was not possible to reliably estimate the amount of RTM bias in a published CMF using a retrospective technique. This same conclusion was reached by Persaud and Lyon (2006). When the RTM bias correction technique was first presented to the TRB Task Force for the Development of the HSM, there were some comments offered about the ability to reliably correct for this bias in a retrospective manner (Harkey et al., 2006). The technique was subsequently reviewed by the TRB Task Force’s Research Subcommittee and the factors that influence the amount of RTM bias were examined (Dixon, 2006). The Subcommittee observed that the selection of the RTM bias correction factor was highly subjective. Their analysis indicated that an equation for this factor could be derived from the elements of the empirical Bayes (EB) estimation equations. The equation they developed indicated that the RTM bias correction factor was a function of the number of crashes in the before period, the over- dispersion parameter, the expected crash frequency for comparison sites, and the proportion of high-crash locations selected for treatment. A similar conclusion was reached by Lord and Kuo (2012) in relation to site-selection bias. 3.4 Correcting CMFs for Change in Traffic Volume The HSM procedure includes a technique for correcting published CMFs when they are believed to be biased because traffic volume change was not controlled in the original study. The guidance offered in the procedure indicates that traffic volume bias may occur in several study designs. However, it can only be corrected when the study reported the traffic volume change over time but did not control for this A-26 

change in the results of the original study. When this condition is satisfied, the following equation is used to compute the unbiased CMF. Equation 4 𝐶𝐶𝑀𝑀𝑀𝑀𝑟𝑟 𝐶𝐶𝐶𝐶𝐶𝐶𝑢𝑢 = 𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓 where CMFu = unbiased CMF; CMFr = reported CMF; and fflow = traffic volume bias correction factor (= volume after change / volume before change). The bias correction factor represents the ratio of the average volume during the time period with the treatment and the average volume during the time period without the treatment. The HSM procedure notes that the value for this factor can be below or above 1.0 (i.e., volume can decrease or increase over time). Elvik (2008) examined several publications documenting the development of CMFs. For those using a before-after study that did not control for changes in traffic volume, he found the bias in the CMF to average about 3 percent and with a maximum of 5 percent. The bias adjustment technique was reviewed by the TRB Task Force’s Research Subcommittee (Dixon, 2006). They noted that the amount of traffic volume bias was dependent on the trend between traffic volume and crash frequency. Safety performance functions (SPFs) that quantify this relationship consistently indicate the trend is nonlinear. However, the assumption inherent to the bias correction factor fflow is that this trend is linear. Traffic volume change can occur due to regional changes in population, activity centers, and road connectivity. In general, traffic volumes increase with time but they can sometimes decrease. Persaud and Lyon (2006) found that the treatment can influence traffic volume levels. They noted that left-turn accommodation, traffic signal installation, red-light camera installation, and conversion to roundabouts have all been shown to influence volume levels by making the treated site more (or less) attractive to motorists. 3.5 Correcting CMF Standard Error The HSM procedure includes a technique for correcting published CMF standard errors. This correction is needed when the standard error is believed to be biased because the study design did not account for important factors. The procedure computes an adjusted standard error by inflating the reported standard error by a “method correction factor” that is larger than 1.0 (the factor value of 1.0 is used when a perfectly controlled randomized trial is used). Guidance for selecting the appropriate correction factor is provided in Table 2. The manner in which the factor is used to compute the adjusted standard error is shown in Equation 2. The development of the technique for correcting the CMF standard errors was documented by iTrans (2004). They recognized that this technique was new to the profession, and that it was based on the expert judgment of the research team. They acknowledged that “To date, there has not been any explicit A-27 

guidance for correction of results of different methodologies ever published or provided” (p. 11, iTrans, 2004). The review of the literature did not reveal any subsequent evaluation of the accuracy of the adjusted standard error produced by this technique. 3.6 Summary of Findings This section summarizes the findings from the review of the literature regarding techniques used to improve CMF quality. These techniques are intended to remove a source of bias from a CMF or its standard error. Most sources of bias that are present in a CMF are due to confounding factors that were not controlled through study design or statistical analysis. A list of the typical confounding factors is provided in Table 8 and Table 10. The HSM procedure includes techniques for correcting CMFs for two sources of bias (i.e., RTM bias, and change in volume bias). Techniques for correcting CMFs for other sources of bias were not identified in the literature review. Elvik (2008) reviewed past studies to assess the types of bias present (if any) and the magnitude and direction of this bias. He found a wide range in the types and amount of bias in several studies. He concluded that the magnitude and direction of the bias often varies from site-to-site based on site characteristics. He also observed that study results may not be biased even though the researchers failed to control for one or more confounding factors. 4. PART III: TASKS 1 AND 2 FINDINGS AND RECOMMENDED RESEARCH FOR IMPROVING THE QUALITY OF CMFS IN THE HSM AND THE CLEARINGHOUSE This section is organized to summarize findings specific to each of the two quality assessment procedures. It should be noted before presenting those separate findings that repeatability of the two CMF quality assessment procedures, especially with regard to the more subjective aspects, is a key issue that needs to be evaluated. The findings are followed by recommended research that would address issues with quality assessment procedures in general and so should be relevant to CMFs in both the HSM and Clearinghouse repositories. 4.1 Findings Specific to the HSM Procedure This section summarizes the key issues identified for the HSM procedure and then elaborates on specific questions related to these issues. The HSM procedure uses MCFs to correct the standard error for confounding factors that were not likely addressed in the original research. These factors are identified in Table 6. They include 5 to 7 sources of bias, depending on study design. The HSM procedure uses Equation 2 to compute the adjusted standard error. This equation includes an MCF that reflects the presence of RTM as well as an RTM bias correction factor. Harkey et al. (2006) questioned why this one source of bias would require two adjustments to the standard error. They rationalized that the effect of RTM bias should only be addressed once in developing an adjusted standard error. A-28 

The HSM procedure requires knowledge of the standard error of the CMF. If the study report being reviewed does not report the standard error, the procedure provides techniques for estimating the standard error (using information in the study report). In some instances, a study report provides neither a standard error nor sufficient information to estimate. In this situation, the CMF quality cannot be assessed using the HSM procedure. Elaboration on specific questions related to the above issues, and other related ones, is provided below. Q1. Should the original reported CMFs be adjusted to remove bias or account for study method? It is unlikely that all sources of CMF bias can be identified through the post hoc review of a published study. Moreover, it is unlikely that all of the information needed to quantify a given level of bias would be found in the final report (Elvik, 2008). It could be argued that published CMFs that are believed to have notable bias should be discarded (i.e., not corrected). This argument recognizes the increasing number of higher-quality CMFs being produced since the publication of the HSM in 2010. It also recognizes the likelihood that many sources of bias are often difficult to identify, quantify, and remove from the CMF. Of note is RTM bias correction. Research by Elvik (2008) and by Persaud and Lyon (2006) indicates it is difficult to accurately quantify the magnitude of the RTM bias. A technique for correcting bias due to traffic-volume change could be developed through research. It would likely require knowledge of the traffic volume before and after the treatment. It would also likely require knowledge of the SPF-relationship between traffic volume and crash frequency. Q2. Should the original reported CMF standard errors be adjusted to remove bias or account for study method? No evidence was found in the literature that addresses the question of whether the adjusted standard error has less bias than the reported standard error. Research is needed to evaluate the HSM adjustment technique, and to either confirm that it improves the accuracy of the standard error estimate or to develop an improved technique. Two important uses of the CMF standard error are (1) to use it in the assessment of CMF quality, and (2) to use it in the calculation of a weighted average CMF when multiple CMFs are available. Regarding the first use, several researchers have indicated that the merit of the adjusted standard error is that it is a useful means of quantifying CMF quality (Harkey et al., 2006; Elvik, 2008). This use does not require that the adjusted standard error be represented as an unbiased estimate of the true standard error. However, research is needed to confirm the utility of this measure as an accurate and complete indication of CMF quality. The need for this research is discussed in item D in the section titled Recommended Research near the end of Part I. Regarding the second use, Elvik (p. 35, 2008) discusses the use of the method correction factor in the calculation of an overall average CMF. The factor is not used to compute an adjusted standard error. Rather, it is used with the reported standard error to compute an adjusted weight for the CMF. This weight is then used to compute the weighted average CMF, which is then offered as the overall average CMF. The need for this research is discussed in Appendix A1. Both of the aforementioned uses are satisfied without the conduct of research to evaluate the HSM technique and to either confirm its validity or develop an improved technique. Thus, this research is probably unneeded and the adjusted standard error should not be reported in the HSM (i.e., the unadjusted A-29 

standard error should be reported). However, if the HSM stakeholders desire to continue to report the adjusted standard error, then some research is needed to confirm that is accurate. Q3. Should the acceptance criteria be modified for rare crash types? For rare crash types (such as pedestrian-vehicle and bicycle-vehicle crashes), the standard errors of the CMFs would tend to be relatively large such that they would be less likely to satisfy the HSM procedure’s acceptance criteria. This trait may result in no CMFs (or very few CMFs) for some treatments being included in the HSM. Specification of the acceptance criteria and threshold (i.e., accepted CMFs with an adjusted standard error of 0.10 or less) is a policy decision for those entities responsible for HSM content. However, the Elvik procedure demonstrates that there are many factors that can affect study quality, some of which are not likely to be reflected in the adjusted standard error. Thus, using the Elvik procedure (or similar), sound research associated with rare crash types may still be designated as having high quality even if it has a relative large standard error. Q4. Should more than one CMF be presented in the HSM for a particular treatment? Two of the procedures reviewed in this document used an acceptance criterion that was intended to identify the one “best” CMF (provided it is of reasonably high quality). The other two procedures simply identified CMF quality or study quality, without using a rigorous acceptance criterion. One of these two procedures is the CMF Clearinghouse. It presents almost all of the crash-based CMFs found in the literature along with an estimate of CMF quality. The subsequent paragraphs outline the advantages of the “present all CMFs” and the “present best CMFs” options as applied to the HSM. Present All Option. Researchers that developed the HSM procedure considered both options (i.e., present best vs. present all) (iTrans, 2006). The advantage of the “present all” option was cited as it increases the likelihood that (1) the HSM is not “silent” on any treatment for which some study of treatment effect has been conducted, and (2) the HSM would be able to present sets of CMFs that collectively describe treatment effect on a crash-type or severity basis (even if CMF quality varied widely with a set). Where multiple CMFs are available, this option provides an indication of the variability in treatment effect. A disadvantage of the “present all” option is that it “places more responsibility on the HSM user to further investigate the CMFs provided in the HSM during the decision-making process” (iTrans, 2006). The iTrans researchers noted that the HSM is not intended to be a standard, so agencies are not obligated to implement the CMFs presented in the HSM. Thus, the presentation of many CMFs for a common treatment (with some indication of their quality) is consistent with the intended purpose of the HSM. Present Best Option. The benefit of the “present best” option is that it informs the analyst about the single best available CMF for each treatment (iTrans, 2006). The two procedures that use this option also use a “liberal” acceptance criterion to confirm that the CMF satisfies a minimum acceptable quality threshold. In this manner, a low quality CMF is not presented if it is found to be the best available CMF. One advantage of this option is that it avoids situations where the analyst is presented with two or more CMFs applicable to the site of interest. For example, it avoids the analyst’s dilemma of having to choose between a high quality CMF that applies to sites that are roughly similar to the subject site and a medium quality CMF that applies to sites that are the same as the subject site. Another advantage of this option is that it provides the analyst with stable guidance (i.e., a CMF that would be unlikely to change significantly as new studies are conducted). A-30 

A disadvantage of this option is that strict application of this criterion could result in accepting only some CMFs from a given study (e.g., accepting the CMF describing treatment effect on fatal and injury crashes, but not accepting the CMF describing treatment effect on property-damage-only crashes). There is concern that, in the absence of a desired CMF for a specified crash type or severity, analysts may inappropriately use the most nearly related CMF in the HSM that is, in fact, applicable to different crash type or severity (Dixon et al., 2006). Which Option Should be Used. The decision regarding whether to present the best CMF or all CMFs is a policy decision for those entities responsible for the HSM content. This decision should reflect the purpose of the HSM, the aforementioned advantages, and the information needs of practitioners. Q5. Is uncertainty in the CMFs adequately represented in the HSM? Uncertainty in a CMF is represented in the HSM by reporting the associated adjusted standard error. The adjusted standard error is based on the standard error obtained from the original study report but inflated for sources of likely bias (Bahar 2010). The bias is identified by a review of the report. The adjusted standard error is offered as a best-estimate of the true standard error; however, the validity of this claim has not been documented. Elvik (2008) examined the HSM procedure (and the adjusted standard error that it produced) to evaluate its ability to accurately describe study quality. He observed that the adjusted standard error was an attractive means of defining the “quality-adjusted” statistical weight of a CMF, as may be used to compute a weighted-average CMF when multiple CMFs are available for a common treatment. He did not address the question of whether the adjusted standard error was an accurate estimate of the true standard error. Hauer et al. (2012) argue that CMF standard deviation is an important indicator of CMF quality, perhaps more so than CMF standard error. This statistic can be computed when multiple CMFs are available for a common treatment. A large standard deviation suggests that the CMF may vary systematically with one or more site characteristics. Thus, a CMF with a large standard deviation may indicate that the effect of a treatment at any one location may more uncertain. Hauer et al. suggest that CMF standard deviation should be reported in the HSM, when it is available. Based on the previous discussion, it is likely that the standard error should be reported in the HSM. Research is needed to determine whether the adjusted standard error is a more accurate estimate of the true standard error. If it is, then it should be reported in the HSM. When multiple CMFs are available, the CMF standard deviation should also be reported. Q6. Should CMFs from multiple studies for the same treatment be combined and if so, how? From a statistical perspective, combining results using an averaging technique can produce a more accurate estimate of the true mean treatment effect. However, this improvement in accuracy is only obtained when the CMFs being averaged have the same underlying true mean value. In other words, if there is evidence that the CMFs being considered have the same true mean (i.e., the characteristics of the sites studied are the same), then the CMFs should be combined to produce a more reliable estimate of treatment effect. Techniques are described in Appendix A1 for determining whether the CMFs are likely to have the same true mean (i.e., that they do not vary because of systematic differences). 4.2 Findings Specific to the CMF Clearinghouse Procedure Factors used in the assessment process. The CMF Clearinghouse procedure identifies five factors to be considered in the assessment of CMF quality; they are listed in Table 3. One of the factors is “data source.” The procedure gives a higher rating to a CMF that is derived using data from a diversity of states A-31 

that represent different geographic regions. However, several researchers have found that CMFs can vary among states and regions due to differences in their site characteristics and driver behaviors (Hauer et al., 2012; Elvik, 2015). It follows that CMF quality may be adversely affected if the CMF is based on sites from different regions unless the researchers explicitly evaluate the potential for these differences and account for them in their results. One approach for this type of evaluation is to separately quantify a CMF for each unique region and then evaluate them collectively for systematic regional differences. Techniques for pursuing this approach are described in Appendix A1. A similar issue is related to sample size and disaggregate analysis by crash type, crash severity, or individual site characteristics. In general, larger samples are associated with smaller standard errors and more reliable results. By nature, disaggregate analyses are based on smaller subsets of the total sample, and receive relatively lower scores than aggregate CMFs due to smaller sample sizes. In some cases, however, a disaggregate analysis can help to eliminate noise by focusing on select subsets. For example, consider an aggregate analysis of a treatment installed in both rural and urban areas. If there is a different treatment effect in rural and urban areas, then the standard error of the aggregate CMF may be smaller than both of the individual disaggregate CMFs by area type. In this case, it may be appropriate to modify the factor for sample size so as not to penalize the smaller disaggregate samples. Consideration of Sample Size and Statistical Significance. Two of the factors considered by the CMF Clearinghouse procedure are sample size and statistical significance. As indicated in Table 3, sample size is described by the number of crashes included in the data used to produce the CMF. Mathematically, the standard error is a function of sample size. By examination of Equation 1, it follows that sample size considerations (directly and indirectly) represent over 40 percent of a CMFs quality score. In contrast, the Elvik procedure considers sample size indirectly through factors associated with “statistical association” (e.g., significance of effect) which represents only 12 percent of the quality score, as shown by the weights in Table 5. Considering Advances in Cross-Sectional Modeling Approaches. There have been several advancements to cross-sectional modeling, including the use of propensity score matching to minimize the potential for confounding effects when comparing sites with and without a given feature. Based on the current procedure, it is not possible for a CMF from a well-designed cross-sectional study to receive a 5- star rating. Cross-sectional methods vary considerably with respect to the level of rigor and ability to control for confounding factors. Some of these methods may even be considered at the same level as a rigorous before-after method. 4.3 Recommended Research This subsection outlines research topics that could be undertaken to provide a stronger basis for the assessment of the quality of both CMFs and CMF functions, and for both the HSM and the CMF Clearinghouse. The findings from these topics are intended to make the quality assessment results more reliable. The distinction between the two repositories is at best fuzzy at the moment, not knowing whether HSM CMFs will in the future reside in the HSM or the Clearinghouse, or both. Thus, the recommended research could, at least for the time being, be regarded as pertinent to both repositories. The recommended research topics are not listed in order of priority. A. Determine the key factors influencing CMF quality. Use the list of factors in Table 8 and Table 10 as a starting point to identify the key factors needed to evaluate CMF quality. The list of factors should be collectively capable of evaluating the quality of a CMF from one study, as well as a set of CMFs from multiple studies of a common treatment. Also, the list of factors should include components for evaluating CMFs and CMF functions. A-32 

Use the experience of the research team to define the levels associated with each factor, similar to the levels identified in Table 5 and Table 6. Assemble a panel of experts having experience in evaluating the safety effects of treatments. Request that the panel members work independently to associate a score with each level and a relative weight for each factor. They may also add a factor if it is not on the list. The score should range from 0 to 1 and the weights for all factors should add to 1.0 (similar to the approach used by Elvik, 2008). The expert would use a weight of 0.0 to identify a factor that should not be considered. Compile the results from the safety experts and disseminate the collective results back to the experts for comment. Convene a conference call with the experts to discuss the results, clarify any misunderstandings, and facilitate a discussion to resolve differences of opinion where possible. Use the feedback to determine the recommended list of factors, relative weight for each factor, and score for each factor level. B. Evaluate the repeatability of each key factor. This research would identify a small number of reviewers and have them evaluate one factor using a common set of published reports that describe the development of one or more CMFs. Each reviewer would judge the score associated with the factor (using the values identified in item B above). The measure of interest in this research is the variability in the scores estimated by the reviewers. A smaller variation in results among reviewers would indicate a more reliable procedure. C. Develop a revised CMF quality assessment procedure. Use the findings from items A and B to develop a recommended procedure for assessing CMF quality. The procedure should be applicable to studies that produce a CMF and studies that produce a CMFunction. It should also be able to evaluate the CMF from a single study as well as a set of CMFs from multiple studies. One component of this research should be to determine whether it is feasible to fully describe CMF quality using an adjusted standard error (where a method correction factor is used to inflate the reported standard error). If it is feasible, use the findings to develop the appropriate method correction factors and standard error adjustment process. D. Evaluate the repeatability of the revised procedure. This research would identify a small number of reviewers and have them apply the revised HSM and Clearinghouse procedures to the same common set of published reports that were used in A. Each reviewer would estimate the quality score for each CMF. The measure of interest in this research is the variability in the quality scores. A smaller variation in results among reviewers would indicate a more reliable procedure. E. Facilitate stakeholder discussion to determine the number of CMFs presented for a given situation and the mechanism for presenting the CMFs. Facilitate discussion by the appropriate stakeholders responsible to identify the options for CMF presentation (i.e., present all, present best) in Part D of the HSM. This discussion should also determine the appropriate mechanism for presenting the CMFs for Part D (e.g., HSM Part D and CMF Clearinghouse, CMF Clearinghouse only, CMF Clearinghouse with indicator for HSM-approved CMFs, etc.) Present the findings from Tasks 1, 2 and 3 to inform the discussion. F. Determine appropriate acceptance criterion for the HSM. If a “present best” option is requested by the stakeholders for the HSM at the conclusion of item E, apply the refined HSM procedure from item E to a subset of the CMFs considered for inclusion in the HSM. Use the findings from this application to determine the minimum quality score value that produces the same rate of acceptance as obtained when the HSM procedure is used. Assess cases where the existing HSM procedure and the revised procedure differ in the specific CMFs considered acceptable. Recommend a minimum threshold A-33 

score that can be used to identify CMFs suitable for inclusion. Finalize the refined HSM procedure using the findings from this research. 5. REFERENCES Amundsen, A., and R. Elvik. “Effects on Road Safety of New Urban Arterial Roads.” Accident Analysis and Prevention, Vol. 36, pp. 115-123. Bahar, G. (2010). Transportation Research Circular E-C142: Methodology for the Development and Inclusion of Crash Modification Factors in the First Edition of the Highway Safety Manual. Transportation Research Board, Washington, D.C. Borenstein, M., L. Hedges, J. Higgins, and H. Rothstein. (2009). Introduction to Meta-Analysis. John Wiley and Sons, New York. Carter, D. (2011). FHWA Crash Modification Factors Clearinghouse - Technical Review Process. Internal memorandum provided by D. Carter on November 20, 2015. Carter, D. (2012). Detailed Thresholds for CMF Clearinghouse Star Quality Rating Process. Internal memorandum provided by D. Carter on November 20, 2015. Carter, D., R., Srinivasan, F. Gross, and F. Council. (2012). Recommended Protocols for Developing Crash Modification Factors. National Cooperative Highway Research Program, Transportation Research Board, Washington, D.C. Dixon, K. (2006). Research Subcommittee Review and Recommendations for Accident Modification Factors and Decision Rule (as proposed for NCHRP Project 17-27). Memorandum to Chuck Niessner, Senior Program Officer, NCHRP, dated April, 19, 2006. Elvik, R. (2005). Introductory Guide to Systematic Reviews and Meta-Analysis. Transportation Research Record: Journal of the Transportation Research Board, No. 1908, Transportation Research Board of the National Academies, Washington, DC, pp. 230–235. Elvik, R. (2008). Making Sense of Road Safety Evaluation. TOI Report 984/2008. Institute of Transport Economics, Oslo, Norway. Elvik, R. (2011). “Publication Bias and Time-Trend Bias in Meta-Analysis of Bicycle Helmet Efficacy. A Reanalysis of Attewell, Glase and McFadden.” Accident Analysis and Prevention, Vol. 43, pp. 1245-1251. Elvik, R. (2015). “Methodological Guidelines for Developing Accident Modification Functions.” Accident Analysis and Prevention, Vol. 80, pp. 26-36. Fleiss, J.L. (1973). Statistical Methods for Rates and Proportions. John Wiley & Sons, New York. Griffin, L. and R. Flowers. (1997). A Discussion of Six Procedures for Evaluating Highway Safety Projects. Report No. FHWA-RD-99-040. Texas Transportation Institute, College Station, Texas. Harkey, D., F. Council, B. Persaud, R. Srinivasan, and K. Eccles. (2006). Review of 17-27 Procedure Used for Developing AMF Estimates and Standard Errors. Memorandum to Ron Pfefer, Chair, Task Force for the Development of a Highway Safety Manual, dated January 12, 2006. Harkey, D., R. Srinivasan, J. Baek, F. Council, K. Eccles, N. Lefler, F. Gross, B. Persaud, C. Lyon, E. Hauer, and J. Bonneson. (2008). NCHRP Report 617: Accident Modification Factors for Traffic Engineering and ITS Improvements. National Cooperative Highway Research Association, Transportation Research Board, Washington, DC. Hauer, E. (1992). “A Note on Three Estimators of Safety Effect.” Traffic Engineering & Control. June 1992, pp. 388-393. Hauer, E., J. Bonneson, F. Council, R. Srinivasan, and C. Zegeer. (2012). Crash Modification Factors: Foundational Issues. Transportation Research Record: Journal of the Transportation Research Board, No. 2279. TRB, The National Academies, Washington, D.C., pp. 67-74. HSM (2010). Highway Safety Manual, 1st Edition. American Association of State Highway and Transportation Officials, Washington, D.C. HSRC. (2015a). Working Paper No. 2: Description of Homogeneity Test of Multiple CMFs. NCHRP Project 17-63, Highway Safety Research Center (HSRC), University of North Carolina, North Carolina, September 30, 2015. A-34 

HSRC. (2015b). Working Paper No. 3: Local Adjustment of CMFs Based on Crash Distribution. NCHRP Project 17-63, Highway Safety Research Center (HSRC), University of North Carolina, North Carolina, September 30, 2015. iTrans (2006). Task 8A: Develop Decision Rule – AMF Acceptance Criteria. Working paper from NCHRP Project 17-27, Parts I and II of the Highway Safety Manual, iTrans Consulting Ltd. iTrans (2004). Quarterly Progress Report. Project 17-27, Parts I and II of the Highway Safety Manual, iTrans Consulting Ltd. Lord, D., and P-F. Kuo. (2012). “Examining the Effects of Site Selection Criteria for Evaluating the Effectiveness of Traffic Safety Countermeasures.” Accident Analysis & Prevention, Vol. 47, pp. 52-63. OECD (2012). Sharing Road Safety: Developing an International Framework for Crash Modification Functions. OECD Publishing. http://dx.doi.org/10.1787/9789282103760-en. Washington, S., D. Lord, and B. Persaud. (2009). Use of Expert Panels in Highway Safety: Critique. Transportation Research Record, Journal of the Transportation Research Board 2001, pp. 101–107. Persaud, B., and C. Lyon. (2006). “Empirical Bayes Before-After Safety Studies: Lessons Learned from Two Decades of Experience and Future Directions.” Accident Analysis & Prevention. Vol. 39, pp. 546-555. Woolf, G. (1955). “On Estimating the Relationship between Blood Group and Disease.” Annals of Human Genetics. Vol., 19, p. 251-253. A-35 

Appendix A.1 Estimating Safety Effect of Using Results from Multiple Studies A-36 

Table of Contents 1. Introduction ......................................................................................................................................... A-38 2. Determining if Systematic Variation Is Present .................................................................................. A-38 3. Combining CMFs When Systematic Variation Is Not Present ........................................................... A-40 4. Evaluating CMFs When Systematic Variation Is Present................................................................... A-43 5. Summary of Findings .......................................................................................................................... A-44 5.1 General Findings .......................................................................................................................... A-44 5.2 Recommended Research .............................................................................................................. A-45 6. References ........................................................................................................................................... A-45 A-37 

1. INTRODUCTION A review of the literature on a given safety treatment will sometimes reveal the existence of multiple studies of the treatment with each study reporting an associated CMF. The CMFs will often not be in exact agreement. Differences in CMFs can be due to random variation, systematic differences in circumstances among the study locations, or both. Systematic differences can result from the treatment effect being influenced by the location’s road geometry, traffic control, traffic characteristics, topography, population demographics, or the driving environment. Differences in CMF should be expected due to the aforementioned reasons. The variability in the set of CMFs can provide some insight as to the certainty (or uncertainty) about the treatment’s effect on safety. If the variability is large, there may be some underlying site characteristics that are influencing the CMF value such that doubt may be cast on the treatment’s transferability to a different location (Hauer et al, 2012). In this situation, it may be prudent to restrict the treatment’s application to very similar sites, or quantify the systematic effect using a CMF function. This appendix provides an overview of techniques described in the literature for estimating the safety effect of a treatment using the CMFs from multiple studies of the same treatment. The first section identifies alternative techniques for determining whether systematic variation is present in the collective set of CMFs. The second section describes alternative techniques for computing an overall average CMF when the presence of systematic variation is unlikely. The third section provides an overview of the process for evaluating CMFs when the presence of systematic variation is likely. The last section summarizes the findings from this review. The discussion in this section is intended to provide a brief overview of key issues related to estimating treatment effect when multiple CMFs are available. The science of meta-analysis deals explicitly with this topic (Borenstein et al., 2009). It has been extended to safety evaluation by Elvik (2005, 2011). 2. DETERMINING IF SYSTEMATIC VARIATION IS PRESENT This section describes alternative statistical techniques that can be used to determine whether the reported CMFs for a common treatment have differences so large as to be likely caused by underlying differences in characteristics among the locations studied. If the differences in CMFs are small, then they are likely due to random variation and the CMFs can be combined into a single CMF that represents the best estimate of treatment effect. If the differences are large, then they are likely due to systematic influences that affect the treatment’s effectiveness at a given location. In this latter case, further investigation of location characteristics may lead to the identification of some of these influences. In turn, these findings may lead to the development of (1) a set of unique CMFs for specified characteristics, or (2) a CMF function that includes variables that quantify the effect of different characteristics on the predicted CMF. Homogeneity Test. One technique is referred to as the homogeneity test by Griffin and Flowers (1997). It produces a test statistic that is chi-square distributed. This statistic is used to test the null hypothesis that the CMF values are equal. More detail on this test is provided by Woolf (1955) and by Fleiss (1973). A-38 

To apply the homogeneity test, two or more CMF values (and their associated standard error) are needed. The following steps describe the calculation process. Step 1. Compute Weight. The first step is to compute the statistical weight to be associated with each CMF. This weight is computed using the following equation. Equation 1 2  CMFi  wi =     si  where wi = weight of CMF observation i; CMFi = value of CMF observation i; and si = standard error of CMF observation i. Step 2. Compute Log. The second step is to compute the natural log of the CMF value. This calculation is shown by the following equation. Equation 2 Li = ln(CMFi ) where Li = natural log of CMF observation i. Step 3. Compute Average Log. The third step is to compute the weighted average value of L. This average is computed using the following equation. Equation 3 ∑w L i i L= i ∑w i i A-39 

where L = weighted average value of L. Step 4. Compute Chi-Square Statistic. The fourth step is to compute the chi-square value for each observation. This calculation is shown by the following equation. Equation 4 χ i2 = wi ( Li − L) 2 where χ i2 = chi-square value of CMF observation i. Step 5. Check Results. The last step is to add the chi-square values and compare this result with the chi-square distribution for n−1 degrees of freedom, where n is the number of CMF observations. If the sum of the chi-square values is larger than the test value from the distribution, then it is unlikely that the variation is solely due to random sources (i.e., some systematic variation in CMF values may be present). The chi-square value obtained from Equation 4 is also called the “Q” test statistic in some research reports and statistics textbooks (Amundsen and Elvik, 2004). Borenstein l2 Statistic. The l2 test statistic is used in meta-analysis to identify the contribution of systematic variation to the overall variation in CMFs (Borenstein et al., 2009). The contribution is stated as a percentage. The guidance offered by Elvik (2015) is that a CMF function should be considered when the systematic variation explains more than 50 percent of the overall variation. 3. COMBINING CMFS WHEN SYSTEMATIC VARIATION IS NOT PRESENT This section describes alternative techniques for combining the CMFs from multiple studies of the same treatment. The benefit of combining the results of several studies is that the combined CMF represents a more accurate estimate of the treatment’s effect on safety. To be candidates for combination, a few requirements should be satisfied. One requirement is that the treatment and site conditions associated with each study should be similar (e.g., same treatment design, same road design, similar traffic volume, etc.) (Bahar, 2010). Similarity among studies in driver behavior and vehicle design (e.g., all sites in North America, all studies conducted in last 25 years, etc.) should also be considered (Harkey et al., 2006). Judgment is required to assess the degree to which this requirement is satisfied. A second requirement is that there is evidence that the variability in the CMFs from multiple studies is likely due to random (i.e., unexplained) sources. Either of the tests described in the previous section can be used to provide this evidence. A-40 

Additional consideration (and possible adjustment) is often extended to publication bias and other sources of bias, as identified in Table 10 of Appendix A. A technique is available for detecting publication bias (Elvik 2011). When it is detected, the missing CMF results can be replicated using fictitious CMFs. The remainder of this section describes three techniques for computing the overall average CMF. Each of the techniques uses a weighted average technique, where the weight of each CMF is a function of its standard error. Elvik (2008) posed that the weight should be computed using both standard error and information about the study quality. In his review of the HSM procedure, he suggested that the weight could be calculated as the reciprocal of the square of the standard error multiplied by the method correction factor (i.e., w = 1/[s x MCF]2). However, he concluded his research by suggesting that the MCF could be replaced the study quality score. He also acknowledged that more research is needed to verify that this approach provides the best estimate of overall treatment effect. HSM Procedure. The following equation was used in Step 6 of the HSM procedure to compute a combined CMF. Equation 5 ∑ wa,i × CMFi i CMFv = ∑ wa,i i with Equation 6 1 wa ,i = s a ,i 2 where CMF v = overall average CMF based on variance weighting; CMFi = value of CMF observation i; wa,i = weight of CMF observation i based on adjusted standard error; and sa,i = adjusted standard error of CMF observation i. Logistic Procedure. Griffin and Flowers (1997) offer the following equation for computing the combined average CMF. A-41 

Equation 7 CMFL = e L where CMF L = overall average CMF based on log transform; and L = weighted average value of L (computed using Equation 3). Hauer (1992) used simulation to evaluate Equation 7 and two other estimators. However, he did not evaluate Equation 5. He found that Equation 7 tended to produce a biased estimate of safety effect. The bias ranged from -0.08 to +0.1, with the value dependent on the number of crashes occurring in the “before” period and the CMF value. Log-Normal Procedure. More recently, researchers for NCHRP Project 17-63 (HSRC, 2015a) evaluated Equation 5 and Equation 7. They found that the estimate from Equation 5 is 20 to 30 percent below the true value. The estimate from Equation 7 is 5 to 10 percent below the true value. The error approaches zero as the crash count increases. They also evaluated the following equation and found that it produced an unbiased estimate of the true average CMF. Equation 8 CMFt = e L × f c with Equation 9  2  ∑ wi ( Li − L)  f c = exp 0.574 i   ∑ wi   i  where CMFt = overall average CMF based on adjusted log transform; fc = correction factor; and all other variables as previously defined. A-42 

When a CMF is transformed by the natural log function, the transformed value is asymptotic to the normal distribution when the CMF is based on a large number of crashes (Griffin and Flowers, 1997). It follows that CMF data are asymptotic to the lognormal distribution and the true correction factor fc has a value of exp[0.5 σ2ln(CMF)], where σ2ln(CMF) is the variance of the log of the CMFs. This variance is computed using the ratio shown in brackets in Equation 9. Theoretically, this variance is multiplied by the value of “0.5”; however, the researchers found the best fit using “0.574”. 4. EVALUATING CMFS WHEN SYSTEMATIC VARIATION IS PRESENT This section describes alternative techniques for evaluating multiple CMFs when systematic variation is present in the collective set of CMFs. Random Effects. This technique can be used to evaluate multiple CMFs when systematic variation is most likely present but the source of this variation is not evident. The next subsection describes a technique that can be used when the source of the variation is evident. The technique uses a random effects modeling approach to inflate the standard error of each CMF in an amount proportional to the additional systematic variation in the CMFs (Amundsen and Elvik, 2004). Equation 5 is used to compute the overall average CMF; however, the adjusted weight is computed using the following equation. Equation 10 −1   s 2  wa ,i =   i  + σ s2   CMF   i  with, Equation 11 ∑ χ i2 − (n − 1) σ s2 = i c Equation 12 ∑ wi2 c = ∑ wi − i i ∑ wi i A-43 

where wa,i = weight of CMF observation i based on adjusted standard error; CMFi = value of CMF observation i; si = standard error of CMF observation i; σ s2 = systematic variance component; χ i2 = chi-square value of CMF observation i (computed using Equation 4); wi = weight of CMF observation i (computed using Equation 1); n = number of CMF observations; and c = equivalent sample size. CMF Function. This technique can be used to evaluate multiple CMFs when systematic variation is likely present and the source of the variation is evident. A function can be developed using regression techniques to quantify the relationship between the CMFs and one or more site characteristics. Guidelines for developing CMF functions are described by Elvik (2015). These guidelines are being updated by the researchers for NCHRP Project 17-63. It should be noted that if publication bias is detected in the CMFs, the technique of creating the missing CMF results may present issues when developing a CMF function (Elvik, 2015). This restriction stems from the lack of descriptive information (e.g., site geometry) associated with the fictitious results. As a result, Elvik (2015) advises that CMF function should not be developed if publication bias is indicated. The procedures described by Elvik (2015) describe the development of a CMF function from CMFs. The researchers for NCHRP Project 17-63 developed a procedure for developing a CMF function from other CMF functions (HSRC, 2015b). The literature review did not identify any other procedures for this purpose. 5. SUMMARY OF FINDINGS This section summarizes the findings from the review of the literature regarding the estimation of safety effect using the results from multiple studies. The summary describes the key issues and the research that could be conducted to address these issues. 5.1 General Findings The process described in the HSM procedure for combining CMFs could be improved through the use of (1) a statistically-based technique for determining whether the variability in the CMFs from multiple A-44 

studies is likely due to random (i.e., unexplainable) sources, and (2) a technique for computing the overall average CMF does not produce a biased estimate. Should the CMFs from studies of the similar treatments in similar locations be combined to obtain a single reliable CMF estimate? There appears to be ample evidence in the road safety literature of the need to combine CMFs and the utility of the information that is derived from this combination. In particular, the calculation of an overall average CMF improves CMF reliability by reducing the standard error of the CMF. Also, the derivation of a CMF function is likely to improve CMF reliability and transferability (Hauer et al, 2012; OECD 2012). As documented in this Appendix, there is a reasonably large body of statistical science that can be used to guide the decision regarding whether and how to combine CMFs. 5.2 Recommended Research This subsection outlines research topics that could be undertaken to provide a stronger basis for the estimation of safety effect using the results from multiple studies. The findings from these topics are intended to make the combined CMFs more reliable. The topics are not listed in order of priority. A. Evaluate the use of a CMF quality score when computing the statistical weight of a CMF. This research should extend the research of Elvik (2008) to the use of a quality-adjusted statistical weight in the calculation of an overall average CMF or in the use of weighted-regression to develop a CMF function. The research should evaluate alternative methods of using a CMF quality score when computing the weight value. The research should also assess the impact of the quality-adjusted weight on the accuracy of the computed CMF. B. Evaluate the procedures from NCHRP Project 17-63 for a wide range of conditions. NCHRP Project 17-63 has developed procedures for (1) combining CMFs to estimate an overall average CMF, (2) combining CMF functions, and (3) developing CMF functions. Additional research is needed to test and evaluate these procedures using CMFs for a wide range of conditions. The findings from this application should be used to confirm the accuracy of these new procedures and identify refinements to them. 6. REFERENCES Amundsen, A., and R. Elvik. “Effects on Road Safety of New Urban Arterial Roads.” Accident Analysis and Prevention, Vol. 36, pp. 115-123. Bahar, G. (2010). Methodology for the Development and Inclusion of Crash Modification Factors in the First Edition of the Highway Safety Manual. TR Circular E-C142. Transportation Research Board, Washington, D.C. Borenstein, M., L. Hedges, J. Higgins, and H. Rothstein. (2009). Introduction to Meta-Analysis. John Wiley and Sons, New York. Elvik, R. (2005). “Introductory Guide to Systematic Reviews and Meta-Analysis.” Transportation Research Record 1908. TRB, The National Academies, Washington, D.C., pp. 230-235. Elvik, R. (2008). Making Sense of Road Safety Evaluation. TOI Report 984/2008. Institute of Transport Economics, Oslo, Norway. A-45 

Elvik, R. (2011). “Publication Bias and Time-Trend Bias in Meta-Analysis of Bicycle Helmet Efficacy. A Reanalysis of Attewell, Glase and McFadden.” Accident Analysis and Prevention, Vol. 43, pp. 1245- 1251. Fleiss, J.L. (1973). Statistical Methods for Rates and Proportions. John Wiley & Sons, New York. Griffin, L. and R. Flowers. (1997). A Discussion of Six Procedures for Evaluating Highway Safety Projects. Report No. FHWA-RD-99-040. Texas Transportation Institute, College Station, Texas. Harkey, D., F. Council, B. Persaud, R. Srinivasan, and K. Eccles. (2006). Review of 17-27 Procedure Used for Developing AMF Estimates and Standard Errors. Memorandum to Ron Pfefer, Chair, Task Force for the Development of a Highway Safety Manual, dated January 12, 2006. Hauer, E. (1992). “A Note on Three Estimators of Safety Effect.” Traffic Engineering & Control. June 1992, pp. 388-393. Hauer, E., J. Bonneson, F. Council, R. Srinivasan, and C. Zegeer. (2012). “Crash Modification Factors: Foundational Issues.” Transportation Research Record 2279. TRB, The National Academies, Washington, D.C., pp. 67-74. HSRC. (2015a). Working Paper No. 2: Description of Homogeneity Test of Multiple CMFs. NCHRP Project 17-63, Highway Safety Research Center (HSRC), University of North Carolina, North Carolina, September 30, 2015. HSRC. (2015b). Working Paper No. 3: Local Adjustment of CMFs Based on Crash Distribution. NCHRP Project 17-63, Highway Safety Research Center (HSRC), University of North Carolina, North Carolina, September 30, 2015. OECD (2012). Sharing Road Safety: Developing an International Framework for Crash Modification Functions. OECD Publishing. http://dx.doi.org/10.1787/9789282103760-en Woolf, G. (1955). “On Estimating the Relationship between Blood Group and Disease.” Annals of Human Genetics. Vol., 19, p. 251-253 A-46