Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
7  Review of CMF Inclusion Criteria and Star Rating System in the CMF Clearinghouse One of the initial activities was to conduct a critical review of the inclusion criteria used for CMFs in the HSM and, at the same time, conduct a critical review of the star rating system used by the CMF Clearinghouse to evaluate CMF quality. The findings from this review were used to guide the development of a recommended procedure for (1) evaluating the quality of CMFs and (2) determining which CMFs are appropriate for use with the HSM. These activities were conducted simultaneously, and a report was developed to document the findings. This report is provided as Appendix A. Appendix A.1 is a supplement to Appendix A and provides an overview of techniques described in the literature for estimating the safety effect of a treatment by using CMFs from multiple studies of the same treatment. A summary of the findings is provided in the following subsections. 2.1 CMF Quality Many investment decisions are made on the basis of CMFs reported in the literature. Each decision is often based on whether the proposed treatment will improve safety and, if it does, whether the safety benefits are sufficient to justify the cost of implementing the treatment. As a result, the correctness of the decision depends on the accuracy of the CMF. The quality ratings in the CMF Clearinghouse are a reminder that some CMFs are not suffi ciently accurate to be the basis for important decisions. In fact, these ratings indicate that there is a wide range of CMF quality reflected in the literature. CMF quality denotes the extent to which (1) the associated treatment is sufficiently well described to be reproduced by others, (2) the site characteristics are sufficiently well described that intended application sites can be correctly identified by others, and (3) the reported CMF is free of bias and sufficiently precise for decisions regarding investment in road infrastructure. 2.1.1 Review of Existing Procedures for Assessing CMF Quality A review of the international literature by Elvik (2008) indicated the existence of several pro cedures for assessing the quality of a study. The discussion of each procedure included the evaluation of various factors describing the quality of the report, data collection procedures, the sampling procedure, control for confounding factors, and statistical analysis. On the basis of his review, Elvik found that the assessment of CMF quality was highly subjective and difficult to repeat between reviewers. Four procedures for assessing CMF quality were identified as applying to road safety studies and as having been developed after the aforementioned literature review by Elvik: ⢠NCHRP Project 17Â25, âCrash Reduction Factors for Traffic Engineering and ITS Improve mentsâ (https://apps.trb.org/cmsfeed/TRBNetProjectDisplay.asp?ProjectID=451); C H A P T E R 2
8 Crash Modification Factors in the Highway Safety Manual: A Review ⢠HSM procedure; ⢠CMF Clearinghouse procedure; and ⢠Elvik (2008) procedure. These procedures are briefly described in Appendix A. 2.1.2 Factors Used to Assess CMF Quality Study quality is considered in all four of the procedures mentioned above. Collectively, these procedures include consideration of 36 factors pertaining to study quality (e.g., sample size, standard error, data quality). These factors are listed in Appendix A. 2.1.3 Summary of Findings Four procedures for assessing CMF quality were reviewed. All of these procedures indicate the need to review the study documentation for the purpose of identifying the likely presence of factors that could bias the results, for example, regression to the mean (RTM). Three procedures (NCHRP Project 17Â25, HSM, and CMF Clearinghouse) are focused on assessing the quality of a CMF. The guidelines for applying these procedures are most clear regarding the evaluation of a CMF (as opposed to CMFunction). The Elvik (2008) procedure is more broadly focused on assessing study quality. The HSM procedure is unique among the four procedures in that it goes beyond quality assessment. Specifically, it includes a technique for combining CMFs from multiple studies of a common treatment. It also includes a technique for correcting the published results to reduce any perceived bias. All four procedures require the original study to have a high level of documentation. This documentation provides the details that are needed to assess the studyâs quality. 2.2 Corrections to Improve CMF Quality The literature review also examined techniques used in the post hoc correction of published study results. These techniques are used to improve the quality of the published CMF. In contrast, the procedures described in Section 2.1 are used to assess the quality of a published CMF. 2.2.1 Background There are many possible factors that could bias a CMF value. Most sources of bias are due to confounding factors that are not controlled for through the studyâs design or statistical analysis (e.g., RTM, change in traffic volume, crash migration). Sixteen confounding factors are identi fied in Appendix A. In the context of a post hoc assessment of CMF quality, a reviewer may believe that one or more sources of bias are present in a reported CMF on the basis of a determination of whether a confounding factor was controlled for. In this case, CMF study quality may be improved by removing the bias from the CMF, its standard error, or both. 2.2.2 Correcting CMFs Techniques for correcting CMFs for RTM bias and bias due to change in traffic volume were used to develop the CMFs listed in Part D of the HSM. Techniques were also used to
Review of CMF Inclusion Criteria and Star Rating System in the CMF Clearinghouse 9 correct the CMF standard errors listed in Part D of the HSM. Techniques were not identified in the literature for the post hoc correction of bias due to most of the other 14 confounding factors. 2.2.3 Summary of Findings Elvik (2008) reviewed past studies to assess the types of bias present (if any) and the magnitude and direction of this bias. He concluded that the magnitude and direction of the bias often vary from site to site according to the site characteristics. He also observed that study results may not be biased, even though the researchers failed to control for one or more confounding factors. Most sources of bias present in a CMF are due to confounding factors that were not controlled for through study design or statistical analysis. The HSM procedure includes techniques for correcting CMFs for two sources of bias. Techniques for correcting CMFs for other sources of bias were not identified in the literature review. 2.3 Recommended Research for Improving the CMF Information in the HSM This section summarizes the recommended research topics that were identified to provide a stronger basis for the assessment of the quality of both CMFs and CMFunctions. More infor mation about each topic is provided in Appendix A. The recommended research topics are listed in the order in which they should be undertaken: A. Determine the key factors influencing CMF quality. Identify the key factors needed to evaluate CMF quality and develop a quantitative system for scoring the degree to which the factor has been addressed by the researchers who produced the CMF. B. Evaluate the repeatability of each key factor. This research would evaluate each factor identified in Item A for the purpose of assessing the repeatability (between reviewers) of the CMF scoring system developed for Item A. The measure of interest in this research is the variability in the scores estimated by the reviewers of a common set of published CMFs. A smaller variation in results between reviewers would indicate a more reliable procedure. C. Develop a revised CMF quality assessment procedure. Use the findings from Items A and B to develop a recommended procedure for assessing overall CMF quality. The procedure should include a numeric weighting system for each factor that produces a score that describes the quality of the CMF relative to other CMFs. D. Evaluate the repeatability of the revised procedure. This research would apply the proce dure developed in Item C to the same set of published CMFs used in Item B. Each reviewer would estimate the quality score for each CMF. The measure of interest in this research is the variability in quality scores between reviewers. A smaller variation in results between reviewers would indicate a more reliable procedure. Refine the procedure on the basis of the findings of this evaluation. E. Determine the best means of presenting CMFs to practitioners. Facilitate discussion by the appropriate HSM stakeholders to identify the options for CMF presentation (i.e., present all or present best) in Part D of the HSM. This discussion should also determine the appropriate mechanism for presenting the CMFs for Part D (e.g., HSM Part D and CMF Clearinghouse, CMF Clearinghouse only, CMF Clearinghouse with indicator for HSMÂapproved CMFs). F. Determine appropriate acceptance criterion for the HSM. Apply the refined procedure from Item D to a subset of the CMFs considered for inclusion in the HSM. Use the findings from this application to determine the minimum quality score that produces the same rate of acceptance as obtained when the existing HSM procedure is used. Recommend a minimum threshold score that can be used to identify CMFs suitable for inclusion in the HSM. Use the findings from this research to finalize the recommended procedure.
