Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results (2021)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

70 Introduction CPMs or SPFs may include only traffic volumes as predictor variables, traffic volumes plus a limited number of geometric and traffic control variables, or traffic volumes and a large number of geometric and traffic control variables. Depending on the data that are used for the estima- tion, the CPMs are considered more reliable within the range of the independent variables that were available in the data used for the estimation of the CPMs. The HSM indicates that the application of the CPMs to “sites with AADTs substantially outside this range may not provide reliable results.” However, when these CPMs are applied by practitioners in their jurisdiction, AADT and other characteristics of a particular site may be outside the range of the data that were used to estimate the CPMs. For Scenario 5: Predicting outside the range of independent variables, the functional form of CPMs and SPFs is an important element in the assessment of the reliability of results when predicting outside the range of the data used to estimate the CPM or SPF. When CPMs are used to predict the number of crashes at sites whose site characteristics (especially AADT) are outside the range of the data used to estimate the CPMs, then the practitioner is implicitly assuming that the functional form of the CPM is applicable and valid outside the range of the original data used to estimate the CPMs. In reality, the true functional form of an SPF is not known. Because traffic volume is often the most important contributor to crashes, this chapter will focus on CPMs for roadway segments with AADT as the only independent variable. The method and procedural steps to assess the potential reliability of predictions can be used for other inde- pendent variables found in a CPM or SPF. Procedures to Assess Potential Reliability Guidance on the potential reliability of using CPMs to predict outside the range of indepen- dent variables is presented here, including five options for improving these predictions. The complexity of execution of the options increases from Option 1 through Option 5. The GOF statistics of the results of each option give the practitioner the information to accept or discard a given option as well as a comparative analytical process to select the best (or feasible) option to improve the CPM predictions. Procedural Steps and Example Application This section describes the procedure for improving the reliability of using CPMs to predict outside the range of independent variables. Because traffic volume is often the most important contributor to crashes, this procedure focuses on CPMs for roadway segments with AADT as the only independent variable. C H A P T E R 6 Reliability Associated with Predicting Outside the Range of Independent Variables

Reliability Associated with Predicting Outside the Range of Independent Variables 71   The procedure comprises two major steps: Step A. Assemble the data needed for the procedure for the examination of different options for improving the predictions outside the range of AADT. Step B. Execute one or more of the five options for improving the predictions outside the range of AADT, and compare the GOF statistics to select the best option. Step A and Step B will also be demonstrated by an example application. The steps are described as follows. Step A. Assemble the Data Needed for the Procedure for the Examination of Different Options for Improving the Predictions Outside the Range of AADT The data needed for the procedure (including calibration of SPFs) can be found in HSM Part C; Bahar and Hauer (2014); or Srinivasan et al. (2016). Step B. Execute One or More of the Five Options for Improving the Predictions Outside the Range of AADT, and Compare the GOF Statistics to Select the Best Option The five options of Step B are as follows: Option 1. Perform calibration. Option 2. Adjust parameter/coefficient for AADT and perform calibration. Option 3. Estimate calibration function or SPF by modifying the coefficient for AADT and perform calibration. Option 4. Estimate calibration function or SPF and perform calibration. Option 5. Estimate calibration function or SPF with different parameters for AADT and the other factors, and perform calibration. Option 1. Perform Calibration Option 1 is the most popular because it is discussed in the HSM. A calibration factor is calculated and applied to the CPM. As presented in the HSM, the most common form for a CPM that relates crash frequency and AADT is the following: [ ]( ) ( )= × + × = × ×N L exp a b ln AADT L e AADTSPF a b where NSPF = predicted average number of crashes on a road segment L = length of the road segment a and b = regression coefficients to be estimated This type of model is sometimes called a power function. In the power model, it is generally accepted that b is positive because the number of crashes is expected to increase with an increase in traffic volume. Procedure: Scenario 5 Predicting outside the range of independent variables (with focus on AADT)

72 Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results This equation can also be used for CPMs for specific base conditions, where roadways meet certain conditions. When the site-specific conditions do not meet the base conditions, the HSM recommends adjusting the crash estimate from this equation by applying CMFs (also called SPF adjustment factors), and then a calibration factor to account for differences between jurisdictions. Step 1. The calibration factor can be calculated as follows: ∑ ∑ =C observed crashes predicted crashes all sites all sites Following the procedure illustrated in HSM Part C, the computed calibration factor is then applied to the CPMs to predict crashes for each site in the new data. The CPM for the new data becomes: ( )= ∗ ∗ ∗ . . .1 2N N C CMF CMF CMFpredicted SPF n where CMF1, CMF2 . . . CMFn = CMFs for local conditions for site characteristics variables 1 through n The Calibrator can be used to calculate the calibration factor. Step 2. Use The Calibrator to assess the performance based on the following GOF measures: Mean Absolute Deviation ∑ = − MAD y y n i ii where yi = predicted values from the SPF yi = observed counts n = validation data sample size CURE Plots and Related Measures CURE plots provide a visual representation of GOF over the range of a given variable and identify potential concerns such as the following: • Long trends: Long trends in the CURE plot (increasing or decreasing) indicate regions of bias that practitioners should rectify through improvement to the SPF. This can be seen from the CURE plots. • Percent exceeding the confidence limits [outside 95% CI (%)]: Cumulative residuals outside the 95% confidence limits indicate a poor fit over that range in the variable of interest. Cumu lative residuals frequently outside the confidence limits indicate possible bias in the SPF. • Vertical changes (Max_Cure): Large vertical changes in the CURE plot are potential indi- cators of outliers, which require further examination. • Maximum value exceeding 95% confidence limits (Max_DCure): This measures the distance between the CURE and the 95% confidence limits if CURE is outside the confidence limits. The bigger the values, the poorer the fit. • Average value exceeding 95% confidence limits (Avg_DCure): While Max_DCure mea- sures the maximum difference between CURE and the 95% confidence limits, Avg_DCure

Reliability Associated with Predicting Outside the Range of Independent Variables 73   measures the overall distance between the CURE and the 95% confidence limits for those outside the confidence limits. Similar to Max_DCure, a smaller average value exceeding 95% indicates less bias in the SPF. Note: Additional GOF measures may also be estimated if there will be a comprehensive comparison of GOF statistics among all five options. These are modified R2, dispersion param- eter (k), and CV(C). Step 3. If the GOF statistics are not satisfactory, proceed to Option 2. Option 2. Adjust Parameter/Coefficient for AADT and Perform Calibration Option 2 provides the next procedure to be followed especially if Option 1 did not provide a satisfactory fit of the new data based on GOF measures indicated by The Calibrator. Under NCHRP Project 17-45, Bonneson et al. (2012) demonstrated the necessity for adjusting the parameter/coefficient for AADT for different AADT ranges. This project illustrated that the CPMs developed using California, Missouri, and Washington freeway data for higher and lower AADTs were different. This implied that the CPMs may provide biased estimates of crashes when they are directly applied to the sites where AADTs are outside the range of the original data used to estimate the CPMs. CMFs for AADTs were applied to both multi-vehicle (MV) and SV crashes to address high traffic-volume effects: For MV crashes: , ,CMF emv hv b Pmv hv hv= where CMFmv,hv = high traffic-volume CMF for multi-vehicle crashes bmv,hv = high traffic-volume calibration coefficient for multi-vehicle crashes Phv = proportion of AADT during hours where volume exceeds 1,000 vehicle/hour/lane For SV crashes: , ,CMF esv hv b Psv hv hv= where CMFsv,hv = high traffic-volume CMF for single-vehicle crashes bsv,hv = high traffic-volume calibration coefficient for single-vehicle crashes Based on this research finding, the coefficient associated with AADT may be different depending on the range of AADT (Bonneson et al. 2012). Thus, since AADTs for the application are outside the range of the data used to estimate the original CPMs, Option 2 aims to identify a more appropriate coefficient for AADT based on trial and error. A trial and error approach may be more time-consuming compared with estimating the parameter, but it can be implemented by practitioners that have limited experience with statistical methods. There are six steps in Option 2: Step 1. Assume the parameter/coefficient b for AADT in the new data as: b_new = b*A_adj, where A_adj = an adjustment factor.

74 Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results There is no current guideline on what this adjustment factor should be, and a trial and error approach is recommended by investigating multiple adjustment factors (e.g., 1.5, 1.25, 0.75, and 0.5). Step 2. Predict the number of crashes based on b_new. Step 3. Calculate calibration factor using the following equation: ∑ ∑ =C observed crashes predicted crashes all sites all sites Step 4. Use The Calibrator to assess the performance based on the GOF measures, as noted in Option 1, Step 2: modified R2, dispersion parameter (k), CV(C), MAD, CURE plots and related measures (Max_Cure, Max_DCure, Avg_DCure, Outside 95% CI). Step 5. If the GOF statistics are not satisfactory, modify A_adj, and repeat the process (Step 1 through Step 4). Step 6. Select the A_Adj that provides the best fit based on the GOF statistics, or proceed to Option 3. Option 3. Estimate Calibration Function or SPF by Modifying the Coefficient for AADT and Perform Calibration Option 3 involves estimating a calibration function or SPF by modifying only the coefficient for AADT and then performing a simple calibration to ensure that the observed and predicted crashes be equal. There are five steps in Option 3: Step 1. Estimate calibration function or SPF of the following form: Nnew SPF = AADT b1 × NSPF Srinivasan et al. provide guidance on using readily available tools, such as Excel, to estimate calibration functions. Step 2. Calculate predicted crashes using the newly developed Nnew SPF. Step 3. Calculate the calibration factor as follows: Npredicted = C × Nnew SPF. Step 4. Use The Calibrator to assess the performance based on the GOF measures, as noted in Option 1, Step 2: modified R2, dispersion parameter (k), CV(C), MAD, CURE plots and related measures (Max_Cure, Max_DCure, Avg_DCure, Outside 95% CI). Step 5. If the GOF statistics are not satisfactory, proceed to Option 4. Option 4. Estimate Calibration Function or SPF and Perform Calibration Option 4 also involves the estimation of a calibration function, but unlike Option 3, the coefficient for all the terms in the SPF/CPM are estimated. If the NSPF includes CMFs (also called SPF adjustment factors), they are also raised to a power (Note: a possible source of criticism). There are five steps in Option 4: Step 1. Estimate calibration function of the following form: Nnew SPF = a1 × (NSPFc1). Step 2. Calculate predicted crashes using the newly developed Nnew SPF. Step 3. Calculate CF as follows: Npredicted = CF × Nnew SPF. Srinivasan et al. provide guidance on using readily available tools, such as Excel, to estimate calibration functions. Step 4. Use The Calibrator to assess the performance based on the GOF measures, as noted in Option 1, Step 2: modified R2, dispersion parameter (k), CV(C), MAD, CURE plots and related measures (Max_Cure, Max_DCure, Avg_DCure, Outside 95% CI). Step 5. If the GOF statistics are not satisfactory, proceed to Option 5.

Reliability Associated with Predicting Outside the Range of Independent Variables 75   Option 5. Estimate Calibration Function or SPF with Different Parameters for AADT and the Other Factors, and Perform Calibration Option 5 is a combination of Option 3 and Option 4. A calibration function is estimated, but different coefficients are introduced for AADT and the other parameters. There are five steps in Option 5: Step 1. Recalibrate using SPF and AADT as independent variables, both variables are assumed to be power functions in the new model as shown: = × ×1 2 2N a aadt Nnew SPF b SPF c Step 2. Calculate predicted crashes using the newly developed Nnew SPF. Step 3. Calculate CF as follows: Npredicted = CF × Nnew SPF. Srinivasan et al. provide guidance on using readily available tools, such as Excel, to estimate calibration functions. Step 4. Use The Calibrator to assess the performance based on the GOF measures, as noted in Option 1, Step 2: modified R2, dispersion parameter (k), CV(C), MAD, CURE plots and related measures (Max_Cure, Max_DCure, Avg_DCure, Outside 95% CI). Step 5. Select the best option for improving the predictions outside the range of AADT (or other independent variables). Example Application—Scenario 5 Predicting outside the range of independent variables (with focus on AADT) Question: The California freeway crash, traffic, and roadway characteristics data stored in the Highway Safety Information System (HSIS) (www.hsisinfo.org), for the 2005–2014 period, was used to estimate SPFs for rural 4-lane, flat terrain freeway segments with maximum AADT < 30,000 vehicles/day. Ramp influence areas (based on 0.3 miles on either side of a ramp) were excluded. Short segments less than 0.01 miles were also excluded. The data were categorized based on number of lanes, terrain, and area types (rural or urban areas). The crash types considered included Total crashes, SV crashes, and MV crashes. The question is how reliable the predicted values would be if using these SPFs for the same facility types but with AADT volumes ranging between 30,000 and 60,000 vehicles/day, and how best to improve these predictions. Outline of Solution Step A. Assemble the Data Needed for the Procedure for the Examination of Different Options for Improving the Predictions Outside the Range of AADT The California freeway crash, traffic, and roadway characteristics data from 2005–2014 (from HSIS) were extracted. The summary statistics for the two datasets are provided in Table 22.

76 Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results Step B. Execute One or More of the Five Options for Improving the Predictions Outside the Range of AADT, and Compare the GOF Statistics to Select the Best Option Compute the GOF measures for one or more of five options and compare the statistics to select the best option. The five options were investigated for Total, MV, and SV crashes, respectively. MV Crashes. For MV crashes, the GOF measures’ statistics are shown in Table 23, and the CURE plots are depicted in Figure 6 through Figure 10. For Option 2, a few values of A_adj (adjustment factor) were investigated, and the best one was used to compare with the other options. Variable Data used for SPF development* (AADT: < 30,000 vehicles/day) (AADT: 30,000 to 60,000 vehicles/day) Min Max Mean St.dev Sum Min Max Mean St.dev Sum AADT (vehicles/day) 1,590 29,909 19,806 5,697.7 NA 30,100 59,706 39,096.06 7,636.01 NA Segment length (mi) 0.01 9.018 0.570 0.850 393.07 0.01 4.127 0.621 0.810 194.50 SV crashes 0 78 6.927 10.315 4,773 0 104 12.661 17.465 3,963 MV crashes 0 51 4.084 6.804 2,814 0 138 13.217 17.929 4,137 Total crashes 0 107 11.012 16.346 7,587 0 231 25.879 34.344 8,100 * Number of segments = 689; ** Number of segments = 313. Data used for testing the different options** Table 22. Summary statistics for the datasets of California rural flat highway segments in the example application—Scenario 5: Predicting outside the range of independent variables (with focus on AADT). Option Number ofcrashes k Modified R 2 CV MAD Max_Cure Max_DCure Avg_DCure Outside 95% CI (proportion) Option 1 4,137 0.268** 0.840* 0.052 4.557 202.206 105.923 28.183 0.524 Option 2: increase AADT coefficient by 40% 4,137 0.259 0.825 0.051 4.721 189.437 93.705 24.031 0.508 Option 3 4,137 0.259 0.820 0.051 4.758 187.201 91.281 24.193 0.514 Option 4 4,137 0.264 0.809 0.051 4.748 193.223 92.279 25.255 0.521 Option 5 4,137 0.256 0.833 0.051 4.767 184.737 89.959 23.211 0.518 *Best option: dark shaded cell; **second-best option: lighter shaded cell. Table 23. GOF measure results for MV crashes (rural 4-lane flat terrain freeways) in the example application— Scenario 5: Predicting outside the range of independent variables (with focus on AADT).

Reliability Associated with Predicting Outside the Range of Independent Variables 77   Figure 7. CURE plots for MV crashes: Option 2. Figure 6. CURE plots for MV crashes: Option 1.

78 Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results Figure 8. CURE plots for MV crashes: Option 3. Figure 9. CURE plots for MV crashes: Option 4.

Reliability Associated with Predicting Outside the Range of Independent Variables 79   Except for modified R2, the lower the value of each GOF measure, the better the performance for the option. The best option depends on the GOF measure that is chosen for consideration. For MV crash estimates, Option 1 would be the best one if modified R2 or MAD is used, while it would be the worst based on the other GOF measures. Overall, Option 5 has the best perfor- mance, and Option 2 and Option 3 are good candidates for the second-best performance. SV Crashes. For SV crashes, the GOF measures’ statistics are shown in Table 24. For Option 2, a few values of A_adj (adjustment factor) were investigated, and the best one was used to com- pare with the other options. Figure 10. CURE plots for MV crashes: Option 5. Option Number ofCrashes k Modified R 2 CV MAD Max_Cure Max_DCure Avg_DCure Outside 95% CI (proportion) Option 1 3,963 0.223** 0.843* 0.048 4.540 164.208 58.179 19.131 0.597 Option 2: increase AADT coefficient by 50% 3,963 0.225 0.835 0.048 4.594 169.912 58.177 17.024 0.604 Option 3 3,963 0.223 0.841 0.048 4.546 170.083 56.191 18.070 0.597 Option 4 3,963 0.224 0.841 0.048 4.544 172.073 56.699 18.485 0.601 Option 5 3,963 0.219 0.836 0.048 4.615 160.145 58.148 15.749 0.594 *Best option: dark shaded cell; **second-best option: lighter shaded cell. Table 24. GOF measure results for SV crashes (rural 4-lane flat terrain freeways) in the example application— Scenario 5: Predicting outside the range of independent variables (with focus on AADT).

80 Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results As noted, except for modified R2, the lower the value of each GOF measure, the better the performance for the option. The best option depends on the GOF measure that is chosen for consideration. For SV crash estimates, Option 1 would be the best one if modified R2 or MAD is used (CV is the same for all options). Overall, Option 5 has very good performance and Option 1, Option 3, and Option 4 have comparable results and are good candidates for the second-best performance. Total Crashes. For Total crashes, the GOF measures’ statistics are shown in Table 25. For Option 2, a few values of A_adj (adjustment factor) were investigated, and the best one was used to compare with the other options. Overall, the results indicate that Option 5 consistently performs the best, while Option 3 provides second-best predictions for all MV, SV, and Total crashes. Option 1 has generally the worst performance based on all GOF measures except for modified R2 and MAD (Note: Option 1 showed the second-best performance for SV crashes for three GOF measures: k, Max_Cure, and outside 95% CI). Overall Option 2 provided better results when compared with the traditional calibration method (Option 1) except for SV crash predictions. The advantage of Option 2 is that it just involves trial and error with different adjustment fac- tors for AADT and does not need the practitioner to conduct statistical analysis as in Option 3 through Option 5. Hence, Option 2 is an option that practitioners with limited statistical experi- ence should consider. However, in some cases, none of the adjustment factors may provide a satisfactory result. In that case, Option 3 through Option 5 may need to be investigated. Although the example application was only provided with adjustment factor AADT, a similar approach could be adopted for adjusting the coefficients for other variables depending on the specific SPF. Based on these results, the practitioner decides to use Option 5 because it consistently per- forms the best and to use Option 3 when updating the SPFs in the upcoming years. *Best option: dark shaded cell; **second-best option: lighter shaded cell. Option Number ofCrashes k Modified R 2 CV MAD Max_Cure Max_DCure Avg_DCure Outside 95% CI (proportion) Option 1 8,100 0.245 0.866* 0.048 7.943 385.276 192.196 66.998 0.649 Option 2: increase AADT coefficient by 50% 8,100 0.237** 0.860 0.047 8.050 359.618 173.546 59.974 0.629 Option 3 8,100 0.237 0.853 0.047 8.177 354.778 174.326 57.655 0.617 Option 4 8,100 0.239 0.852 0.047 8.143 362.567 178.741 59.359 0.613 Option 5 8,100 0.233 0.858 0.047 8.296 327.990 170.257 52.475 0.623 Table 25. GOF measure results for Total crashes (rural 4-lane, flat terrain freeways) in the example application— Scenario 5: Predicting outside the range of independent variables (with focus on AADT).