Understanding and Communicating Reliability of Crash Prediction Models (2021)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

88 Chapter 8. Reliability Associated with Predictions Using CPMs Estimated for Other Facility Types: Problem Illustration with Possible Solutions Background Chapters in Part C of the 1st edition of the HSM (AASHTO, 2010) provided CPMs for rural two-lane roads, rural multilane roads, and urban and suburban arterials. The supplement chapters (Chapters 18 and 19) included CPMs for freeways and ramps. The upcoming 2nd edition of the HSM is expected to provide updated CPMs for the original Part C’s chapters. In addition, other NCHRP projects are also providing CPMs for consideration for inclusion in the 2nd edition of the HSM for different facility types including roundabouts, one-way streets, six-lane + multilane highways, and intersection types not included in the 1st edition of the HSM. However, even with these additional prediction models, there may always be specific facility types for which specific CPMs will not be available. In these cases, practitioners may use a CPM that has been estimated for other similar facility types to predict the number of crashes for facility types for which CPMs are not available. However, the reliability of predicting using CPMs estimate for other facility types is not known. In many ways, the issues associated with this question are similar to the previous question: “Predicting Outside the Range of the Independent Variable”. For example, the functional form of the CPM and the range of the site characteristics could potentially affect the reliability of using the CPM to predict the number of crashes at a different facility type. In addition, when CPMs estimated for other facility types are used, we may be predicting outside the applicable AADT range. Table 35 presents the CPMs that were estimated for daytime crashes on freeway segments as part of Srinivasan et al., (2011) using data from Ohio and North Carolina. The functional form of the CPM was as follows: Equation 75 𝑌 exp 𝛽 𝛽 𝑋 𝛽 𝑋 ⋯… . .𝛽 𝑋 Where, Y is the predicted average number of crashes per year during daytime, the β’s are the coefficient estimates, and X’s are the independent variables in the model (shown in the first column in the Table 35). The independent variables include day traffic volume, indicator variable to represent whether the segment is within the influence of an interchange or ramp, urban/rural indicator variable, indicator variable If the CPM for one facility type can be used to predict the crashes for another facility type, then there is an implicit assumption that the functional form of the traffic volume term is the same, and coefficient (i.e., β) is the same, for the two facility types (the rows corresponding to the traffic volume terms are highlighted in the Table). As it can be seen in Table 35, even within the same state, comparing the coefficients for urban four to five lanes with six or more lanes, the coefficients are quite different. This indicates that there needs to be some caution before a CPM for one facility type is used to predict crashes for another facility type. Objective of this Chapter The objective of this chapter is to provide guidance on the potential reliability of using CPMs to predict the number of crashes at a different facility type. The bias, variance, and repeatability associated with this condition is provided in Table 36.

89 Table 35. CPMs for Daytime Crashes on Freeway Segments from Ohio and North Carolina. Variables/Statistics Ohio North Carolina 4 to 5 lanes 6 or more lanes 4 to 5 lanes 6 or more lanes Urban Rural Urban Rural Estimate (S.E.) Estimate (S.E.) Estimate (S.E.) Estimate (S.E.) Estimate (S.E.) Estimate (S.E.) Intercept 0.8510 (0.0749) 0.9561 (0.0898) -0.8581 (0.2615) 1.2379 (0.1990) 0.4860 (0.0504) 0.7854 (0.0630) ln(Day Vol/10000) 1.3687 (0.0549) 0.9084 (0.1068) 1.6454 (0.0910) 1.4408 (0.0829) 0.7397 (0.1335) Day Vol/10000 0.2164 (0.0479) 0.2610 (0.0086) Within Influence of Interchange/Ramp? (1 for yes, 0 for no) 0.8902 (0.0814) 0.7628 (0.3997) 0.9702 (0.0684) 0.5209 (0.0873) 0.7014 (0.1038) 0.4881 (0.0455) Urban? (1 for yes, 0 for no) 0.8235 (0.1828) 0.1750 (0.0513) 6 or 7 lanes? (1 for yes, 0 for 8+ lanes) 0.1814 (0.0512) Right Shoulder Width (ft) -0.0652 (0.0119) Left Shoulder Width (ft) -0.0310 (0.0031) k1 0.2655 0.2948 0.2744 0.3062 0.2689 0.1915 Freeman-Tukey R2 0.538 0.853 0.475 0.593 0.746 0.680 Pseudo R2 0.505 0.285 0.336 0.425 0.711 0.550 Crashes 9944 3778 24256 7267 10221 15238 Mile-years 585.1 693.7 719.9 488.2 2283.6 857.3 Table 36. Bias, Variance, and Repeatability Associated with Predicting Crashes at a Different Facility Type. Influence Category Factor Effect of Factor on Reliability of CPM Bias Variance Repeatability Application- related factors influencing reliability Application site has characteristics that are not represented by CPM Less reliable Less reliable Less reliable if error is due to poor description of conditions to which CPM applies. If only CPMs from one other similar facility type is available, practitioners can follow the procedure in the Chapter 7 in determining whether to use the CPMs. However, if multiple similar facility types have CPMs, then the practitioner needs to decide which CPMs will be used. This Chapter provides further discussion of this issue. A few examples are explored, and a heuristic procedure is suggested for practitioners to improve the reliability of a specific application.

90 Goodness-of-Fit Measures Many goodness-of-fit measures have been proposed including mean absolute deviation (MAD), modified R2 value, dispersion parameter (K), coefficient of variation of calibration factor (defined as CV), cumulative residual (CURE) Plots, percent of CURE plot ordinates for fitted values (after calibration) exceeding 2σ limits, and the maximum absolute deviation from zero. The definitions of these criteria can be found in the User Guide for the FHWA Calibrator Tool (Lyon et al., 2016) and are described below. Mean Absolute Deviation (MAD) The mean absolute deviation is a measure of the average value of the absolute difference between observed and predicted crashes. Equation 76 𝑀𝐴𝐷 ∑ |𝑦 𝑦 |𝑛   where: 𝑦 = predicted values from the SPF. 𝑦 = observed counts. n = validation data sample size. Modified R2 This GOF measure seeks to measure the amount of systematic variation explained by the SPF. Larger values indicate a better fit to the data in comparing two or more competing SPFs. Values greater than 1.0 indicate that the SPF is over-fit and some of the expected random variation is incorrectly explained as the systematic variation. Equation 77 𝑅 ∑ 𝑦 𝑦 ∑ 𝜇∑ 𝑦 𝑦 ∑ 𝑦   where: 𝑦 = observed counts. 𝑦 = predicted values from the SPF. 𝑦= sample average. 𝜇 = 𝑦 -𝑦 . Dispersion Parameter (k) The dispersion parameter is a measure of the variability in the data. It can be expressed as follows: Equation 78 𝑘 𝑉𝑎𝑟 𝑚 𝐸 𝑚𝐸 𝑚   where: k = estimate of the dispersion parameter in the calibration procedure. Var{m} = estimated variance of mean crash rate. E{m} = estimated mean crash rate.                                                               

91 The estimated variance increases as dispersion increases, and consequently the standard errors of estimates increase as well. As a result, an SPF with lower dispersion parameter estimates (i.e., smaller values of k) is preferred to an SPF with more dispersion. Note that the FHWA Calibrator Tool (Lyon et al., 2016) can provide either a constant dispersion parameter, or one that varies by length (for road segments). Coefficient of Variation of Calibration Factor (Defined As CV) The CV of the calibration factor is the standard deviation of the calibration factor divided by the estimate of the calibration factor as shown in the following equation. Equation 79 𝐶𝑉 𝑉 CC Where: CV = coefficient of variation of the calibration factor. C = estimate of the calibration factor. V(C) = variance of the calibration factor, can be calculated as follows: Equation 80 𝑉 𝐶 ∑ 𝑦 𝑘 ∗ 𝑦∑ 𝑦 Where: 𝑦i = observed counts. 𝑦 = uncalibrated predicted values from the SPF. k = dispersion parameter. CURE Plots and Related Measures A CURE plot is a graph of the cumulative residuals (observed minus predicted crashes) against a variable of interest sorted in ascending order (e.g., major road traffic volume). CURE plots provide a visual representation of GOF over the range of a given variable, and help to identify potential concerns such as the following:  Long trends: long trends in the CURE plot (increasing or decreasing) indicate regions of bias that analysts should rectify through improvement to the SPF. This can be seen from the CURE plots.  Percent exceeding the confidence limits (Outside 95% CI (%)): cumulative residuals outside the 95% confidence limits indicate a poor fit over that range in the variable of interest. Cumulative residuals frequently outside the confidence limits indicate possible bias in the SPF.  Vertical changes (Max_Cure): Large vertical changes in the CURE plot are potential indicators of outliers, which require further examination. Further information can be found in Chapter 7 of Hauer’s book (Hauer, 2015).  Maximum value exceeding 95% confidence limits (Max_DCure): This measures the distance between the CURE and the 95% confidence limits if CURE is outside the confidence limits. The bigger the values, the poorer the fit.  Average value exceeding 95% confidence limits (Avg_DCure): While Max_DCure measures the maximum difference between CURE and the 95% confidence limits, Avg_DCure measures the overall distance between the CURE and the 95% confidence limits for those outside the confidence limits. Similar to Max_DCure, smaller average value exceeding 95% indicates less bias in the SPF.

92 The FHWA Calibrator Tool (Lyon et al., 2016) provides CURE plots, percent exceeding the confidence limits, and maximum vertical change of the CURE plot. Maximum value exceeding 95% confidence limits and average value exceeding 95% confidence limits were added in this study to compare the proposed options. If CPMs from two similar facility types are available, then the performance of these CPMs can be assessed using the GOF measures discussed below. Based on this assessment, if one of the CPMs performs better, then the practitioner can choose that CPM for prediction. Assessment of Applying CPMs Using California Data California traffic and crash data from HSIS for the years 2005-2014 were used to demonstrate the assessment of applying CPMs from other facilities. The facility types used in this assessment are provided in Table 37. Ramp influence areas were removed from the database. Table 37. Facility Types by Area Type, Terrain, and Number of Lanes. Group Facility Types used for Estimating CPMs (estimation group) Facility Types Used for Applying the Estimated CPMs (application group) Crash Types Facility type Segments Facility type Segments Group 1 Rural 4 lane, Flat terrain 1075 Rural 6 lane, Flat terrain 102 SV, MV, Total Urban 6 lane, Flat terrain 437 Group 2 Rural 4 lane, Flat terrain 1075 Urban 4 lane, Flat terrain 428 SV, MV, Total Urban 6 lane, Flat terrain 437 Group 3 Rural 4 lane, Rolling terrain 421 Rural 6 lane, Rolling terrain 58 SV, MV, Total Urban 6 lane, Rolling terrain 253 Group 4 Rural 4 lane, Rolling terrain 421 Urban 4lane, Rolling terrain 263 SV, MV, Total Urban 6 lane, Rolling  terrain   253  The summary statistics for Groups 1 and 2 are available in Table 38, and the summary statistics for Groups 3 and 4 are provided in Table 39. For Groups 1 and 2, the CPMs for single-vehicle (SV), multi- vehicle (MV), and total crashes were developed using negative binomial model with data from rural 4- lane flat terrain highways and data from urban 6-lane flat terrain highway segments, respectively. These groups can be called as the ‘estimation group’. The CPMs were to predict the crashes in rural 6-lane flat terrain highways for Group 1, and urban 4-lane flat terrain highways for Group 2 (these are ‘application’ groups), after calibration. In essence, the goals was to determine how well the CPMs estimated based on a particular facility type fit the data for a different facility type. The assumption was that sufficient data may not be available to estimate CPMs directly for the application groups. The performance of the CPMs in the application groups was evaluated using the assessment criteria (i.e., GOF measures).

93 Table 38. Summary Statistics for Groups 1 and 2. Variable (estimation group) Data 1a: Urban 6-lane, Flat terrain for CPM development (437 segments) Data 1b: Rural 4-lane, Flat terrain for CPM development (1075 segments) Min Max Mean Stdev Sum Min Max Mean Stdev Sum Segment length (mi) 0.011 2.46 0.19 0.26 83.97 0.01 9.02 0.56 0.82 605.81 Single-vehicle crashes 0 143 8.67 13.30 3,788 0 104 8.77 13.20 9,424 Multi-vehicle crashes 0 288 20.49 29.44 8,954 0 164 7.83 14.36 8,419 Total crashes 0 345 29.16 39.61 12,742 0 231 16.60 25.97 17,843 AADT 6580 262,079 98,471 15,275 NA 1590 98,913 26,965 10,022 NA Median width (ft) 3 99 46.06 9.69 NA 0 99 81.83 13.56 NA Shoulder width (ft) 0.00 32.00 16.35 2.10 NA 0.00 33.00 15.99 1.96 NA Design speed (mph) 45 70 69.31 1.29 NA 50 70 69.80 0.93 NA Variable (application group) Group 1: Rural 6-lane, Flat terrain, for applying the estimated CPMs (102 segments) Group 2: Urban 4-lane, Flat terrain, for applying the estimated CPMs (428 segments) Min Max Mean Stdev Sum Min Max Mean Stdev Sum Segment length (mi) 0.01 2.00 0.38 0.45 39.01 0.01 2.23 0.20 0.27 86.82 Single-vehicle crashes 0 122 13.28 18.99 1355 0 67 5.37 8.59 2299 Multi-vehicle crashes 0 178 20.39 32.58 2080 0 168 9.86 18.73 4221 Total crashes 0 244 33.68 49.61 3435 0 235 15.23 25.04 6520 AADT 27,870 128,100 70,586 19,207 NA 4,122 226,400 55,343 12,476 NA Median width (ft) 20 84 42.89 8.33 NA 3 99 51.04 11.90 NA Shoulder width (ft) 0.00 23.00 17.50 2.15 NA 0.00 33.00 13.64 2.11 NA Design speed (mph) 65 70 69.89 0.46 NA 35 70 68.51 1.92 NA

94 Table 39. Summary Statistics for Groups 3 and 4. Variable (estimation group) Data 2a: Rural 4-lane, Rolling terrain, for CPM development (421 segments) Data 2b: Urban 6-lane, Rolling terrain for CPM development (253 segments) Min Max Mean Stdev Sum Min Max Mean Stdev Sum Segment length (mi) 0.01 4.39 0.46 0.61 194.10 0.011 1.65 0.21 0.28 53.12 Single-vehicle crashes 0 166 7.95 14.64 3345 0 51 7.40 9.19 1873 Multi-vehicle crashes 0 141 6.17 14.63 2598 0 138 15.88 21.76 4017 Total crashes 0 241 14.12 27.46 5943 0 184 23.28 29.04 5890 AADT 3,212 70,300 25,196 9,507 NA 10,001 212,261 94,112 17,779 NA Median width (ft) 4 99 68.33 19.60 NA 6 99 50.36 11.42 NA Shoulder width (ft) 0.00 36.00 15.02 1.80 NA 0.00 34.00 17.88 1.82 NA Design speed (mph) 60 70 69.48 1.20 NA 45 70 69.72 0.64 NA Variable (application group) Group 3: Rural 6-lane, Rolling terrain, for applying the estimated CPMs (58 segments) Group 4: Urban 4-lane, Rolling terrain highway for applying the estimated CPMs (263 segments) Min Max Mean Stdev Sum Min Max Mean Stdev Sum Segment length (mi) 0.01 4.26 0.35 0.72 20.18 0.01 1.72 0.23 0.29 59.51 Single-vehicle crashes 0 88 9.16 15.57 531 0 93 7.03 11.27 1850 Multi-vehicle crashes 0 127 15.97 24.83 926 0 128 10.48 17.21 2755 Total crashes 0 215 25.12 39.55 1457 0 169 17.51 26.55 4605 AADT 15,865 179,200 59,796 14,696 NA 8,160 133,100 51,058 13,357 NA Median width (ft) 14 99 53.87 14.72 NA 4 99 44.60 13.26 NA Shoulder width (ft) 0.00 22.00 17.70 2.24 NA 0.00 32.00 12.06 2.92 NA Design speed (mph) 65 70 69.66 0.75 NA 45 70 68.70 1.87 NA        

95 Results of the Assessment Assessment results for Groups 1, 2, 3, and 4, are listed in Table 40 through Table 43. For group 1, the CPMs from urban 6-lane highways with flat terrain (facility 1a) are much better in predicting the crashes for rural 6-lane highways with flat terrain, especially for MV and total crashes. For group 2, both the facility groups are not very effective in prediction crashes for urban 4-lane roadways with flat terrain, although the CPMs from urban 6-lane highways with flat terrain (facility 1a) perform better. Table 40. Assessment Results for Rural 6-lane Flat Terrain (Group 1). Crash type Number of crashes k Modified R2 CV MAD Max_Cure Max_DCure Avg_DCure Outside 95% CI Data to develop CPMs sv 1,355 0.18 0.84 0.08 4.72 70.55 0 0 0 Facility 1a 0.19 0.78 0.08 5.10 82.06 8.10 0.41 14% Facility 1b mv 2,080 0.49 0.83 0.13 7.72 112.35 10.37 1.30 29% Facility 1a 0.75 0.57 0.16 11.57 424.24 262.20 92.26 84% Facility 1b total 3,435 0.34 0.88 0.10 10.52 135.63 9.85 0.54 11% Facility 1a 0.49 0.59 0.12 17.84 610.66 375.62 119.32 83% Facility 1b Note: Facility 1a: CA urban 6-lane flat terrain highway; Facility 1b: CA rural 4-lane flat terrain highway Table 41. Assessment Results for Urban 4-lane Flat Terrain (Group 2). Crash type Number of crashes k Modified R2 CV MAD Max_Cure Max_DCure Avg_DCure Outside 95% CI Data to develop CPMs sv 2,299 0.55 0.45 0.07 3.25 222.63 90.87 13.24 54% Facility 1a 0.65 0.13 0.08 3.75 375.29 211.77 68.35 73% Facility 1b mv 4,221 1.12 0.39 0.11 6.71 548.54 270.70 96.31 73% Facility 1a 1.36 -2.79 0.12 10.28 1,681.41 1,209.20 406.37 95% Facility 1b total 6,520 0.83 0.43 0.09 8.98 728.09 393.66 139.18 70% Facility 1a 1.03 -1.14 0.10 12.96 2,092.56 1,579.80 529.32 92% Facility 1b Note: Facility 1a: CA urban 6-lane flat terrain highway; Facility 1b: CA rural 4-lane flat terrain highway For group 3, CPMs from both facility types (urban 6-lane rolling terrain highways and rural 4-lane rolling terrain highways) provide promising results for predicting crashes on rural 6-lane highways with rolling terrain, while the CPMs from urban 6-lane highways with rolling terrain performs better especially multi-vehicle and total crashes.

96 Table 42. Assessment Results for Rural 6-lane Rolling Terrain (Group 3). Crash type Number of crashes k Modified R2 CV MAD Max_Cure Max_DCure Avg_DCure Outside 95% CI Data to develop CPMs sv 531 0.96 0.73 0.26 4.72 62.47 0.00 0.00 0% Facility 2a 0.90 0.73 0.25 4.67 62.67 0.21 0.00 2% Facility 2b mv 926 1.71 0.41 0.32 11.05 186.41 51.83 5.82 29% Facility 2a 0.76 0.73 0.21 7.46 118.27 17.88 0.51 5% Facility 2b total 1,457 1.45 0.72 0.29 12.85 242.92 84.79 6.58 22% Facility 2a 0.94 0.72 0.24 12.02 196.39 34.60 1.52 10% Facility 2b Note: Facility 2a: CA rural 4-lane rolling terrain highway; Facility 2b: CA urban 6-lane rolling terrain highway In group 4, the CPMs for single-vehicle crashes from both facility types performed very well in predicting crashes for urban 4-lane highways with rolling terrain. However, CPMs for multi-vehicle and total crashes from the two facility types are not very effective in predicting crashes on urban 4-lane highways with rolling terrain. Table 43. Assessment Results for Urban 4-lane Rolling Terrain. Crash type Number of crashes k Modified R2 CV MAD Max_Cure Max_DCure Avg_Dcure Outside 95% CI Data to develop CPMs sv 1,850 0.60 0.57 0.09 4.15 126.42 20.92 0.71 14% Facility 2a 0.63 0.57 0.09 4.15 131.36 16.36 0.49 9% Facility 2b mv 2,755 1.01 -1.66 0.12 7.74 754.18 465.76 145.99 84% Facility 2a 0.83 0.65 0.11 5.72 261.77 95.96 19.63 44% Facility 2b total 4,605 0.68 -0.24 0.09 10.70 764.83 398.52 129.15 88% Facility 2a 0.66 0.67 0.09 8.60 346.24 127.56 28.92 52% Facility 2b Note: Facility 2a: CA rural 4-lane rolling terrain highway; Facility 2b: CA urban 6-lane rolling terrain highway Conclusions and Recommendations The results indicate that there is not a consistent pattern in terms of the types of facility types would be most appropriate to serve as ‘estimation’ group for a particular ‘application’ group. In some cases, the estimation group was appropriate for a particular crash type, but not for another crash type. So, to an extent, the practitioner would need to identify similar facility types to serve as estimation groups, and follow the process provided in Chapter 6 for assessing the performance of these groups in predicting the crashes for the application group.

97 Depending on how many facility types have CPMs and, practitioners may follow different procedures as shown below. Step 1: Check how many similar facility types have CPMs based on number of lanes, area type, and access control. Step 2: If only one facility type has CPMs that can be applied to the application group, and if the available CPMs are base models with only the AADT terms in the model, then practitioners can follow the Option 2 from the previous Chapter. Option 2 involves adjusting the parameter/coefficient for AADT and performing a calibration. Step 3: If only one facility type has CPMs, and they are fully specified models (i.e., the CPMs include other variables in addition to AADT), practitioners can follow options 1, 3, 4, and 5 in Chapter 6 to select the best option based on the assessment results. The five options described in Chapter 6 are:  Option 1: Perform calibration  Option 2: Adjust parameter/coefficient for AADT and perform calibration  Option 3: Estimate calibration function or SPF by modifying the coefficient for AADT and perform calibration  Option 4: Estimate calibration function or SPF and perform calibration  Option 5: Estimate calibration function or SPF with different parameters for AADT and the other factors, and perform calibration Step 4: If CPMs from multiple similar facility types are available, practitioners can select the CPMs that provide the best performance in terms of predicting crashes in the application group. This assessment can be done using the GOF measures discussed in Chapter, and the options discussed above. Step 5: If none of the CPMs from multiple facility types are able to provide satisfactory predictions in the application group, and sufficient data are not available to estimate CPMs from the application group, the following methods could be explored: Method 1: For each site in the application group, calculate the prediction as a weighted average of the predictions from the CPMs from multiple groups. For example p_CPM1 is the prediction from CPM1, and p_CPM2 is the prediction from CPM2, then a weighted average could be 𝑎 𝑝 1 𝑎 𝑝 , where a is parameter between 0 and 1; a will have to be chosen by trial and error. The value of a that gives the best GOF for the application dataset can be chosen. Method 2: Instead of calculating the prediction as a weighted average of the predictions, it may be possible to estimate a combined CPM for the application group as follows: CPM_application_group = 𝑎 𝐶𝑃𝑀1 exp 𝑐 𝐶𝑀𝑃2 Where, CPM1 and CPM2 are the equations corresponding to the two CPMs, and a, b, and c, are parameters to be estimated using negative binomial regression.