Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results (2021)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

81   Introduction Chapters in HSM Part C provided CPMs for rural two-lane roads, rural multilane roads, and urban and suburban arterials. The supplement chapters (Chapters 18 and 19) included CPMs for freeways and ramps. In addition, other NCHRP projects are also providing CPMs for consideration for inclusion in the second edition of the HSM for different facility types, including roundabouts, one-way streets, six-lane + multilane highways, and intersection types not included in the current HSM. However, even with these additional prediction models, there may be specific facility types for which specific CPMs will not be available. In these cases, practitioners may use a CPM that has been estimated for other similar facility types to predict the number of crashes for facility types for which CPMs are not avail- able. However, the reliability of predictions using CPM estimates for other facility types is not known. The issues associated with this question are similar to the issues associated with Scenario 5: Predicting outside the range of independent variables (with focus on AADT) (Chapter 6). For example, the functional form of the CPM and the range of the site characteristics could poten- tially affect the reliability of using the CPM to predict the number of crashes at a different facility type. In addition, when CPMs estimated for other facility types are used, the resulting predic- tions may be outside the applicable AADT range. Procedures to Assess Potential Reliability The goal is to determine how well the CPMs estimated based on a particular facility type (namely, the estimation facility type or estimation group) fit the data for a different facility type (namely, the application facility type or application group). The assumption is that suf- ficient data are not available to estimate CPMs directly for the application groups. Procedural Steps and Example Illustration This section describes the procedure for assessing one or more CPMs and selecting the CPM with the best GOF measure results to predict the crash frequency for facility types for which CPMs are not available. The facility types used for estimating CPMs form the estimation group of facilities and the facility types used for applying the estimated and calibrated CPMs for the application group of facilities. C H A P T E R 7 Reliability Associated with Crash Prediction Models Estimated for Other Facility Types

82 Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results The procedure comprises four or five main steps: Step 1. Define the application facility type. Step 2. Identify estimation facility type(s). Step 3. Assemble the data needed for the procedure. Step 4A, Case A. One facility type with CPMs base models with only variable AADT. Step 4B, Case B. One facility type with CPMs with other variables in addition to variable AADT. Step 4C, Case C. Multiple similar facility types with CPMs with other variables in addition to variable AADT. Step 5C, Case C. Multiple similar facility types with CPMs with other variables in addition to variable AADT—additional methods. These steps are described as follows. Step 1. Define the Application Facility Type The facility type is defined for which there is not a CPM to predict the crash frequency, and there are insufficient data to estimate its CPM. Step 2. Identify Estimation Facility Type(s) The facility types are identified by meeting two conditions: • Similar site characteristics to the facility type defined in Step 1 (such as number of lanes, area type, access control), and • Own CPMs. Step 3. Assemble the Data Needed for the Procedure The data needed for the procedure (including calibration of SPFs) can be found in HSM Part C; Bahar and Hauer; and Srinivasan et al. Step 4A, Case A. One Facility Type With CPMs Base Models With Only Variable AADT For Case A, there is only one facility type that has CPMs that can be applied to the application facility type, and the CPMs are base models with only AADT as the independent variable. This Case A is similar to Scenario 5: Predicting outside the range of independent variables but with only one variable (AADT). Thus, the next sub-steps for Step 4A follow Scenario 5, Option 2: Adjust Parameter/Coefficient for AADT and Perform Calibration: Step 4A-1. Assume the parameter/coefficient b for AADT in the new data as b_new = b ∗ A_adj, where A_adj = an adjustment factor. There is no current guideline on what this adjustment factor should be, and a trial and error approach is recommended by investigating multiple adjustment factors, e.g., 1.5, 1.25, 0.75, and 0.5. Step 4A-2. Predict the number of crashes based on b_new. Procedure: Scenario 6 Predicting using CPMs estimated for other facility types (application sites have characteristics that are not represented by CPM estimation sites and vice versa)

Reliability Associated with Crash Prediction Models Estimated for Other Facility Types 83   Step 4A-3. Calculate calibration factor using the following equation: ∑ ∑ =C observed crashes predicted crashes all sites all sites Step 4A-4. Use The Calibrator to assess the performance based on the following GOF measures. Mean Absolute Deviation ∑ = − MAD y y n i ii where yi = predicted values from the SPF yi = observed counts n = validation data sample size CURE Plots and Related Measures CURE plots provide a visual representation of GOF over the range of a given variable, and identify potential concerns such as the following: • Long trends: Long trends in the CURE plot (increasing or decreasing) indicate regions of bias that practitioners should rectify through improvement to the SPF. This can be seen from the CURE plots. • Percent exceeding the confidence limits [outside 95% CI (%)]: Cumulative residuals outside the 95% confidence limits indicate a poor fit over that range in the variable of interest. Cumu lative residuals frequently outside the confidence limits indicate possible bias in the SPF. • Vertical changes (Max_Cure): Large vertical changes in the CURE plot are potential indi- cators of outliers, which require further examination. • Maximum value exceeding 95% confidence limits (Max_DCure): This measures the dis- tance between the CURE and the 95% confidence limits if CURE is outside the confidence limits. The bigger the values, the poorer the fit. • Average value exceeding 95% confidence limits (Avg_DCure): While Max_DCure mea- sures the maximum difference between CURE and the 95% confidence limits, Avg_DCure measures the overall distance between the CURE and the 95% confidence limits for those outside the confidence limits. Similar to Max_DCure, a smaller average value exceeding 95% indicates less bias in the SPF. Note: Additional GOF measures may also be estimated if there will be a comprehensive comparison of GOF statistics among all five options. These are modified R2, dispersion parameter (k), and CV(C). Step 4A-5. If the GOF statistics are not satisfactory, modify A_adj, and repeat the process (Step 1 through Step 4). Step 4A-6. Select the A_Adj that provides the best fit based on the GOF statistics, or proceed to Option 3 (refer to Step 4B). Step 4B, Case B. One Facility Type with CPMs with Other Variables in Addition to Variable AADT For Case B, there is only one facility type that has CPMs that can be applied to the applica- tion facility type, and the CPMs are fully specified models (models that include other variables in addition to AADT).

84 Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results This Case B is similar to Scenario 5: Predicting outside the range of independent variables (with focus on AADT). Thus, the sub-steps for Step 4B follow Scenario 5’s Option 1 and Option 3 through Option 5 to select the best option based on the assessment of the GOF measure results. Option 1 and Option 3 through Option 5 are as follows: 4B-Option 1: Perform Calibration As presented in the HSM, the most common form for a CPM that relates crash frequency and AADT is the following: [ ]( ) ( )= × + × = × ×N L exp a b ln AADT L e AADTSPF a b where NSPF = predicted average number of crashes on a road segment L = length of the road segment a and b = regression coefficients to be estimated This equation can also be used for CPMs for specific base conditions, where roadways meet certain conditions. When the site-specific conditions do not meet the base conditions, the HSM recommends adjusting the crash estimate from this equation by applying CMFs (also called SPF adjustment factors) and then a calibration factor to account for differences between jurisdictions. Step 4B-1-1. The calibration factor can be calculated as follows: ∑ ∑ =C observed crashes predicted crashes all sites all sites Following the procedure illustrated in HSM Part C, the computed calibration factor is then applied to the CPMs to predict crashes for each site in the new data. The CPM for the new data becomes: ( )= ∗ ∗ ∗ . . .1 2N N C CMF CMF CMFpredicted SPF n where CMF1, CMF2 . . . CMFn = CMFs for local conditions for site characteristics variables 1 through n. The Calibrator can be used to calculate the calibration factor. Step 4B-1-2. Use The Calibrator to assess the performance based on the following GOF measures: modified R2, dispersion parameter (k), CV(C), MAD, CURE plots and related measures (Max_Cure, Max_DCure, Avg_DCure, Outside 95% CI). They are described in Step 4A-4. Step 4B-1-3. If the GOF statistics are not satisfactory, proceed to 4B-Option 3. 4B-Option 3: Estimate Calibration Function or SPF by Modifying the Coefficient for AADT and Perform Calibration 4B-Option 3 involves estimating a calibration function or SPF by modifying only the coefficient for AADT and then performing a simple calibration to ensure that the observed and predicted crashes are equal. There are five steps in 4B-Option 3. Step 4B-3-1. Estimate calibration function or SPF of the following form: = ×1N AADT Nnew SPF b SPF

Reliability Associated with Crash Prediction Models Estimated for Other Facility Types 85   Srinivasan et al. provide guidance on using readily available tools, such as Excel, to estimate calibration functions. Step 4B-3-2. Calculate predicted crashes using the newly developed Nnew SPF. Step 4B-3-3. Calculate the calibration factor as follows: = ×N C Npredicted new SPF Step 4B-3-4. Use The Calibrator to assess the performance based on the following GOF measures: modified R2, dispersion parameter (k), CV(C), MAD, CURE plots and related measures (Max_Cure, Max_DCure, Avg_DCure, Outside 95% CI). These are described in Step 4A-4. Step 4B-3-5. If the GOF statistics are not satisfactory, proceed to 4B-Option 4. 4B-Option 4: Estimate Calibration Function or SPF and Perform Calibration 4B-Option 4 also involves the estimation of a calibration function, but unlike Option 3, the coefficient for all the terms in the SPF/CPM are estimated. If the NSPF includes CMFs (also called SPF adjustment factors), they are also raised to a power (Note: a possible source of criticism). There are five steps in 4B-Option 4: Step 4B-4-1. Estimate calibration function of the following form: ( )= ×1 1N a Nnew SPF SPFc Step 4B-4-2. Calculate predicted crashes using the newly developed Nnew SPF. Step 4B-4-3. Calculate calibration factor as follows: = ×N C Npredicted new SPF Srinivasan et al. provide guidance on using readily available tools, such as Excel, to estimate calibration functions. Step 4B-4-4. Use The Calibrator to assess the performance based on the following GOF measures: modified R2, dispersion parameter (k), CV(C), MAD, CURE plots and related measures (Max_Cure, Max_DCure, Avg_DCure, Outside 95% CI). These are described in Step 4A-4. Step 4B-4-5. If the GOF statistics are not satisfactory, proceed to 4B-Option 5. 4B-Option 5: Estimate Calibration Function or SPF with Different Parameters for AADT and the Other Factors, and Perform Calibration 4B-Option 5 is a combination of 4B-Option 3 and 4B-Option 4. A calibration function is estimated, but different coefficients are introduced for AADT and the other parameters. There are five steps in 4B-Option 5: Step 4B-5-1. Recalibrate using SPF and AADT as independent variables, both variables are assumed to be power functions in the new model as follows: = × ×1 2 2N a aadt Nnew SPF b SPF c Step 4B-5-2. Calculate predicted crashes using the newly developed Nnew SPF. Step 4B-5-3. Calculate calibration factor as follows: = ×N C Npredicted new SPF

86 Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results Srinivasan et al. provide guidance on using readily available tools, such as Excel, to estimate calibration functions. Step B-5-4. Use The Calibrator to assess the performance based on the following GOF measures: modified R2, dispersion parameter (k), CV(C), MAD, CURE plots and related measures (Max_Cure, Max_DCure, Avg_DCure, outside 95% CI). These are described in Step 4A-4. Step B-5-5. If the GOF statistics are still not satisfactory, the practitioner may consider searching for other estimation facility types, or the practitioner may still decide to use (with due caution) the best option currently available for prediction of the crash frequency for facility type for which CPMs are not available. Step 4C, Case C. Multiple Similar Facility Types with CPMs with Other Variables in Addition to Variable AADT For Case C, multiple similar facility types have CPMs that can be applied to the application facility type, and the CPMs are fully specified models (models that include other variables in addition to AADT). The practitioner needs to decide which CPM performs better and gives the most reliable prediction estimates. For Case C, follow Option 1 through Option 5 to select the CPM under the best option based on the assessment of the GOF measure results. The description of Option 2 is found under Case 4A, and the descriptions of Option 1 and Option 3 through Option 5 are found under Case 4B. If, after following Option 5 in Case C, the GOF statistics are still not satisfactory for any of the multiple facility types in the application group, proceed to Step 5, Case C. Step 5, Case C. Multiple Similar Facility Types with CPMs with Other Variables in Addition to Variable AADT— Additional Methods If none of the CPMs from multiple facility types provide satisfactory predictions in the application group discussed, and sufficient data are not available to estimate CPMs from the application group, two alternative methods are proposed to improve the prediction esti- mates. The methods are as follows: Step 5, Case C. Method 1 For each site in the application group, calculate the prediction as a weighted average of the predictions from the multiple calibrated CPMs from the application group. The calibration of the CPMs is as presented in Step 4B-1-1. For example, in the case of two CPMs: [ ]( )= × + − ×Weighted average of predictions 11 2a p a pCPM CPM where pCPM1 = prediction from calibrated CPM1 pCPM2 = prediction from calibrated CPM2 a = parameter between 0 and 1 (chosen by trial and error) Use The Calibrator to assess the performance based on the following GOF measures: modi- fied R2, dispersion parameter (k), CV(C), MAD, CURE plots and related measures (Max_ Cure, Max_DCure, Avg_DCure, outside 95% CI). These are described in Step 4A-4. The value of a that gives the best GOF statistics for the application dataset can be chosen.

Reliability Associated with Crash Prediction Models Estimated for Other Facility Types 87   Step 5, Case C. Method 2 Estimate a combined CPM for the calibrated CPMs from the application group. For example, in the case of two CPMs: ( )= × × ×CPM_application_group 1 exp 2a CPM c CMPb where CPM1 and CPM2 = equations corresponding to the two calibrated CPMs a, b, c = parameters to be estimated using negative binomial regression Example Application—Procedure: Scenario 6 Predicting using CPMs estimated for other facility types (application sites have characteristics that are not represented by CPM estimation sites and vice versa) Question: The California freeway crash, traffic, and roadway characteristics data stored in the HSIS (www.hsisinfo.org), for the 2005–2014 period, were used to estimate CPMs for the following facility types: • Rural 4-lane, flat terrain • Urban 6-lane, flat terrain • Rural 4-lane, rolling terrain • Urban 6-lane, rolling terrain The data were categorized based on number of lanes, terrain, and area types (rural or urban areas). The crash types considered included: Total crashes, SV crashes, and MV crashes. Ramp influence areas (based on 0.3 miles on either side of a ramp) were excluded. Short segments less than 0.01 miles were also excluded. Data were not sufficient to estimate CPMs for the following facility types: • Rural 6-lane, flat terrain • Urban 4-lane, flat terrain • Rural 6-lane, rolling terrain • Urban 4-lane, rolling terrain The question is how well the CPMs estimated based on one of the facility types in the estimation group can predict the number of crashes for a different facility type in the application group. Outline of Solution Step 1. Define the Application Facility Type The application facility types in this example application are as follows: • Rural 6-lane, flat terrain • Urban 4-lane, flat terrain • Rural 6-lane, rolling terrain • Urban 4-lane, rolling terrain

88 Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results Step 2. Identify Estimation Facility Type(s) The estimation facility types in this example application are as follows: • Rural 4-lane, flat terrain • Urban 6-lane, flat terrain • Rural 4-lane, rolling terrain • Urban 6-lane, rolling terrain Step 3. Assemble the Data Needed for the Procedure Data summaries are shown in Table 26 through Table 30. Each facility type in the applica- tion group was matched to a suitable sub-group of facility types for which CPMs are available (Table 26). For example, estimation Group 1 consists of rural 4-lane, flat terrain highways and urban 6-lane, flat terrain highways; and the respective application Group 1 consists of rural 6-lane, flat terrain highways. The summary statistics for Group 1 and Group 2 are available in Table 27 and Table 28, respectively, and the summary statistics for Group 3 and Group 4 are provided in Table 29 and Table 30, respectively. For Group 1 and Group 2, the CPMs for SV, MV, and Total crashes were developed using a negative binomial model with data from rural 4-lane, flat terrain highways, and data from urban 6-lane, flat terrain highway segments, respectively. These groups are called the estimation group. The CPMs were to predict the crashes in rural 6-lane, flat terrain highways for Group 1, and urban 4-lane, flat terrain highways for Group 2 (application groups), after calibration. Step 4C, Case C. Multiple Similar Facility Types with CPMs with Other Variables in Addition to Variable AADT This example application represents Case C where there are multiple similar facility types with CPMs with other variables in addition to variable AADT. The CPMs were calibrated as described in Option 1, Step 4B-1-1. Group (estimation group) CPMs (application group) Crash Types Facility Type Segments Facility Type Segments Group 1 Rural 4-lane, flat terrain 1,075 Rural 6-lane, flat terrain 102 SV, MV, TotalUrban 6-lane, flat terrain 437 Group 2 Rural 4-lane, flat terrain 1,075 Urban 4-lane, flat terrain 428 SV, MV, TotalUrban 6-lane, flat terrain 437 Group 3 Rural 4-lane, rolling terrain 421 Rural 6-lane, rolling terrain 58 SV, MV, TotalUrban 6-lane, rolling terrain 253 Group 4 Rural 4-lane, rolling terrain 421 Urban 4-lane, rolling terrain 263 SV, MV, TotalUrban 6-lane, rolling terrain 253 Facility types used for estimating CPMs Facility types used for applying the estimated Table 26. Facility types by area type, terrain, and number of lanes used in the example application—Scenario 6: Predicting using CPMs estimated for other facility types.

Variable (estimation group) Data 1a: Urban 6-lane, flat terrain for CPM development (437 segments) Data 1b: Rural 4-lane, flat terrain for CPM development (1,075 segments) Min Max Mean St.dev Sum Min Max Mean St.dev Sum Segment length (mi) 0.011 2.46 0.19 0.26 83.97 0.01 9.02 0.56 0.82 605.81 SV crashes 0 143 8.67 13.30 3,788 0 104 8.77 13.20 9,424 MV crashes 0 288 20.49 29.44 8,954 0 164 7.83 14.36 8,419 Total crashes 0 345 29.16 39.61 12,742 0 231 16.60 25.97 17,843 AADT 6,580 262,079 98,471 15,275 NA 1,590 98,913 26,965 10,022 NA Median width (ft) 3 99 46.06 9.69 NA 0 99 81.83 13.56 NA Shoulder width (ft) 0.00 32.00 16.35 2.10 NA 0.00 33.00 15.99 1.96 NA Design speed (mph) 45 70 69.31 1.29 NA 50 70 69.80 0.93 NA Table 27. Summary statistics for the estimation datasets used for Group 1 and Group 2 in the example application—Scenario 6: Predicting using CPMs estimated for other facility types. Variable (application group) Group 1: Rural 6-lane, flat terrain, for applying the estimated CPMs (102 segments) Group 2: Urban 4-lane, flat terrain, for applying the estimated CPMs (428 segments) Min Max Mean St.dev Sum Min Max Mean St.dev Sum Segment length (mi) 0.01 2.00 0.38 0.45 39.01 0.01 2.23 0.20 0.27 86.82 SV crashes 0 122 13.28 18.99 1,355 0 67 5.37 8.59 2,299 MV crashes 0 178 20.39 32.58 2,080 0 168 9.86 18.73 4,221 Total crashes 0 244 33.68 49.61 3,435 0 235 15.23 25.04 6,520 AADT 27,870 128,100 70,586 19,207 NA 4,122 226,400 55,343 12,476 NA Median width (ft) 20 84 42.89 8.33 NA 3 99 51.04 11.90 NA Shoulder width (ft) 0.00 23.00 17.50 2.15 NA 0.00 33.00 13.64 2.11 NA Design speed (mph) 65 70 69.89 0.46 NA 35 70 68.51 1.92 NA Table 28. Summary statistics for the application datasets used for Group 1 and Group 2 in the example application—Scenario 6: Predicting using CPMs estimated for other facility types.

Variable (estimation group) Data 2a: Rural 4-lane, rolling terrain, for CPM development (421 segments) Data 2b: Urban 6-lane, rolling terrain for CPM development (253 segments) Min Max Mean St.dev Sum Min Max Mean St.dev Sum Segment length (mi) 0.01 4.39 0.46 0.61 194.10 0.011 1.65 0.21 0.28 53.12 SV crashes 0 166 7.95 14.64 3,345 0 51 7.40 9.19 1,873 MV crashes 0 141 6.17 14.63 2,598 0 138 15.88 21.76 4,017 Total crashes 0 241 14.12 27.46 5,943 0 184 23.28 29.04 5,890 AADT 3,212 70,300 25,196 9,507 NA 10,001 212,261 94,112 17,779 NA Median width (ft) 4 99 68.33 19.60 NA 6 99 50.36 11.42 NA Shoulder width (ft) 0.00 36.00 15.02 1.80 NA 0.00 34.00 17.88 1.82 NA Design speed (mph) 60 70 69.48 1.20 NA 45 70 69.72 0.64 NA Table 29. Summary statistics for the estimation datasets used for Group 3 and Group 4 in the example application—Scenario 6: Predicting using CPMs estimated for other facility types. Variable (application group) Group 3: Rural 6-lane, rolling terrain, for applying the estimated CPMs (58 segments) Group 4: Urban 4-lane, rolling terrain highway for applying the estimated CPMs (263 segments) Min Max Mean St.dev Sum Min Max Mean St.dev Sum Segment length (mi) 0.01 4.26 0.35 0.72 20.18 0.01 1.72 0.23 0.29 59.51 SV crashes 0 88 9.16 15.57 531 0 93 7.03 11.27 1,850 MV crashes 0 127 15.97 24.83 926 0 128 10.48 17.21 2,755 Total crashes 0 215 25.12 39.55 1,457 0 169 17.51 26.55 4,605 AADT 15,865 179,200 59,796 14,696 NA 8,160 133,100 51,058 13,357 NA Median width (ft) 14 99 53.87 14.72 NA 4 99 44.60 13.26 NA Shoulder width (ft) 0.00 22.00 17.70 2.24 NA 0.00 32.00 12.06 2.92 NA Design speed (mph) 65 70 69.66 0.75 NA 45 70 68.70 1.87 NA Table 30. Summary statistics for the application datasets used for Group 3 and Group 4 in the example application—Scenario 6: Predicting using CPMs estimated for other facility types.

Reliability Associated with Crash Prediction Models Estimated for Other Facility Types 91   GOF measure results for Group 1 through Group 4, are listed in Table 31 through Table 34. For Group 1 (Table 31), the CPMs from urban 6-lane highways with flat terrain (Facility 1a) are much better in predicting the crashes for rural 6-lane highways with flat terrain, especially for MV and Total crashes. For Group 2 (Table 32), both the facility groups are not very effective in predicting crashes for urban 4-lane roadways with flat terrain, although the CPMs from urban 6-lane highways with flat terrain (Facility 1a) perform better. For Group 3 (Table 33), CPMs from both facility types (urban 6-lane, rolling terrain highways; and rural 4-lane, rolling terrain highways) provide promising results for predicting crashes on rural 6-lane highways with rolling terrain, while the CPMs from urban 6-lane highways with rolling terrain perform better especially for MV and Total crashes. For Group 4 (Table 34), the CPMs for SV crashes from both facility types perform very well in predicting crashes for urban 4-lane highways with rolling terrain. However, CPMs for MV and Total crashes from the two facility types are not highly effective in predicting crashes on urban 4-lane highways with rolling terrain. Overall, the GOF measure results indicate that there is not a consistent pattern in terms of facility types being most appropriate to serve as estimation group for a particular application group. In some cases, the estimation group was appropriate for a particular crash type but not for another crash type. Since the calibration of the multiple CPMs (Option 1) did not result in acceptable results, the practitioner proceeds to Option 2. The description of Option 2 is found under Case 4A. As described in Step 4C, Case C, the practitioner will proceed further to Option 3 through Option 5 to select which CPMs perform best and give the most reliable prediction estimates based on the assessment of the GOF measure results. Option 2 through Option 5 are not shown here for this example application, Scenario 6, Case C, and the descriptions of Option 1 and Option 3 through Option 5 are found under Case 4B. As noted in Step 4C, Case C, “. . . if after Option 5, the GOF statistics are still not satisfactory for any of the multiple facility types in the application group, the practitioner will proceed to Step 5, Case C.” Crash Type Number of Crashes k Modified R2 CV MAD Max_Cure Max_DCure Avg_DCure Outside 95% CI Data to Develop CPMs* SV 1,355 0.18 0.84 0.08 4.72 70.55 0 0 0 Facility 1a 0.19 0.78 0.08 5.10 82.06 8.10 0.41 14% Facility 1b MV 2,080 0.49 0.83 0.13 7.72 112.35 10.37 1.30 29% Facility 1a 0.75 0.57 0.16 11.57 424.24 262.20 92.26 84% Facility 1b Total 3,435 0.34 0.88 0.10 10.52 135.63 9.85 0.54 11% Facility 1a 0.49 0.59 0.12 17.84 610.66 375.62 119.32 83% Facility 1b * Facility 1a: California urban 6-lane, flat terrain highway; Facility 1b: California rural 4-lane, flat terrain highway. Table 31. GOF measure results for rural 6-lane, flat terrain (Group 1)—Scenario 6: Predicting using CPMs estimated for other facility types.

92 Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results * Facility 2a: California rural 4-lane, rolling terrain highway; Facility 2b: California urban 6-lane, rolling terrain highway. Crash Type Number of Crashes k Modified R2 CV MAD Max_Cure Max_DCure Avg_DCure Outside 95% CI Data to Develop CPMs* SV 1,850 0.60 0.57 0.09 4.15 126.42 20.92 0.71 14% Facility 2a 0.63 0.57 0.09 4.15 131.36 16.36 0.49 9% Facility 2b MV 2,755 1.01 -1.66 0.12 7.74 754.18 465.76 145.99 84% Facility 2a 0.83 0.65 0.11 5.72 261.77 95.96 19.63 44% Facility 2b Total 4,605 0.68 -0.24 0.09 10.70 764.83 398.52 129.15 88% Facility 2a 0.66 0.67 0.09 8.60 346.24 127.56 28.92 52% Facility 2b Table 34. GOF measure results for urban 4-lane, rolling terrain (Group 4)—Scenario 6: Predicting using CPMs estimated for other facility types. * Facility 1a: California urban 6-lane, flat terrain highway; Facility 1b: California rural 4-lane, flat terrain highway. Crash Type Number of Crashes k Modified R2 CV MAD Max_Cure Max_DCure Avg_DCure Outside 95% CI Data to Develop CPMs* SV 2,299 0.55 0.45 0.07 3.25 222.63 90.87 13.24 54% Facility 1a 0.65 0.13 0.08 3.75 375.29 211.77 68.35 73% Facility 1b MV 4,221 1.12 0.39 0.11 6.71 548.54 270.70 96.31 73% Facility 1a 1.36 -2.79 0.12 10.28 1,681.41 1,209.20 406.37 95% Facility 1b Total 6,520 0.83 0.43 0.09 8.98 728.09 393.66 139.18 70% Facility 1a 1.03 -1.14 0.10 12.96 2,092.56 1,579.80 529.32 92% Facility 1b Table 32. GOF measure results for urban 4-lane, flat terrain (Group 2)—Scenario 6: Predicting using CPMs estimated for other facility types. Crash Type Number of Crashes k Modified R2 CV MAD Max_Cure Max_DCure Avg_DCure Outside 95% CI Data to Develop CPMs* SV 531 0.96 0.73 0.26 4.72 62.47 0.00 0.00 0% Facility 2a 0.90 0.73 0.25 4.67 62.67 0.21 0.00 2% Facility 2b MV 926 1.71 0.41 0.32 11.05 186.41 51.83 5.82 29% Facility 2a 0.76 0.73 0.21 7.46 118.27 17.88 0.51 5% Facility 2b Total 1,457 1.45 0.72 0.29 12.85 242.92 84.79 6.58 22% Facility 2a 0.94 0.72 0.24 12.02 196.39 34.60 1.52 10% Facility 2b * Facility 2a: California rural 4-lane, rolling terrain highway; Facility 2b: California urban 6-lane, rolling terrain highway. Table 33. GOF measure results for rural 6-lane, rolling terrain (Group 3)—Scenario 6: Predicting using CPMs estimated for other facility types.