Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results (2021)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

9   Introduction The CPMs in HSM Part C include a SPF, one or more CMFs, and a local calibration factor (C). Part C (Section C.7) of the HSM describes four methods for estimating the average crash frequency for a site. The methods are indicated to provide different levels of predictive reli- ability. However, the HSM does not quantify the reduction in reliability associated with each of the four methods, so practitioners do not have the information they need to make an informed choice among the methods. These methods are identified in the following list in order of predic- tive reliability, with the most reliable method listed first: • Method 1. Apply the Part C CPM to evaluate the existing and proposed conditions. • Method 2. Apply the Part C CPM to evaluate the existing condition. Use a Part D CMF with the Part C CPM to evaluate the proposed condition. • Method 3. Apply a jurisdiction-specific SPF to evaluate the existing condition. Use a Part D CMF with this SPF to evaluate the proposed condition. • Method 4. Use observed crash frequency to evaluate the existing condition. Use a Part D CMF with the observed crash frequency to evaluate the proposed condition. Method 1 relates to the use of CPMs where there is a “balance” between the CMFs and the base conditions associated with the SPF. A balanced CPM application occurs when the set of CMFs used collectively matches all the SPF’s base conditions. Method 2 presents the situation where there is a lack of balance between the CMFs used and the SPF. In this situation, the Part C SPF base conditions do not include those associated with the Part D CMFs. Method 3 can be implemented in one of two ways. In the first way, there is a lack of bal- ance between the CMF obtained from Part D and the jurisdiction-specific SPF (similar to that described for Method 2). In the second way, there is a balance between the CMF from Part D and the jurisdiction-specific SPF. There is a variation of Methods 1 or 2 that can sometimes occur in the application. For this variation, one or more CMFs that are part of the Part C CPM are not used (i.e., omitted from calculations). This situation may occur when the practitioner is interested in using the CPM to evaluate a site but does not have ready access to the data needed for one or more of the other CMFs in the CPM. C H A P T E R 2 Quantifying the Reliability of CPM Estimates for Mismatches Between Crash Modification Factors and SPF Base Conditions

10 Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results Method 4 is not based on the use of an SPF. Rather, the observed crash frequency for the site is used to estimate the expected crash frequency. Thus, the issue of CMF-SPF balance does not apply. For this reason, the reliability of Method 4 is not addressed in this Guide. Three cases for Scenario 1 are identified where the practitioner’s selection of the CMFs used with a CPM results in a reduction in estimate reliability. Table 1 contains the three cases for Scenario 1. Procedures to Assess Potential Reliability The procedures for quantifying CPM estimate reliability for Cases A, B, and C in Scenario 1 are based on theoretically derived equations that predict the bias in, and increased variance of, the CPM estimate of predicted crash frequency. The predicted bias and increased variance can be used in two ways: (1) they can be used to correct the CPM estimates by removing the bias and increased variance; or (2) the CPM estimates can be retained (i.e., bias not removed and variance unchanged) and used only to quantify their overall reliability. The former use may be appropriate when evaluating a specific site. The latter use may be appropriate when making policy decisions about CPM development and application. Procedural Steps and Example Application This section describes the procedures for quantifying the bias and added uncertainty (i.e., variance) associated with each of the cases in Table 1. The procedure associated with each case is described in a separate subsection. An example application illustrating each case is found immediately after each case procedure. Case Description Associated HSM Method CMF-SPF Balance* 1A CMF from HSM Part D, or from another source (e.g., FHWA CMF Clearinghouse), consistent with SPF base conditions used with jurisdiction-specific SPF 3 Yes 1B One or more CMFs used in the CPM do not have a corresponding base condition in the SPF 2, 3 No 1C One or more CMFs are not used in the CPM yet the corresponding base condition exists in the SPF 1, 2 No * A balanced CPM application occurs when the set of CMFs used collectively matches all the SPF’s base conditions. Table 1. Scenario 1: Mismatch between CMFs and SPF base conditions. Procedure: Scenario 1, Case A CMF from HSM Part D, or from another source (e.g., FHWA CMF Clearinghouse), consistent with SPF base conditions used with jurisdiction-specific SPF For this CPM application, a CMF from HSM Part D, or from another source (e.g., FHWA CMF Clearinghouse) is used with a jurisdiction-specific SPF. There is a balance between the CMF and the base conditions associated with the SPF.

Quantifying the Reliability of CPM Estimates for Mismatches Between Crash Modification Factors and SPF Base Conditions 11   In Scenario 1, Case A, the CMF obtained from HSM Part D or another source represents the “CMF of interest.” It describes the safety effect of treatment of interest. The GOF measures that are computed describe the reliability of the estimated average crash frequency when the treat- ment is applied to one or more sites of interest. There are seven steps in this procedure: Step 1. Assemble the data needed to apply the procedure. Step 2. Compute estimation coefficient. Step 3. Compute bias adjustment factor. Step 4. Compute the predicted crash frequency for site of interest. Step 5. Compute the unbiased predicted crash frequency for site of interest. Step 6. Compute the increased root mean square error and CV. Step 7. Compute the amount of bias. The following steps describe data needs, equations, variable estimation, selected GOF mea- sures, and outcome related to the quantitative assessment of the degree of reliability. Step 1. Assemble the Data Needed to Apply the Procedure The data needed to apply the procedure are listed in Table 2. The data needed to apply the SPF are not listed. If there are several sites of interest, the CV (CVI) and bias percent (Bias) computed in Step 6 and Step 7, respectively, were derived to have the characteristic that they are insensitive to the value of the predicted crash frequency (NP). As a result, the predicted CV and bias percent rea- sonably describe the reliability of the predicted crash frequency for each of the sites of interest, regardless of their AADT or segment length. Therefore, for purposes of the reliability evaluation when there are several sites of interest, a typical site representing the collective set of sites can be used to compute the predicted crash frequency in Step 4. Typical values of AADT, segment length, and so forth can be used to apply the SPF in Step 4. The average of the independent variable at the sites used to estimate the SPF (X– SPF) can be obtained from one of several sources. It can be obtained from the SPF development report if that report provides summary statistics of the variable X in the data used to estimate the SPF. Alternatively, the average of X can be obtained through access to the original database used to Data and Relationships Potential Data Sources Input values needed for jurisdiction-specific SPF Agency files Overdispersion parameter for jurisdiction-specific SPF kreported SPF development report CMF for geometric design element or traffic control feature of interest CMFD HSM Part D; CMF Clearinghouse Average independent variable value at sites used to estimate the SPF SPF development report, archived SPF database, agency files, or field measurements Standard deviation of the independent variable at sites used to estimate NSPF,j-s σx,b SPF development report, archived SPF database, agency files, or field measurements Average independent variable value associated with CMF of interest at sites of interest Site data from agency files, plans, or field measurements Standard deviation of the independent variable at sites of interest σx,n Site data from agency files, plans, or field measurements Table 2. Required data to apply the procedure for Scenario 1, Case A.

12 Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results estimate the SPF. Finally, if this database is not available or it does not contain the variable of interest X, then it may be possible to identify the regions from which the data were obtained to develop the SPF and obtain values of the variable X at a representative set of sites. These values would be obtained from agency files or field measurements. The value X–SPF would then be com- puted using this representative set of sites. If the treatment is discrete (e.g., “add beacon”), an indicator variable is used to indicate treatment presence (where 1 corresponds to treatment present, and zero corresponds to treatment not present). The average X–SPF is then computed using the 1 or zero values at the collective set of study sites. The standard deviation of the independent variable at the sites used to estimate the SPF (σx,b) is obtained using the same data source that was used to obtain X–SPF. The practitioner may be seeking the reliability evaluation of one site of interest or a group of sites of interest: • When One Site of Interest Is Being Evaluated. The average of the independent variable value associated with the CMF of interest (X–) can be obtained in several ways. One source of this information is agency files (e.g., design plans) for the site that is planned for treatment (or has been treated). Another source is field measurement at the site that is planned for treatment (or has been treated). Regardless of the source, X– equals the variable value for the site of interest. If the treatment is discrete (e.g., “add beacon”), an indicator variable is used to indicate treat- ment presence (where 1 corresponds to treatment present, and zero corresponds to treatment not present). The standard deviation of the independent variable at the sites of interest (σx,n) is equal to 0.0 if X is known with certainty. If X has been judged or measured, then the standard deviation should equal a small positive value, reflecting the practitioner’s uncertainty in the judgment or measurement. • When Several Sites of Interest Are Being Evaluated. The average of the independent variable value associated with the CMF of interest (X–) can be obtained in several ways. One source of this information is agency files (e.g., design plans) for the sites that are planned for treatment (or have been treated). Another source is field measurements at the sites that are planned for treatment (or have been treated). Regardless of the source, if the treatment is applied uni- formly to the sites of interest as part of a planned change, then X has the same value at each site and X– equals X. Alternatively, if the treatment is not applied uniformly to all the sites of interest, then X– equals the value of X averaged for all sites. If the treatment is discrete (e.g., “add beacon”), an indicator variable is used to indicate treatment presence (where 1 corre- sponds to treatment present, and zero corresponds to treatment not present). The average X– is then computed using the 1 or zero values for the collective set of sites. The standard deviation of the independent variable at the sites of interest (σx,n) is obtained using the same data source that was used to obtain X–. Step 2. Compute Estimation Coefficient There are three sub-steps in this computation: 1. Compute the value of the CMF of interest for the sites used to estimate the SPF [CMFD (X – SPF)]. If the CMF of interest corresponds to a discrete treatment (e.g., “add beacon”), then the CMF value is computed using the corresponding site data, where each site with the treatment is assigned the CMF value and each site without the treatment is assigned a value of 1.0. For example, if the CMF for “add beacon” is 0.95 and 30% of the sites used to estimate the SPF have the beacon, then the value of CMFD (X – SPF) is 0.985 [= 0.95 × 0.30 + 1.0 × (1 − 0.30)]. If the CMF of interest corresponds to a continuous treatment (e.g., lane width), then the CMF value used is computed using the CMF function with the average value of X (i.e., X–SPF).

Quantifying the Reliability of CPM Estimates for Mismatches Between Crash Modification Factors and SPF Base Conditions 13   2. Compute the value of the CMF of interest for the sites of interest [CMFD(X –)]. If the CMF of interest corresponds to a discrete treatment (e.g., “add beacon”), then the CMF value is computed using the corresponding site data, where each site with the treatment is assigned the CMF value and each site without the treatment is assigned a value of 1.0. If all the sites of interest are planned to have the beacon added, then the value of CMFD(X –) is 0.95 [= 0.95 × 1.0 + 1.0 × (1 − 1.0)]. If the CMF of interest corresponds to a continuous treatment (e.g., lane width), then the CMF value used is computed using the CMF function with the average value of X (i.e., X–). If X– is equal to X–SPF, then multiply X – by 1.01 and use the resulting value of X– to compute CMFD(X –). Use this computed CMF value and the resulting value of X– in the equation provided in sub-step 3. Multiplication by the constant 1.01 is used to avoid division by zero in this equation. 3. Use the following equation to compute the estimation coefficient, b. [ ] [ ]( ) ( )= − − b Ln CMF X Ln CMF X X X D D SPF SPF where b = estimation coefficient CMFD = CMF for geometric design element, or traffic control feature of interest X– = average independent variable value associated with CMF of interest at sites of interest X– SPF = average independent variable value at sites used to estimate the SPF CMFD(X –) = CMF value associated with X– CMFD(X – SPF) = CMF value associated with X – SPF Step 3. Compute Bias Adjustment Factor Determine the correction term ct based on the sign of the estimation coefficient b. If b is greater than zero, then ct equals 1.120; otherwise, ct equals 0.880. Then, use this term in the fol- lowing equation to compute the bias adjustment factor fA. { }= + σ − σf b cA x n x b t1 0.5 2 ,2 ,2 with = >1.120 if 0; 0.880 otherwisec bt where fA = bias adjustment factor for Scenario 1, Case A σ2x,n = variance of the independent variable at sites of interest σ2x,b = variance of the independent variable at sites used to estimate the jurisdiction-specific SPF b = estimation coefficient ct = correction term Step 4. Compute the Predicted Crash Frequency for Site of Interest The following equation is used to compute the predicted crash frequency for the site of interest. ( )= ×−,N N CMF Xp SPF j s D

14 Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results where Np = predicted average crash frequency, crashes/year NSPF,j-s = predicted crash frequency for site with base conditions that are in balance with the CMFs (jurisdiction-specific) CMFD(X –) = CMF value associated with X– Step 5. Compute the Unbiased Predicted Crash Frequency for Site of Interest The following equation is used to compute the unbiased predicted crash frequency for the site of interest. =,N N fp true p A where Np,true = predicted true crash frequency, crashes/year Np = predicted average crash frequency, crashes/year fA = bias adjustment factor for Scenario 1, Case A Step 6. Compute the Increased Root Mean Square Error and CV There are four sub-steps in this computation: 1. Compute the error in the predicted crash frequency using the following equation: = −e N Np p true, where e = error in predicted crash frequency Np,true = predicted true crash frequency, crashes/year Np = predicted average crash frequency, crashes/year 2. Compute the absolute difference in the change in variance of the predicted value using the following equation: ( ) ( )σ = × − ×k N k Nabs reported p reported p true2 2 ,2 where σ2abs = absolute difference of the change in variance of the predicted value kreported = reported overdispersion parameter for the jurisdiction-specific SPF Np,true = predicted true crash frequency, crashes/year Np = predicted average crash frequency, crashes/year 3. Compute the increased root mean square error using the following equation: [ ]σ = σ +, 2 2 0.5ee I abs

Quantifying the Reliability of CPM Estimates for Mismatches Between Crash Modification Factors and SPF Base Conditions 15   where σe,I = increased root mean square error σ2abs = absolute difference of the change in variance of the predicted value e = error in predicted crash frequency 4. Compute the CV using the following equation: = σ CV NI e I p true , , where CVI = coefficient of variation for the increased root mean square error σe,I = increased root mean square error Np,true = predicted true crash frequency, crashes/year Step 7. Compute the Amount of Bias The percent bias in the predicted crash frequency is computed using the following equation: = × − 100 , , Bias N N N p p true p true where Bias = the percent bias in the reported value Np,true = predicted true crash frequency, crashes/year Np = predicted average crash frequency, crashes/year Example Application—Scenario 1, Case A CMF from HSM Part D, or from another source (e.g., FHWA CMF Clearinghouse), consistent with SPF base conditions used with jurisdiction-specific SPF Question: An agency desires to conduct a safety evaluation of a rural two-lane, two-way road segment. The segment is of interest because it has a relatively narrow lane width. The results of the evaluation will be used to determine the site’s potential for safety improvement. From a prior project, the agency has developed a jurisdiction-specific SPF for rural two-lane, two-way road segments. The SPF predicts the frequency of crashes of all types and severities. The typical lane width for rural two-lane, two-way road segments in the agency’s jurisdiction is 12 ft. The agency has obtained a CMF from HSM Part D, Chapter 13 for lane width. This CMF is applicable to rural two-lane, two-way road segments. The CMF is a function of lane width and segment AADT. The practitioner desires to assess the reliability of the predicted crash frequency given that the lane width CMF used is obtained from HSM Part D.

16 Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results Outline of Solution Step 1. Assemble the Data Needed to Apply the Procedure The data needed to apply the procedure are listed in Table 3. The data needed to apply the SPF and the HSM Part D, CMF are also needed but not shown here. A review of agency files indicates that the segment of interest has an AADT of 10,000 vehicles/ day and a segment length of 0.5 miles. The SPF was used (with an AADT of 10,000 vehicles/day and segment length of 0.5 miles) to compute the predicted crash frequency for a segment with 12-ft lane width. The computed value of NSPF,j-s is 1.34 crashes/year. As indicated, the SPF was developed using sites having a typical lane width of 12 ft. A review of the SPF development report prepared by the agency indicated that the sites used to estimate the SPF had an average lane width of 12 ft and a standard deviation of 2.0 ft. Thus, the average independent variable value X–SPF equals 12 ft and the standard deviation of lane width at these sites σx,b equals 2.0 ft. Thus, based on this SPF development approach, the base-condition lane width is defined as 12 ft. The segment of interest was investigated using measurements from aerial photographs. It was learned that the average lane width at the site X– is 10 ft. However, measurements along the segment length suggested that the standard deviation σx,n of the segment’s lane width is 0.10 ft. Step 2. Compute Estimation Coefficient Three sub-steps are as follows: 1. The value of the CMF of interest at the segments used to estimate the SPF [CMFD (X – SPF)] is obtained from HSM Part D. For an AADT of 10,000 vehicles/day, the CMF for an average lane width of 12 ft is 1.00 [i.e., CMFD (12) = 1.00]. 2. The value of the CMF of interest for the segment of interest [CMFD (X –)] is also obtained from HSM Part D. For the stated AADT, the CMF for an average lane width of 10 ft is 1.30 [i.e., CMFD (10) = 1.30]. Required Data and Relationships Potential Data Sources Input values needed for jurisdiction-specific SPF AADT = 10,000 vehicles/day Segment length = 0.5 miles NSPF, j-s = 1.34 crashes/year Overdispersion parameter for jurisdiction-specific SPF kreported kreported = 0.472 HSM Part D, CMF for geometric design element or traffic control feature of interest CMFD HSM Part D, Table 13-2 Average independent variable value at sites used to estimate the SPF Average lane width = 12 ft Standard deviation of the independent variable at sites used to estimate NSPF,j-s σx,b Standard deviation σx,b = 2.0 ft Average independent variable value associated with CMF of interest at sites of interest Average lane width = 10 ft Standard deviation of the independent variable at sites of interest σx,n Standard deviation σx,n = 0.1 ft Table 3. Required data to apply the procedure for Scenario 1, Case A, example application.

Quantifying the Reliability of CPM Estimates for Mismatches Between Crash Modification Factors and SPF Base Conditions 17   3. These two CMF values are used in the following equation to compute the estimation coefficient: [ ] [ ]( ) ( )= − − = −0.131b Ln CMF X Ln CMF X X X D D SPF SPF Step 3. Compute Bias Adjustment Factor Initially, equation ct =1.120 if b > 0; otherwise 0.880 is used to determine that a correction factor ct value of 0.88 is needed. Then to compute the bias adjustment factor fA, the following equation is used: { }= + σ − σ =f b cA x n x b t1 0.5 0.9702 ,2 ,2 Step 4. Compute the Predicted Crash Frequency for Site of Interest To compute the predicted average crash frequency Np, the SPF value NSPF,j-s (1.34 crashes/ year) and the CMF value CMFD(X –) [i.e., CMFD (10) = 1.30] are used in the following equation: ( )= × =− 1.742 crashes/year,N N CMF Xp SPF j s D Step 5. Compute the Unbiased Predicted Crash Frequency for Site of Interest To compute the predicted true crash frequency Np,true, the predicted crash frequency Np and bias adjustment factor fA are used in the following equation: = =1.796 crashes/year,N N fp true p A Step 6. Compute the Increased Root Mean Square Error and CV Four sub-steps are as follows: 1. To compute the error in the predicted crash frequency e, the predicted crash frequency Np and the predicted true crash frequency Np,true are used in the following equation: 0.054 crashes/year,= − = −e N Np p true 2. To compute the absolute difference in the change in variance of the predicted value σ2abs, the reported overdispersion parameter kreported, the predicted crash frequency Np, and the pre- dicted true crash frequency Np,true are used in the following equation: 0.09062 2 ,2k N k Nabs reported p reported p true( ) ( )σ = × − × = 3. To estimate the increased root mean square error σe,I, the predicted value σ2abs and the pre- dicted error e are used in the following equation: [ ]σ = σ + = 0.306 crashes/year, 2 2 0.5ee I abs If the variance of the predicted value were computed, it should be increased by the square of σe, I. to include the error in the predicted value Np.

18 Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results 4. To compute the CV for the increased root mean square error CVI, the predicted value σe,I and the predicted true crash frequency Np,true are used in the following equation: = σ =CV NI e I p true 0.17, , This CVI means that there is a relatively large amount of error-related variability in the pre- dicted crash frequency. As noted, values over 0.20 are considered to be unreliable for most applications. Step 7. Compute the Amount of Bias To compute the percent bias in the predicted crash frequency Bias, the predicted crash fre- quency Np and the predicted true crash frequency Np,true are used in the following equation: = × − = −100 3.0%, , Bias N N N p p true p true As noted, an absolute value of bias in excess of 10% is considered to be unreliable for most applications. The percent bias measure indicates the relative error in the prediction. Based on these results, the practitioner concludes that the use of the lane width CMF obtained from HSM Part D, does add bias to the predicted crash frequency, and it does lead to a relatively large amount of error-related variability in the predicted crash frequency. Although neither the bias nor the increased error exceeds their respective thresholds, which would indicate unreliable results, the practitioner decides to search for other lane width CMFs that will yield more reliable predictions, and if a more reliable CMF is not found, to use this CMF with due caution, knowing that there is uncertainty of the predicted crash frequency. Procedure: Scenario 1, Case B CMFs from HSM Part D, or from another source (e.g., FHWA CMF Clearinghouse) do not have a corresponding base condition in the SPF For this CPM application, one or more CMFs used with a CPM do not have a corresponding base condition in the SPF. These CMFs are called herein “external CMFs” because their variables were not considered when the SPF was developed, and its base conditions were established. In this Scenario 1, Case B, an HSM Part D, CMF is used with an HSM Part C, CPM, and the HSM Part D, CMF’s variables are not included in the CPM’s base conditions. The procedure described here is sufficiently general that it can be applied to CPMs from other sources, such as a CPM developed for a specific jurisdiction. There are eight steps in this procedure: Step 1. Assemble the data needed to apply the procedure. Step 2. Compute estimation coefficient. Step 3. Compute bias adjustment factor. Step 4. Compute the predicted crash frequency for site of interest. Step 5. Compute the unbiased predicted crash frequency for site of interest. Step 6. Compute the unbiased overdispersion parameter for the CPM with the external CMF. Step 7. Compute the increased root mean square error and CV. Step 8. Compute the amount of bias.

Quantifying the Reliability of CPM Estimates for Mismatches Between Crash Modification Factors and SPF Base Conditions 19   Each step is described in terms of data needs, equations, variable estimation, selected GOF measures, and outcome related to the quantitative assessment of the degree of reliability. Step 1. Assemble the Data Needed to Apply the Procedure The data needed to apply the procedure are listed in Table 4. The data needed to apply the SPF are not listed here. If there are several sites of interest, the CV (CVI) and bias percent (Bias) computed in Step 7 and Step 8, respectively, were derived to have the characteristic that they are insensitive to the value of the predicted crash frequency NP. As a result, the predicted CV and bias percent reason- ably describe the reliability of the predicted crash frequency for each site of interest, regardless of their AADT or segment length. Therefore, for purposes of the reliability evaluation when there are several sites of interest, a typical site representing the collective set of sites can be used to compute the predicted crash frequency in Step 4. Typical values of AADT, segment length, and so forth can be used to apply the SPF in Step 4. The average of the independent variable at the sites used to estimate the CPM (X–CPM) can be obtained from one of several sources. It can be obtained from the CPM development report if that report provides summary statistics of the variable X in the data used to estimate the CPM. Alternatively, the average of X can be obtained through access to the original database used to estimate the CPM. Finally, if this database is not available or it does not contain the variable of interest X, then it may be possible to identify the regions from which the data were obtained to develop the CPM and obtain values of the variable X at a representative set of sites. These values would be obtained from agency files or field measurements. The value X–CPM would then be computed using this representative set of sites. If the treatment is discrete (e.g., “add beacon”), an indicator variable is used to indicate treatment presence (where 1 corresponds to treatment present, and zero corresponds to treatment not present). The average X–CPM is then computed using the 1 or zero values at the collective set of study sites. The standard deviation of the independent variable at the sites used to estimate the CPM (σx,b) is obtained using the same data source that was used to obtain X–CPM. Required Data and Relationships Potential Data Sources Input values needed for CPM Agency files Overdispersion parameter for CPM kreported HSM Part C HSM Part C, CMF for geometric design element or traffic control feature i CMFi HSM Part C External CMF for geometric design element or traffic control feature of interest CMFex HSM Part D; CMF Clearinghouse Average independent variable value at sites used to estimate the CPM CPM development report, archived CPM database, agency files, or field measurements Standard deviation of the independent variable at sites used to estimate the CPM σx,b CPM development report, archived CPM database, agency files, or field measurements Average independent variable value associated with external CMF at sites of interest Site data from agency files, plans, or field measurements Standard deviation of the independent variable at sites of interest σx,n Site data from agency files, plans, or field measurements Table 4. Required data to apply the procedure for Scenario 1, Case B.

20 Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results The practitioner may be seeking the reliability evaluation of one site of interest or a group of sites of interest: • When One Site of Interest Is Being Evaluated. The average of the independent variable value associated with the external CMF (X–) can be obtained in several ways. One source of this information is agency files (e.g., design plans) for the site that is planned for treatment (or has been treated). Another source is field measurements at the site that is planned for treatment (or has been treated). Regardless of the source, X– equals the variable value for the site of interest. If the treatment is discrete (e.g., “add beacon”), an indicator variable is used to indicate treatment presence (where 1 corresponds to treatment present, and zero corresponds to treatment not present). The standard deviation of the independent variable at the site of interest (σx,n) is equal to 0.0 if X is known with certainty. If X has been judged or measured, then the standard deviation should equal a small positive value reflecting the practitioner’s uncertainty in the judgment or measurement. • When Several Sites of Interest Are Being Evaluated. The average of the independent variable value associated with the external CMF (X–) can be obtained in several ways. One source of this information is agency files (e.g., design plans) for the sites that are planned for treatment (or have been treated). Another source is field measurements at the sites that are planned for treatment (or have been treated). Regardless of the source, if the treatment is applied uniformly to the sites of interest as part of a planned change, then X has the same value at each site and X– equals X. Alternatively, if the treatment is not applied uniformly to all the sites of interest, then X– equals the value of X averaged for all sites. If the treatment is discrete (e.g., “add beacon”), an indicator variable is used to indicate treatment presence (where 1 corresponds to treatment present, and zero corresponds to treatment not present). The average X– is then computed using the 1, zero values for the collective set of sites. The standard deviation of the independent variable at the sites of interest (σx,n) is obtained using the same data source that was used to obtain X–. Step 2. Compute Estimation Coefficient There are three sub-steps in this computation: 1. Compute the value of the external CMF for the sites used to estimate the CPM [CMFex (X – CPM)]. If the external CMF corresponds to a discrete treatment (e.g., “add beacon”), then the CMF value is computed using the corresponding site data, where each site with the treatment is assigned the CMF value and each site without the treatment is assigned a value of 1.0. For example, if the CMF for “add beacon” is 0.95 and 30% of the sites used to estimate the CPM have the beacon, then the value of CMFex (X – CPM) is 0.985 [= 0.95 × 0.30 + 1.0 × (1 − 0.30)]. If the external CMF corresponds to a continuous treatment (e.g., lane width), then the CMF value used is computed using the CMF function with the average value of X (i.e., X–CPM). 2. Compute the value of the external CMF for the sites of interest [CMFex(X –)]. If the external CMF corresponds to a discrete treatment (e.g., “add beacon”), then the CMF value is com- puted using the corresponding site data, where each site with the treatment is assigned the CMF value and each site without the treatment is assigned a value of 1.0. If all the sites of interest plan to have the beacon added, then the value of CMFex(X –) is 0.95 [= 0.95 × 1.0 + 1.0 × (1 − 1.0)]. If the external CMF corresponds to a continuous treatment (e.g., lane width), then the CMF value used is computed using the CMF function with the average value of X (i.e., X–). If X– is equal to X–CPM then multiply X – by 1.01 and use the resulting value of X– to com- pute CMFex (X –). Use this computed CMF value and the resulting value of X– in the equation provided in sub-step 3. Multiplication by the constant 1.01 is used to avoid division by zero in this equation.

Quantifying the Reliability of CPM Estimates for Mismatches Between Crash Modification Factors and SPF Base Conditions 21   3. Use the following equation to compute the estimation coefficient b. [ ] [ ]( ) ( )= − − b Ln CMF X Ln CMF X X X ex ex CPM CPM where b = estimation coefficient CMFex = external CMF (i.e., not associated with the SPF’s base conditions) X– = average independent variable associated with CMF of interest at sites of interest X– CPM = average independent variable value at sites used to estimate the CPM CMFex(X –) = external CMF value associated with X– CMFex(X – CPM) = external CMF value associated with X – CPM Step 3. Compute Bias Adjustment Factor Determine the correction term ct based on the sign of the estimation coefficient b. If b is greater than zero, then ct equals 1.120; otherwise, ct equals 0.880. Then, use this term in the fol- lowing equation to compute the bias adjustment factor fB. = + σ1 0.5 2 ,2f b cB x n t with = >1.120 if 0; 0.880 otherwisec bt where fB = bias adjustment factor for Scenario 1, Case B σ2x,n = variance of the independent variable at sites of interest b = estimation coefficient ct = correction term Step 4. Compute the Predicted Crash Frequency for Site of Interest The following equation is used to compute the predicted crash frequency for the site of interest. ( )( )= × × × × ×. . .1N C N CMF CMF CMF Xp SPF n ex where Np = predicted average crash frequency, crashes/year C = local calibration factor NSPF = predicted crash frequency for site with base conditions, crashes/year CMFi = HSM Part C, CMF for geometric design element, or traffic control feature i (i = 1 to n) n = total number of CMFs CMFex (X –) = value associated with X– (i.e., not associated with the SPF’s base conditions)

22 Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results Step 5. Compute the Unbiased Predicted Crash Frequency for Site of Interest The following equation is used to compute the unbiased predicted crash frequency for the site of interest. ( ) ( )= × ×,N N CMF X f CMF Xp true p ex CPM B ex where Np,true = predicted true crash frequency, crashes/year Np = predicted average crash frequency, crashes/year fB = bias adjustment factor for Scenario 1, Case B Step 6. Compute the Unbiased Overdispersion Parameter for the CPM with the External CMF Compute the adjustment factor Δ0 using the relevant equation below, then, compute the over- dispersion parameter for the CPM with the external CMF using the following equation: = − σ ∆1.13, 2 ,2k k bp true reported x b o with [ ]( )∆ = − × −1 0.10 2 5, 1min po where kp,true = predicted true overdispersion parameter for CPM with external CMF kreported = reported overdispersion parameter for the CPM b = estimation coefficient σ2x,b = variance of the independent variable at sites used to estimate the CPM p = number of empirically derived constants in (a) the CMFs associated with the CPM, and (b) the external CMF (exclude those in the SPF for intercept, AADT, and seg- ment length) (refer to the following text) Δ0 = adjustment factor for the incremental effect of additional empirical coefficients The number of empirically derived constants p is determined by inspecting the CMFs in the CPM. Regression constants in the SPF (e.g., intercept, AADT coefficient, segment length coefficient) are not considered when determining the value of p. If the CMF is associated with a discrete treatment (e.g., add beacon), then the CMF is often represented by a single empirical constant. If the CMF is associated with a continuous variable (e.g., lane width), then it is often a function that includes one or more empirical constants. In general, there is at least one empiri- cally derived constant for each CMF used to develop the CPM plus one for the external CMF. For practical applications of this procedure, the variable p can be estimated as equal to the number of CMFs used in the equation in Step 4 plus one for the external CMF (i.e., p = n + 1). Step 7. Compute the Increased Root Mean Square Error and CV There are four sub-steps in this computation: 1. Compute the error in the predicted crash frequency using the following equation: = − ,e N Np p true

Quantifying the Reliability of CPM Estimates for Mismatches Between Crash Modification Factors and SPF Base Conditions 23   where e = error in predicted crash frequency Np = predicted average crash frequency, crashes/year Np,true = predicted true crash frequency, crashes/year 2. Compute the absolute difference in the change in variance of the predicted value using the following equation: k N k Nabs reported p p true p true( ) ( )σ = × − ×2 2 , ,2 where σ2abs = absolute difference of the change in variance of the predicted value kreported = reported overdispersion parameter for the CPM kp,true = predicted true overdispersion parameter for CPM with external CMF Np = predicted average crash frequency, crashes/year Np,true = predicted true crash frequency, crashes/year 3. Compute the increased root mean square error using the following equation: [ ]σ = σ +, 2 2 0.5ee I abs where σe,I = increased root mean square error σ2abs = absolute difference of the change in variance of the predicted value e = error in predicted crash frequency 4. Compute the CV using the following equation: = σ , , CV NI e I p true where CVI = coefficient of variation for the increased root mean square error σe,I = increased root mean square error Np,true = predicted true crash frequency, crashes/year Step 8. Compute the Amount of Bias The percent bias (Bias) in the predicted crash frequency is computed using the following equation: = × − 100 , , Bias N N N p p true p true where Bias = percent bias in the reported value Np = predicted average crash frequency, crashes/year Np,true = predicted true crash frequency, crashes/year

24 Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results Outline of Solution Step 1. Assemble the Data Needed to Apply the Procedure The data needed to apply the procedure are listed in Table 5. The data needed to apply the SPF and the HSM Part D, CMF are also needed but not shown here. A review of agency files indicates that the major road AADT is 10,000 vehicles/day and the minor road AADT is 2,000 vehicles/day. The SPF was used with these AADT values to compute the predicted crash frequency NSPF of 4.97 crashes/year. The CMFs identified in Chapter 10 were individually considered. Their product is equal to 1.00. A review of the CPM development report prepared by the researchers that developed the CPM did not indicate whether the intersections used to estimate the CPM included flashing bea- cons. The NCHRP Project 17-78 research team obtained the database used by these researchers, and the locations of the intersections were identified. The team reviewed Google Earth’s his- torical aerial photos for each intersection and determined that a few of the intersections had a flashing beacon during the period for which data were collected to develop the CPM. A “1” was added to the database for each site with a beacon and zero was recorded for those sites without a beacon. The average of these values X–CPM was computed as 0.10, and the standard deviation σx,b was computed as 0.30. There is only one intersection of interest for this application. The safety effect of adding a flashing beacon at this intersection is being evaluated. This treatment is discrete so a “1” is used to indicate presence, which makes X– equal to 1.0. Because only one intersection is of interest and its treatment is specified with certainty, there is no uncertainty with X–, so σx,n is 0.0. Example Application—Scenario 1, Case B CMFs from HSM Part D, or from another source (e.g., FHWA CMF Clearinghouse), do not have a corresponding base condition in the SPF Question: An agency desires to conduct a safety evaluation of a four-leg, two-way stop control (TWSC) intersection on a rural two-lane, two-way road. The intersection is of interest because it has a relatively high crash frequency, which suggests it may have potential for safety improvement. The results of the evaluation will be used to determine the safety benefit of adding flashing beacons at the intersection. The practitioner has elected to use the CPM for TWSC intersections in Chapter 10 of the HSM. This CPM predicts the frequency of crashes of all types and severities. The CPM does not include a CMF for “add flashing beacon” at TWSC intersection. Hence, the CPM does not include a base condition that corresponds to this CMF. The agency has obtained a CMF from HSM Part D, Chapter 14, Provide Flashing Beacon at Stop-Controlled Intersection. The CMF for “all crash types and severities” is 0.95. The practitioner desires to assess the reliability of the predicted crash frequency given that the CMF from HSM Part D, does not have a corresponding base condition in the SPF.

Quantifying the Reliability of CPM Estimates for Mismatches Between Crash Modification Factors and SPF Base Conditions 25   Step 2. Compute Estimation Coefficient There are three sub-steps in this computation: 1. The value of the CMF of interest at the intersections used to estimate the CPM [CMFex (X – CPM)] is computed as a weighted average for the collective set of sites. It is known that 10% of the sites have a beacon. At each site, the CMF value is 0.95. The remaining sites have no beacon, so the CMF value is effectively 1.0. The weighted average CMF is computed as CMFex (0.10) is 0.995 [= 0.95 × 0.10 + 1.0 × (1 − 0.10)]. 2. The value of the CMF of interest for the intersection of interest [CMFex (X –)] is obtained from HSM Part D. The CMF for adding the beacon is 0.95 [i.e., CMFex (1) = 0.95]. 3. To compute the estimation coefficient b, the CMF values are used in the following equation: [ ] [ ]( ) ( )= − − = −0.0514b Ln CMF X Ln CMF X X X ex ex CPM CPM Step 3. Compute Bias Adjustment Factor Initially, equation ct = 1.120 if b > 0; otherwise, 0.880 is used to determine that a correction factor ct value of 0.880 is needed. Then, to compute the bias adjustment factor fB, the following equation is used: = + σ =1 0.5 1.002 ,2f b cB x n t Step 4. Compute the Predicted Crash Frequency for Site of Interest To compute the predicted average crash frequency Np, the SPF value NSPF of 4.97 crashes/year, the CMF product of 1.0, and the external CMF CMFex(X –) value of 0.95 are used in the following equation: ( )( )= × × × × × =. . . 4.72 crashes/year1N C N CMF CMF CMF Xp SPF n ex Input values needed for CPM AADTmajor = 10,000 vehicles/day AADTminor = 2,000 vehicles/day NSPF = 4.97 crashes/year Overdispersion parameter for CPM kreported kreported = 0.24 HSM Part C, CMF for geometric design element or traffic control feature i CMFi Product of CMF1 to CMFn = 1.00 External CMF for geometric design element or traffic control feature of interest CMFex HSM Part D, Chapter 14 CMFex, add beacon = 0.95 Average independent variable value at sites used to estimate the CPM 0.10 Standard deviation of the independent variable at sites used to estimate the CPM σx,b 0.30 Average independent variable value associated with external CMF at sites of interest 1.00 Standard deviation of the independent variable at sites of interest σx,n 0.00 Required Data and Relationships Potential Data Sources Table 5. Required data to apply the procedure for Scenario 1, Case B, example application.

26 Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results Step 5. Compute the Unbiased Predicted Crash Frequency for Site of Interest To compute the predicted true crash frequency Np,true, the predicted crash frequency Np, CMFex(X – CPM) CMFex(X –), and bias adjustment factor fB are used in the following equation: ( ) ( )= × × = 4.94 crashes/year,N N CMF X f CMF Xp true p ex CPM B ex Step 6. Compute the Unbiased Overdispersion Parameter for the CPM with the External CMF The CPM in HSM Chapter 10 for four-leg, TWSC intersections has four CMFs, so the variable p equals 5 (= n + 1). The adjustment factor Δ0 is computed using the following equation: [ ]( )∆ = − × − =1 0.10 2 5, 1 0.10min po To compute kp,true, the values of Δ0, kreported, b, and σx,b are used in the following equation: = − σ ∆ =1.13 0.240, 2 ,2k k bp true reported x b o Step 7. Compute the Increased Root Mean Square Error and CV There are four sub-steps in this computation: 1. To compute the error in the predicted crash frequency e, the predicted average crash fre- quency Np and the predicted true crash frequency Np,true are used in the following equation: = − = −e N Np p true 0.224 crashes/year, 2. To compute the absolute difference in the change in variance of the predicted value σ2abs, the reported overdispersion parameter for the CPM kreported, the predicted true overdispersion parameter for CPM with external CMF kp,true, the predicted average crash frequency, crashes/ year Np, and the predicted true crash frequency, crashes/year Np,true are used in the following equation: ( ) ( )σ = × − × =k N k Nabs reported p p true p true 0.518 crashes/year2 2 , ,2 3. To compute the increased root mean square error σe,I, the absolute difference in the change in variance of the predicted value σ2abs and the error in predicted crash frequency e are used in the following equation: [ ]σ = σ + = 0.753 crashes/year, 2 2 0.5ee I abs If the variance of the predicted value were computed, it should be increased by the square of σe,I. to include the error in the predicted value Np. 4. To compute the CV for the increased root mean square error CVI, the predicted value σe,I and the predicted true crash frequency Np,true are used in the following equation: = σ = 0.15, , CV NI e I p true

Quantifying the Reliability of CPM Estimates for Mismatches Between Crash Modification Factors and SPF Base Conditions 27   This CVI means that there is a relatively large amount of error-related variability in the predicted crash frequency. As noted, values over 0.20 are considered to be unreliable for most applications. Step 8. Compute the Amount of Bias To compute the percent bias in the predicted crash frequency (Bias), the predicted crash frequency Np and the predicted true crash frequency Np,true are used in the following equation: = × − = −100 4.5%., , Bias N N N p p true p true As noted, an absolute value of bias in excess of 10% is considered to be unreliable for most applications. The percent bias measure indicates the relative error in the prediction. Based on these results, the practitioner concludes that the use of the CMF for flashing beacon at stop-controlled intersections obtained from HSM Part D, does add bias to the predicted crash frequency, and it does lead to a relatively large amount of error-related variability in the pre- dicted crash frequency. Although neither the bias nor the increased error exceeds their respec- tive thresholds, which would indicate unreliable results, the practitioner decides to search for other sources of CMFs for flashing beacon at stop-controlled intersections that will yield more reliable predictions, and, if a more reliable CMF is not found, to use this CMF with due caution, knowing that there is uncertainty of the predicted crash frequency. Procedure: Scenario 1, Case C CMF (included in the CPM when developed) not used in CPM application but base condition accommodated in the SPF For this CPM application, the practitioner chooses not to use one or more of the CMFs that were included in the CPM when it was developed. All these CMFs (whether used or not used) have a corresponding base condition in the SPF. The unused CMFs are called herein “omitted CMFs.” An example of this application is when the practitioner does not have ready access to the data needed for a CMF and, as a result, chooses not to use the CMF when evaluating one or more sites. In Scenario 1, Case C, the likely source of the CPM is HSM Part C; however, this procedure is sufficiently general that it can be applied to CPMs from other sources (e.g., a CPM developed for a specific jurisdiction). There are eight steps in this procedure: Step 1. Assemble the data needed to apply the procedure. Step 2. Compute estimation coefficient. Step 3. Compute bias adjustment factor. Step 4. Compute the predicted crash frequency for site of interest. Step 5. Compute the unbiased predicted crash frequency for site of interest. Step 6. Compute the unbiased overdispersion parameter for the CPM with the omitted CMF. Step 7. Compute the increased root mean square error and CV. Step 8. Compute the amount of bias. The following steps describe data needs, equations, variable estimation, selected GOF mea- sures, and outcome related to the quantitative assessment of the degree of reliability.

28 Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results Step 1. Assemble the Data Needed to Apply the Procedure The data needed to apply the procedure are listed in Table 6. The data needed to apply the CPM and the HSM Part C, CMFs are not listed here. If there are several sites of interest, the CV (CVI) and bias percent (Bias) computed in Step 7 and Step 8, respectively, were derived to have the characteristic that they are insensitive to the value of the predicted crash frequency (NP). As a result, the predicted CV and bias percent reasonably describe the reliability of the predicted crash frequency for each site of interest, regardless of their AADT, segment length, and so forth. Therefore, for purposes of the reliability evaluation when there are several sites of interest, a typical site representing the collective set of sites can be used to compute the predicted crash frequency in Step 4. Typical values of AADT, segment length, and variables in the CMFs (other than the omitted CMF) can be used to apply the CPM in Step 4. The average of the independent variable at the sites used to estimate the CPM (X–CPM) can be obtained from one of several sources. It can be obtained from the CPM development report if that report provides summary statistics of the variable X in the data used to estimate the CPM. Alternatively, the average of X can be obtained through access to the original database used to estimate the CPM. Finally, if this database is not available or it does not contain the variable of interest X, then it may be possible to identify the regions from which the data were obtained to develop the CPM and obtain values of the variable X at a representative set of sites. These values would be obtained from agency files or field measurements. The value X–CPM would then be computed using this representative set of sites. If the treatment is discrete (e.g., “add beacon”), an indicator variable is used to indicate treatment presence (where 1 corresponds to treatment present, and zero corresponds to treatment not present). The average X–CPM is then computed using the 1 or zero values at the collective set of study sites. The standard deviation of the independent variable at the sites used to estimate the CPM (σx,b) is obtained using the same data source that was used to obtain X–CPM. The practitioner may be seeking the reliability evaluation of one site of interest or a group of sites of interest: • When One Site of Interest Is Being Evaluated. The average of the independent variable value associated with the omitted CMF (X–) can be obtained in several ways. One source of this information is agency files (e.g., design plans) for the site that is planned for treatment Required Data and Relationships Potential Data Sources Input values needed for CPM Agency files Overdispersion parameter for CPM kreported HSM Part C HSM Part C, CMF for geometric design element or traffic control feature i CMFi HSM Part C Omitted CMF for geometric design element or traffic control feature of interest CMFom HSM Part C Average independent variable value at sites used to estimate the CPM CPM development report, archived CPM database, agency files, or field measurements Standard deviation of the independent variable at sites used to estimate the CPM σx,b CPM development report, archived CPM database, agency files, or field measurements Average independent variable value associated with omitted CMF at sites of interest Site data from agency files, plans, or field measurements Standard deviation of the independent variable at sites of interest σx,n Site data from agency files, plans, or field measurements Table 6. Required data to apply the procedure for Scenario 1, Case C.

Quantifying the Reliability of CPM Estimates for Mismatches Between Crash Modification Factors and SPF Base Conditions 29   (or has been treated). Another source is field measurements at the site that is planned for treatment (or has been treated). Regardless of the source, X– equals the variable value at the site of interest. If the treatment is discrete (e.g., “add beacon”), an indicator variable is used to indicate treatment presence (where 1 corresponds to treatment present, and zero corresponds to treatment not present). The standard deviation of the independent variable at the sites of interest (σx,n) is equal to 0.0 if X is known with certainty. If X has been judged or measured, then the standard deviation should equal a small positive value reflecting the practitioner’s uncertainty in the judgment or measurement. • When Several Sites of Interest Are Being Evaluated. The average of the independent variable value associated with the omitted CMF (X–) can be obtained in several ways. One source of this information is agency files (e.g., design plans) for the sites that are planned for treatment (or have been treated). Another source is field measurements at the sites that are planned for treatment (or have been treated). Regardless of the source, if the treatment is applied uni- formly to the sites of interest as part of a planned change, then X has the same value at each site and X– equals X. Alternatively, if the treatment is not applied uniformly to all the sites of interest, then X– equals the value of X averaged for all sites. If the treatment is discrete (e.g., “add beacon”), an indicator variable is used to indicate treatment presence (where 1 corre- sponds to treatment present, and zero corresponds to treatment not present). The average X– is then computed using the 1 or zero values for the collective set of sites. The standard deviation of the independent variable at the sites of interest (σx,n) is obtained using the same data source that was used to obtain X–. Step 2. Compute Estimation Coefficient There are three sub-steps in this computation: 1. Compute the value of the omitted CMF for the sites used to estimate the CPM [CMFom (X – CPM)]. If the omitted CMF corresponds to a discrete treatment (e.g., “add beacon”), then the CMF value is computed using the corresponding site data, where each site with the treatment is assigned the CMF value and each site without the treatment is assigned a value of 1.0. For example, if the CMF for “add beacon” is 0.95 and 30% of the sites used to estimate the CPM have the beacon, then the value of CMFom (X – CPM) is 0.985 [= 0.95 × 0.30 + 1.0 × (1 − 0.30)]. If the omitted CMF corresponds to a continuous treatment (e.g., lane width), then the CMF value used is computed using the CMF function with the average value of X (i.e., X–CPM). 2. Compute the value of the omitted CMF for the sites of interest [CMFom (X –)]. If the omitted CMF corresponds to a discrete treatment (e.g., “add beacon”), then the CMF value is computed using the corresponding site data, where each site with the treatment is assigned the CMF value and each site without the treatment is assigned a value of 1.0. If all the sites of interest plan to have the beacon added, then the value of CMFom (X –) is 0.95 [= 0.95 × 1.0 + 1.0 × (1 − 1.0)]. If the omitted CMF corresponds to a continuous treatment (e.g., lane width), then the CMF value used is computed using the CMF function with the average value of X (i.e., X–). If X– is equal to X–CPM, then multiply X – by 1.01 and use the resulting value of X– to compute CMFom (X –). Use this computed CMF value and the resulting value of X– in the equation provided in sub- step 3. Multiplication by the constant 1.01 is used to avoid division by zero in this equation. 3. Use the following equation to compute the estimation coefficient b: [ ] [ ]( ) ( )= − − b Ln CMF X Ln CMF X X X om om CPM CPM where b = estimation coefficient CMFom = omitted CMF (i.e., associated with the SPF’s base conditions but excluded from CPM)

30 Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results X– = average independent variable associated with CMF of interest at sites of interest X– CPM = average independent variable value at sites used to estimate the CPM CMFom(X –) = omitted CMF value associated with X– CMFom(X – CPM) = omitted CMF value associated with X – CPM Step 3. Compute Bias Adjustment Factor Determine the correction term ct based on the sign of the estimation coefficient b. If b is greater than zero, then ct equals 1.120; otherwise, ct equals 0.880. Then, use this term in the fol- lowing equation to compute the bias adjustment factor fC: = + σ1 0.5 2 ,2f b cC x n t with = >1.120 if 0; 0.880 otherwisec bt where fC = bias adjustment factor for Scenario 1, Case C σ2x,n = variance of the independent variable at sites of interest b = estimation coefficient ct = correction term Step 4. Compute the Predicted Crash Frequency for Site of Interest The following equation is used to compute the predicted crash frequency for the site of interest: ( )= × × × × −. . .1 1N C N CMF CMFp SPF n =CMF CMFn om where Np = predicted average crash frequency, crashes/year C = local calibration factor NSPF = predicted crash frequency for site with base conditions, crashes/year CMFi = HSM Part C, CMF for geometric design element, or traffic control feature i (i = 1 to n) n = total number of HSM Part C, CMFs CMFom = omitted CMF (i.e., associated with the SPF’s base conditions but excluded from CPM) Step 5. Compute the Unbiased Predicted Crash Frequency for Site of Interest The following equation is used to compute the unbiased predicted crash frequency for the site of interest: ( ) ( )= × × ,N N f CMF X CMF Xp true p C om om CPM

Quantifying the Reliability of CPM Estimates for Mismatches Between Crash Modification Factors and SPF Base Conditions 31   where Np,true = predicted true crash frequency, crashes/year Np = predicted average crash frequency, crashes/year fC = bias adjustment factor for Scenario 1, Case C CMFom(X –) = omitted CMF value associated with X– CMFom(X – CPM) = omitted CMF value associated with X – CPM Step 6. Compute the Unbiased Overdispersion Parameter for the CPM with the Omitted CMF Compute the adjustment factor Δ0 using the relevant equation shown below, then, compute the overdispersion parameter for the CPM with the omitted CMF using the following equation: = + σ ∆1.16, 2 ,2k k bp true reported x b o with [ ]( )∆ = − × −1 0.10 2 5, 1min po where kp,true = predicted true overdispersion parameter for CPM with omitted CMF kreported = reported overdispersion parameter for the CPM b = estimation coefficient σ2x,b = variance of the independent variable at sites used to estimate the CPM Δ0 = adjustment factor for the incremental effect of additional empirical coefficients p = number of empirically derived constants in the CMFs associated with the CPM which would include those in the omitted CMF (exclude those in the SPF for inter- cept, AADT, and segment length) The number of empirically derived constants p is determined by inspecting the CMFs in the CPM. Regression constants in the SPF (e.g., intercept, AADT coefficient, segment length coeffi- cient) are not considered when determining the value of p. If the CMF is associated with a discrete treatment (e.g., add beacon), then the CMF is often represented by a single empirical constant. If the CMF is associated with a continuous variable (e.g., lane width), then it is often a function that includes one or more empirical constants. In general, there is at least one empirically derived constant for each CMF used to develop the CPM. For practical applications of this procedure, the variable p can be estimated as equal to the number of CMFs used in Step 4 (i.e., p = n). Step 7. Compute the Increased Root Mean Square Error and CV There are four sub-steps in this computation: 1. Compute the error in the predicted crash frequency using the following equation: = − ,e N Np p true where e = error in predicted crash frequency Np = predicted average crash frequency, crashes/year Np,true = predicted true crash frequency, crashes/year

32 Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results 2. Compute the absolute difference in the change in variance of the predicted value using the following equation: 2 2 , , 2( ) ( )σ = × − ×k N k Nabs reported p p true p true where σ2abs = absolute difference of the change in variance of the predicted value kreported = reported overdispersion parameter for the CPM Np = predicted average crash frequency, crashes/year kp,true = predicted true overdispersion parameter for CPM with omitted CMF Np,true = predicted true crash frequency, crashes/year 3. Compute the increased root mean square error using the following equation: [ ]σ = σ +, 2 2 0.5ee I abs where σe,I = increased root mean square error σ2abs = absolute difference of the change in variance of the predicted value e = error in predicted crash frequency 4. Compute the CV using the following equation: = σ , , CV NI e I p true where CVI = coefficient of variation for the increased root mean square error σe,I = increased root mean square error Np,true = predicted true crash frequency, crashes/year Step 8. Compute the Amount of Bias The percent bias in the predicted crash frequency is computed using the following equation: = × − 100 , , Bias N N N p p true p true where Bias = percent bias in the reported value Np = predicted average crash frequency, crashes/year Np,true = predicted true crash frequency, crashes/year Example Application—Scenario 1, Case C CMF (included in the CPM when developed) not used in CPM application but base condition accommodated in the SPF For this CPM application, the practitioner chooses not to use one or more of the CMFs that were included in the CPM when it was developed. All these CMFs

Quantifying the Reliability of CPM Estimates for Mismatches Between Crash Modification Factors and SPF Base Conditions 33   Outline of Solution Step 1. Assemble the Data Needed to Apply the Procedure The data needed to apply the procedure are listed in Table 7. The data needed to apply the SPF and the HSM Part C, CMFs are also needed but not shown here. A review of agency files indicates that the AADT varies among the intersections of interest. However, the CV and bias percent computed in Step 7 and Step 8, respectively, are derived to have the characteristic that they are insensitive to the predicted crash frequency (NP). As a result, the predicted CV and bias percent reasonably describe the reliability of the predicted crash frequency for each of the intersections of interest, regardless of the intersection AADT. For (whether used or not used) have a corresponding base condition in the SPF. The unused CMFs are called herein “omitted CMFs.” An example of this application is when the practitioner does not have ready access to the data needed for a CMF, and, as a result, chooses not to use the CMF when evaluating one or more sites. Question: An agency desires to conduct a safety evaluation of several four-leg, TWSC intersections on rural two-lane, two-way roads in one region of its jurisdiction. The intersections of interest have a relatively high crash frequency, which suggests they may have the potential for safety improvement. The practitioner has elected to use the CPM for TWSC intersections in Chapter 10 of the HSM. This CPM predicts the frequency of crashes of all types and severities. The CPM includes a CMF for skew angle at four-leg, TWSC intersections. However, data describing the skew angle at the intersections of interest is not readily available, so the practitioner is inclined to assume that the skew angle is 0.0 degrees, such that the skew angle CMF value is 1.0. Before accepting this inclination (which is equivalent to omitting the skew angle CMF from the analysis), the practitioner assesses its impact on the reliability of the predicted crash frequency. Required Data and Relationships Potential Data Sources Input values needed for CPM AADTmajor = 10,000 vehicles/day AADTminor = 2,000 vehicles/day NSPF = 4.97 crashes/year Overdispersion parameter for CPM kreported kreported = 0.24 HSM Part C, CMF for geometric design element or traffic control feature i CMFi Product of CMF1 to CMFn = 1.00 Omitted CMF for geometric design element or traffic control feature of interest CMFex HSM Part C Average independent variable value at sites used to estimate the CPM 10 degrees Standard deviation of the independent variable at sites used to estimate the CPM σx,b 15 degrees Average independent variable value associated with omitted CMF at sites of interest 3.0 degrees Standard deviation of the independent variable at sites of interest σx,n 10 degrees Table 7. Required data to apply the procedure for Scenario 1, Case C, example application.

34 Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results purposes of the reliability evaluation, typical AADT values can be used for the calculations. The typical intersection major road AADT is 10,000 vehicles/day and the typical minor road AADT is 2,000 vehicles/day. The SPF was used with these AADT values to compute the predicted crash frequency NSPF of 4.97 crashes/year. The CMFs identified in HSM Chapter 10 were individually considered. The skew angle CMF was not included in this consideration. The key reliability measures are insensitive to the pre- dicted crash frequency. Thus, as a simplifying assumption, the product of the considered CMFs was set to equal to 1.00 for the intersections of interest. A review of the CPM development report prepared by the researchers that developed the CPM did not include the skew angle at the intersections used to estimate the CPM. The database used by these researchers was obtained and the skew angle for a sample of the intersections in the database was measured using Google Earth aerial imagery. The average of the measured skew angle values X–CPM was computed as 10 degrees, and the standard deviation σx,b was computed as 15 degrees. A similar sampling process was used to measure the skew angle at some of the intersections of interest. The average of the measured skew angle values X– was computed as 3 degrees, and the standard deviation σx,n was computed as 10 degrees. Step 2. Compute Estimation Coefficient There are three sub-steps in this computation: 1. The value of the CMF of interest at the intersections used to estimate the CPM [CMFom (X – CPM)] is computed using the skew angle CMF function provided in HSM Chapter 10. For an average angle of 10 degrees, the CMF is computed as 1.055 [i.e., CMFom (10) = 1.055]. 2. The value of the CMF of interest for the intersection of interest [CMFom (X –)] is computed using the same CMF function. For an average angle of 3 degrees, the CMF is computed as 1.016 [i.e., CMFom (3) = 1.016]. 3. To compute the estimation coefficient b, the two CMF values are used in the following equation: [ ] [ ]( ) ( )= − − = 0.0054b Ln CMF X Ln CMF X X X om om CPM CPM Step 3. Compute Bias Adjustment Factor Initially, equation ct = 1.120 if b > 0; otherwise, 0.880 is used to determine that a correction factor ct value of 1.12 is needed. Then, to compute the bias adjustment factor fC the following equation is used: = + σ =1 0.5 1.0022 ,2f b cC x n t Step 4. Compute the Predicted Crash Frequency for Site of Interest To compute the predicted average crash frequency Np, the SPF value NSPF of 4.97 and the CMF product of 1.0 are used in the following equation: ( )= × × × × =−. . . 4.97 crashes/year1 1N C N CMF CMFp SPF n

Quantifying the Reliability of CPM Estimates for Mismatches Between Crash Modification Factors and SPF Base Conditions 35   Step 5. Compute the Unbiased Predicted Crash Frequency for Site of Interest To compute the predicted true crash frequency Np,true, the predicted crash frequency Np, CMFom(X – CPM) CMFom(X –), and bias adjustment factor fC are used in the following equation: ( ) ( )= × × =N N f CMF X CMF Xp true p C om om CPM 4.79 crashes/year, Step 6. Compute the Unbiased Overdispersion Parameter for the CPM with the Omitted CMF The CPM in HSM Chapter 10 for four-leg, TWSC intersections has four CMFs, so the variable p equals 4 (= n). The adjustment factor Δ0 is computed using the following equation: [ ]( )∆ = − × − =1 0.10 2 5, 1 0.30min po To compute kp,true, the values of Δ0, kreported, b, and σx,b are used in the following equation: = + σ ∆ =1.16 0.242, 2 ,2k k bp true reported x b o Step 7. Compute the Increased Root Mean Square Error and CV There are four sub-steps in this computation: 1. To compute the error in the predicted crash frequency e, the predicted crash frequency Np and the predicted true crash frequency Np,true are used in the following equation: = − = 0.176 crashes/year,e N Np p true 2. To compute the absolute difference in the change in variance of the predicted value σ2abs, the reported overdispersion parameter kreported, the predicted true overdispersion parameter kp,true, the predicted crash frequency Np, and the predicted true crash frequency Np,true are used in the following equation: k N k Nabs reported p p true p true( ) ( )σ = × − × = 0.3612 2 , ,2 3. To estimate the increased root mean square error σe,I., the predicted value σ2abs and the pre- dicted error e are used in the following equation: [ ]σ = σ − =ee I abs 0.626 crashes/year, 2 2 0.5 If the variance of the predicted value were computed, it should be increased by the square of σe,I. to include the error in the predicted value Np. 4. To compute the CV for the increased root mean square error CVI., the predicted value σe,I and the predicted true crash frequency Np,true are used in the following equation: = σ = 0.13, , CV NI e I p true

36 Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results This CVI means that there is a relatively large amount of error-related variability in the pre- dicted crash frequency. Values over 0.20 are considered to be unreliable for most applications. Step 8. Compute the Amount of Bias To compute the percent bias in the predicted crash frequency (Bias), the predicted crash frequency Np and the predicted true crash frequency Np,true are used in the following equation: = × − =100 3.7%, , Bias N N N p p true p true As noted, an absolute value of bias in excess of 10% is considered to be unreliable for most applications. The percent bias measure indicates the relative error in the prediction. Based on these results, the practitioner concludes that assuming that the skew angles of those intersections are homogeneously 0.0 degrees (i.e., that the skew angle CMF value is 1.0) does add bias to the predicted crash frequency, and it does lead to a modest amount of error-related variability in the predicted crash frequency. Although neither the bias nor the increased error exceeds their respective thresholds, which would indicate unreliable results, the practitioner believes they are sufficiently large to justify the need for the collection of data on intersection skew angles to enable the agency to make a more reliable and defensible selection of inter- sections with potential for safety improvement.