Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results (2021)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

57   Introduction In HSM Part C, a simple two-stage approach is used when estimating average predicted crash frequencies by type and severity using an SPF: 1. The total crash frequency is estimated for a specific facility type. 2. A fixed proportion for each type or severity is applied to the estimate for Total crashes to obtain the estimates of crashes by type or severity for the specific facility type. However, using a pre-specified fixed proportion is questionable because crash type or severity distribution may differ by road segment, intersection characteristics, or other factors, and con- sequently, the proportion for each type or severity level may differ as well. To address the limitations of the simple two-stage approach suggested in Part C of the HSM, CPMs for various crash types and severities were developed under NCHRP Project 17-62. These CPMs are available in NCHRP Web-Only Document 295: Improved Prediction Models for Crash Types and Crash Severities (Ivan et al. 2021) and are being considered for inclusion in the second edition of the HSM. However, there are two types of cases where SPFs could not be reliably estimated, and a third type pertaining to crash types and severities for which estimation of SPFs was not considered because of the small sample sizes or odd estimation results. For example, reliable SPFs could not be developed for KA (fatal and incapacitating injury) and KAB (fatal, incapacitating injury, non-incapacitating injury) crashes at four-leg stop-controlled (4ST) intersections or KA single-vehicle (SV) crashes at four-leg signalized intersections. These are some of the rare crash types and severities for which crash frequency estimates still need to use the simple two-stage approach. Scenario 4 This chapter addresses the reliability of predictions obtained from the simple two-stage approach as it applies to rare crash types and severities for the following cases. Scenario 4, Case A There are no recommended SPFs for specific/given rare crash types and severities because Ivan et al. (under NCHRP Project 17-62) concluded that the potential models did not con- verge or were illogical (e.g., AADT exponents were negative or statistically insignificant at the 10% level). C H A P T E R 5 Reliability Associated with Using a Crash Prediction Model to Estimate Frequency of Rare Crash Types and Severities

58 Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results Scenario 4, Case B There are a few SPFs for specific/given rare crash types and severities with Low confidence because Ivan et al. (under NCHRP Project 17-62) concluded that these SPFs did not validate well or had poor GOF statistics. Scenario 4, Case C There are no SPFs for specific/given rare crash types and severities because Ivan et al. (under NCHRP Project 17-62) concluded that the estimation of SPFs was not contemplated either because they were not of primary interest generally (e.g., nighttime crashes) or because there are typically too few crashes to attempt SPF development (e.g., bicycle, pedestrian, and fatal crashes). In all three cases of Scenario 4, a reliable parent SPF is available in the HSM or another source, such as NCHRP Web-Only Document 295, to which a crash type and severity pro- portion using the applicable jurisdiction’s data can be applied. A parent SPF would be the one with the lowest crash frequency that includes the crash type and severity of interest. For example, a KAB parent SPF, if reliable, and presented as such in the HSM, would be consid- ered for KA crashes. Otherwise, a KABC parent SPF, if reliable, would be considered for both KA and KAB crashes, and so on. Because the impact on reliability can be so variable depending on the relative frequencies of the rare crash and parent crash types, the objective here is to demonstrate, using actual data and SPFs estimated in the NCHRP Project 17-62, a heuristic procedure that a practitioner can use to assess how reliability may be impacted for a specific application. Procedure Steps and Example Application This section describes the procedures to assess the reliability of the results when using a calibrated parent SPF from another jurisdiction (or the HSM Part C) to estimate the frequen- cies of rare crash types and severities (Scenario 4, Case A, or Scenario 4, Case C) or when using an uncalibrated parent SPF or an uncalibrated SPF with Low confidence (i.e., did not validate well or has poor GOF statistics) to estimate the frequencies of rare crash types and severities (Scenario 4, Case B). Procedure: Scenario 4, Case A, or Scenario 4, Case C A calibrated parent SPF from another jurisdiction (or the HSM Part C) used to estimate the frequencies of rare crash types and severities For Scenario 4, Case A, or Scenario 4, Case C, a crash type and severity proportion developed from the jurisdiction’s data are applied to a prediction from the recommended and calibrated parent SPF. There are six steps in this procedure: Step 1. Select the parent SPF(s). Step 2. Assemble data needed to calibrate the parent SPF(s) and to develop the crash type and severity proportions. Step 3. Collect the observed rare crash records for a given facility type, and compute crash type and severity proportions. Step 4. Apply the crash type and severity proportions to the respective parent SPF(s). Step 5. Calibrate the parent SPF(s) using local data.

Reliability Associated with Using a Crash Prediction Model to Estimate Frequency of Rare Crash Types and Severities 59   Step 6. Compute the CV of the estimated calibration factor for each parent SPF and the CURE plots. The following steps describe the data needs, equations, variable estimation, selected GOF measures, and outcome related to the quantitative assessment of the degree of reliability. Step 1. Select the Parent SPFs Define the rare crash type and severity of interest. One or more reliable parent SPF(s) are sourced from the HSM Part C or another source, such as NCHRP Web-Only Document 295 or another jurisdiction, to which a crash type and severity proportion using the application juris- diction’s data can be applied. A parent SPF would be the one with the lowest crash frequency that includes the crash type and severity of interest. For example, a KAB parent SPF, if reliable, would be considered for KA crashes. Otherwise, a KABC (all injury severity combined) parent SPF, if reliable, would be considered for both KA and KAB crashes, and so on. Step 2. Assemble Data Needed to Calibrate the Parent SPF(s) and to Develop the Crash Type and Severity Proportions Assemble the data required for calibrating the SPF(s). HSM guidance (Part C, Appendix A2) can be used to determine the number of locations required (i.e., minimum sample sizes). Fur- ther, follow the HSM guidance (Part C, Appendix A) for calibrating the CPM if it was devel- oped in another jurisdiction or a different time period than the data to which it will be applied. Further, in The Calibrator: An SPF Calibration and Assessment Tool: User Guide, Report No. FHWA-SA-17-016, Lyon et al. (2016) provide comprehensive guidance about the data needs as well as how to use The Calibrator. Assemble the data required to develop the crash type and severity proportions for the selected parent SPF(s). HSM Part C guidance can be used to determine the process and sample sizes to estimate the proportions for the crash type and severity combinations. Step 3. Collect the Observed Rare Crash Records for a Given Facility Type, and Compute Crash Type and Severity Proportions Using the jurisdiction’s database, extract all observed crash records for the rare crash type of interest for all severities for the specific facility type for a given time period (e.g., 3 years). Compute the following: • Total number of rare observed crashes for the specific facility type • Proportion estimates of crashes by type and severity Step 4. Apply the Crash Type and Severity Proportions to the Respective Parent SPF(s) Apply the crash type and severity proportions calculated in Step 3 to the respective parent SPF(s). Step 5. Calibrate the Parent SPF(s) Using Local Data The parent SPF(s) are calibrated using local data. The Calibrator is a useful tool; it is a spreadsheet-based tool developed to assess compatibility and applicability of SPFs and CMFs for application in a different time or place. The tool calculates and provides a single calibration

60 Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results factor for each SPF or SPF and CMF combination. The Calibrator can be downloaded from the link: (https://safety.fhwa.dot.gov/rsdp/toolbox-content.aspx?toolid=150). Step 6. Compute the CV of the Estimated Calibration Factor for Each Parent SPF and the CURE Plots Compute the GOF measures to assist in the assessment of how the predictions are per- forming over the range of the independent variables. The two GOF measures selected for this assessment are the CV(C) and the percent of cumulative residuals exceeding the confidence limits (outside the 95% confidence limits): Coefficient of Variation of the Calibration Factor The CV(C) is the standard deviation of the calibration factor divided by the estimate of the calibration factor, as shown in the following equation. ( ) ( )=CV C V C C where CV(C) = coefficient of variation of the calibration factor C = estimate of the calibration factor V(C) = variance of the calibration factor, can be calculated as follows:  ∑ ∑( ) ( ) ( )= − ∗ 2 2V C y k y y i ii ıi where yi = observed crash counts yı = uncalibrated predicted values from the SPF k = overdispersion parameter (recalibrated) It is suggested that a CV > 0.15 means an unsuccessful calibration (Lyon et al. 2016). CURE Plots The graph of the cumulative residuals (observed minus predicted crashes) is plotted against a variable of interest sorted in ascending order (e.g., major road AADT). CURE plots provide a visual representation of GOF over the range of a given variable and help to estimate the percent exceeding the confidence limits [outside 95% CI (%)]. The cumulative residuals outside the 95% confidence limits indicate a poor fit over that range in the variable of interest. Example Application—Scenario 4, Case A, or Scenario 4, Case C A calibrated parent SPF from another jurisdiction (or the HSM Part C) used to estimate the frequencies of rare crash types and severities

Reliability Associated with Using a Crash Prediction Model to Estimate Frequency of Rare Crash Types and Severities 61   Outline of Solution Step 1. Select the Parent SPF(s) Two parent SPFs that were estimated using the California database could be considered: 1. SPF for all severities combined SD crashes (SD-KABCO) 2. SPF for KA crashes for all crash types combined (ALL-KA) Step 2. Assemble Data Needed to Calibrate the Parent SPF(s) and to Develop the Crash Type and Severity Proportions The Illinois validation dataset is used in this example for estimating the crash type/severity proportion as well as for assessing the resulting SPF when the proportion is applied to the parent SPF from California estimation data. Step 3. Collect the Observed Rare Crash Records for a Given Facility Type, and Compute Crash Type and Severity Proportions The eight crashes in the Illinois dataset constituted 17.02% of all SD crashes and 38.01% of all KA crashes in the Illinois validation dataset. Step 4. Apply the Crash Type and Severity Proportions to the Respective Parent SPF(s) Applying these proportions to the respective parent SPFs estimated for NCHRP Project 17-62 for California, the following base-condition SPFs are considered for estimating SD-KA crashes in Illinois: 1. SPF for all severities combined SD crashes (SD-KABCO) ( )( )= × − × ×1: 0.1702 exp 14.701 1.479Option Crashes year segment length AADT 2. SPF for KA crashes for all crash types combined (ALL-KA) ( )( )= × − × ×2 : 0.3801 exp 7.690 0.508Option Crashes year segment length AADT Question: The agency needs to estimate the predicted crash frequency for same direction (SD), fatal and incapacitating injury (KA) crashes on 4-lane divided (4D) road segments. The agency reviewed the NCHRP Project 17-62 Final Report (published as NCHRP Web-Only Document 295) (Ivan et al. 2021) and concluded that such a base model for these crashes could not be estimated in this project because there were none in the California database used in the project. However, the database for another jurisdiction (Illinois) that was used for model validation in that project contained eight such crashes. Thus, the question is: What SPFs can be used for estimating SD-KA crashes for base conditions in Illinois?

62 Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results Step 5. Calibrate the Parent SPF(s) Using Local Data Using The Calibrator, these SPFs are calibrated to Illinois base-condition validation data being used for the example. The data used for this calibration should be the largest set that is feasibly available. Step 6. Compute the CV of the Estimated Calibration Factor for Each Parent SPF and the CURE Plots If either provides reliable predictions, the assessment of which option is best is based on the two selected GOF measures. These GOF measures were estimated by The Calibrator. The CV(C) is shown in Table 20. The CURE plots based on the SPF predicted values (shown on the x-axis) for the two options are shown in Figure 1 and Figure 2. Table 20 notes the CV(C) is 0.36 for the Option 1 and Option 2 SPFs. It is concluded that the calibrations of both SPFs were unsuccessful. Lyon et al. recommended an upper threshold of 0.15 for the CV(C). Nevertheless, the practitioner may still decide to adopt one of the calibrated parent SPFs and use it with due caution. The Calibrator estimated the percent of the cumulative residuals outside the 95% confidence limits (exceeding the 2σ limits) as depicted in the CURE plots. The Figure 1 Option 1 SPF for all severities combined SD crashes (SD-KABCO) has 23% of CURE data points exceed ing the 2σ limits, and the Figure 2 Option 2 SPF for KA crashes for all crash types combined (ALL-KA) has 41% of CURE data points exceeding the 2σ limits. Cumulative residuals SPF Option V(C) CV(C) SD-KA Option 1 8 12.86 0.62 0.05 0.36 SD-KA Option 2 8 4.67 1.71 0.37 0.36 Calibration Factor Total Predicted Crashes Total Observed Crashes Table 20. GOF measures for the two SPF options for estimation of SD-KA for Illinois 4D. Figure 1. The Calibrator CURE plot of residuals based on modified NCHRP Project 17-62 California estimated base- condition SPF (Option 1) predictions (x-axis) for Illinois.

Reliability Associated with Using a Crash Prediction Model to Estimate Frequency of Rare Crash Types and Severities 63   outside the 95% confidence limits indicate a poor fit over that range in the variable of interest. If these residuals are frequently outside the confidence limits, possible bias in the SPF is indi- cated. In this case, the CURE plots suggest that Option 1 would be somewhat superior to Option 2 in that the cumulative residuals oscillate closer to the x-axis and stay largely within the 2σ limits. Based on the result that the CV(C)s for both options were very high, indicating unsuccessful calibrations, the practitioner decides that none of the SPFs should be used in Illinois—even though Option 1’s cumulative residuals oscillate closer to the x-axis and close to 80% are within the 2σ limits. The practitioner decides to search for and assess other SPFs, and, if this is unsuccessful, the practitioner may use the Option 1 SPF in the short term with due caution. Figure 2. The Calibrator CURE plot of residuals based on modified NCHRP Project 17-62 California estimated base- condition SPF (Option 2) predictions (x-axis) for Illinois. Procedure: Scenario 4, Case B An uncalibrated parent SPF or an uncalibrated SPF with Low confidence used to estimate the frequencies of rare crash types and severities For Scenario 4, Case B, the practitioner has two potential approaches to select the SPF that produces the most reliable crash predictions: • Approach 1: A Case B uncalibrated SPF that did not validate well or has poor GOF statistics. Such an SPF may not be presented in the HSM but may be retrieved from another source, such as NCHRP Web-Only Document 295. • Approach 2: A modified SPF in which a crash type and severity proportion developed from the jurisdiction’s data are applied to a prediction from the HSM recommended and uncali- brated parent SPF. There are seven steps in this procedure: Step 1. Select the uncalibrated and unreliable SPF. Step 2. Select the parent SPF(s).

64 Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results Step 3. Assemble data needed to develop the crash type and severity proportions. Step 4. Collect the observed rare crash records for the given facility type, and compute crash type and severity proportions. Step 5. Apply the crash type and severity proportions to the respective parent SPF(s). Step 6. Calibrate the parent SPF(s) and the unreliable SPF using local data. Step 7. Compute the CV of the estimated calibration factor for each parent SPF and the CURE plots. The following steps describe the data needs, equations, variable estimation, selected GOF measures, and outcome related to the quantitative assessment of the degree of reliability. Step 1. Select the Uncalibrated and Unreliable SPF Define the rare crash type and severity of interest for the given facility type. NCHRP Web- Only Document 295 is one source of several base-condition SPFs that did not validate well or had poor GOF statistics. Select the appropriate SPF for the rare crash type and severity for the given facility type. Step 2. Select the Parent SPF(s) One or more reliable parent SPF(s) are sourced from the HSM Part C or another source, such as NCHRP Web-Only Document 295 or another jurisdiction, to which a crash type and severity proportion using the application jurisdiction’s data can be applied. A parent SPF would be the one with the lowest crash frequency that includes the crash type and severity of interest. For example, a KAB parent SPF, if reliable, would be considered for KA crashes. Otherwise, a KABC parent SPF, if reliable, would be considered for both KA and KAB crashes, and so on. Step 3. Assemble Data Needed to Develop the Crash Type and Severity Proportions Assemble the data required to develop the crash type and severity proportions for the selected parent SPF(s). HSM Part C guidance can be used to determine the process and sample sizes to estimate the proportions for the crash type and severity combinations. Step 4. Collect the Observed Rare Crash Records for the Given Facility Type, and Compute Crash Type and Severity Proportions Using the jurisdiction’s database, extract all observed crash records for the rare crash type of interest for all severities for the specific facility type for a given time period (e.g., 3 years). Compute the following: • Total number of rare observed crashes for the specific facility type • Proportion estimates of crashes by type and severity Step 5. Apply the Crash Type and Severity Proportions to the Respective Parent SPF(s) Apply the crash type and severity proportions calculated in Step 3 to the respective parent SPF(s).

Reliability Associated with Using a Crash Prediction Model to Estimate Frequency of Rare Crash Types and Severities 65   Step 6. Calibrate the Parent SPF(s) and the Unreliable SPF Using Local Data Assemble the data required for calibrating the SPF(s). HSM guidance (Part C, Appendix A2) can be used to determine the number of locations required (i.e., minimum sample sizes). Follow the HSM guidance (Part C, Appendix A) for calibrating the CPM that was developed in another jurisdiction or a different time period than the data to which it will be applied. Lyon et al. provide comprehensive guidance about the data needs as well as how to use The Calibrator. Step 7. Compute the CV of the Estimated Calibration Factor for Each Parent SPF and the CURE Plots Compute the GOF measures to assist in the assessment of how the predictions are per- forming over the range of the independent variables. The two GOF measures selected for this assessment are the CV(C) and the percent of cumulative residuals exceeding the confidence limits (outside the 95% confidence limits): Coefficient of Variation of the Calibration Factor The CV(C) is the standard deviation of the calibration factor divided by the estimate of the calibration factor, as shown in the following equation. ( ) ( )=CV C V C C where CV(C) = coefficient of variation of the calibration factor C = estimate of the calibration factor V(C) = variance of the calibration factor, can be calculated as follows:  ∑ ∑( ) ( ) ( )= − ∗ 2 2V C y k y y i ii ıi where yi = observed crash counts yı = uncalibrated predicted values from the SPF k = overdispersion parameter (recalibrated) It is suggested that a CV > 0.15 means an unsuccessful calibration (Lyon et al. 2016). CURE Plots The graph of the cumulative residuals (observed minus predicted crashes) is plotted against a variable of interest sorted in ascending order (e.g., major road AADT). CURE plots provide a visual representation of GOF over the range of a given variable and help estimate the percent exceeding the confidence limits [outside 95% CI (%)]. The cumulative residuals outside the 95% confidence limits indicate a poor fit over that range in the variable of interest.

66 Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results Outline of Solution Step 1. Select the Uncalibrated and Unreliable SPF Several base-condition SPFs developed under NCHRP Project 17-62 did not validate well or had poor GOF statistics. There is such an SPF for SD-KAB crashes at 4ST intersections on multi- lane roads. The base-condition SPF for this crash type and severity is as follows: ( )( )= − ×exp 9.502 0.641Crashes year Total Entering AADT The SPF was estimated based on only 12 crashes at 139 sites in Minnesota, and the large standard error for the AADT exponent (0.558) indicated not only that it is highly insignificant statistically, but that the SPF is likely to be highly unreliable. Step 2. Select the Parent SPF(s) Under NCHRP Project 17-62, two recommended and potential parent SPFs from Minnesota data for this facility type were developed. One SPF is SD-KABCO and the other SPF is KAB-ALL. These two SPF and could be considered and compared for the modified SPF predictions. Step 3. Assemble Data Needed to Develop the Crash Type and Severity Proportions Assemble the data required to develop the crash type and severity proportions for the selected parent SPF(s). NCHRP Web-Only Document 295 has validation data available for Ohio that would be suitable to assess the validity of applying the SPFs to another jurisdiction. Step 4. Collect the Observed Rare Crash Records for the Given Facility Type, and Compute Crash Type and Severity Proportions The Ohio dataset contained 12 SD-KAB crashes at 83 sites. The 12 SD-KAB crashes consti- tuted 17.39% of all KAB crashes and 28.57% of all SD crashes in the Ohio validation dataset. Example Application—Scenario 4, Case B An uncalibrated parent SPF or an uncalibrated SPF with Low confidence used to estimate the frequencies of rare crash types and severities Question: The agency needs to estimate the predicted crash frequency for SD-KAB crashes at 4ST intersections on multilane roads. An SPF is available from NCHRP Web-Only Document 295, but it is thought to be unreliable because it did not validate well and has poor GOF statistics. The question for the practitioner in this jurisdiction is whether it is better to use that unreliable SPF or to apply a modified SPF in which a crash type and severity proportion developed from the jurisdiction’s data are applied to a prediction from recommended and uncalibrated parent SPF(s).

Reliability Associated with Using a Crash Prediction Model to Estimate Frequency of Rare Crash Types and Severities 67   Step 5. Apply the Crash Type and Severity Proportions to the Respective Parent SPF(s) The crash type and severity proportion developed from the subject jurisdiction’s data (Ohio) is applied to modify predictions from the recommended and calibrated parent SPF from another jurisdiction (Minnesota). Applying these proportions to the respective parent SPFs from NCHRP Web-Only Docu- ment 295, the following modified base-condition SPFs are considered for estimating SD-KAB crashes in Ohio: ( ) ( )( )= × − × ×1: 0.1739 exp 8.843 0.441 0.509Option Crashes year Major AADT Minor AADT ( ) ( )( )= × − × ×2 : 0.2857 exp 14.343 1.158 0.345Option Crashes year Major AADT Minor AADT Step 6. Calibrate the Parent SPF(s) and the Unreliable SPF Using Local Data Using The Calibrator, these two SPFs (Step 5), along with the unreliable SPF estimated from the Minnesota base-condition data (Step 1) are calibrated to Ohio base-condition validation data from NCHRP Web-Only Document 295. The data used for this should be the largest set that is feasibly available. Step 7. Compute the CV of the Estimated Calibration Factor for Each Parent SPF and the CURE Plots If any of them provides reliable predictions, the assessment of which of the three SPFs is best is based on several GOF measures estimated by The Calibrator. The two GOF measures selected for this example application are the CV of the estimated calibration factor, and measures based on CURE plots. If either provides reliable predictions, the assessment of which option is best is based on the two selected GOF measures. These GOF measures were estimated by The Calibrator. The CV(C) is shown in Table 21. The CURE plots based on the SPF predicted values (shown on the x-axis) for the original SPF and the two modified SPF options are shown in Figure 3 through Figure 5. Table 21 notes the CV(C) is 0.29 for the original SPF for this crash type and severity as well as for Option 1 SPF and Option 2 SPF. It is concluded that the calibrations of the three SPFs were unsuccessful. Lyon et al. recommended an upper threshold of 0.15 for the CV(C). Nevertheless, the practitioner may still decide to select an SPF and use it with due caution. SPF Option V(C) CV(C) Original 12 7.11 1.69 0.24 0.29 Option 1 12 11.57 1.04 0.09 0.29 Option 2 12 19.14 0.63 0.03 0.29 Calibration Factor Total Predicted Crashes Total Observed Crashes Table 21. GOF outputs for original and Option 1 and Option 2 (SD-KAB).

68 Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results Figure 3. The Calibrator CURE plot of residuals based on NCHRP Project 17-62 estimated base-condition SPF (Step 1) predictions (x-axis)—(SD-KAB) SPF at 4ST intersections on multilane roads. Figure 4. The Calibrator CURE plot of residuals based on modified NCHRP Project 17-62 estimated base-condition SPF (Option 1, Step 5) predictions (x-axis). Figure 5. The Calibrator CURE plot of residuals based on modified NCHRP Project 17-62 estimated base-condition SPF (Option 2, Step 5) predictions (x-axis).

Reliability Associated with Using a Crash Prediction Model to Estimate Frequency of Rare Crash Types and Severities 69   In this case, the CURE plots suggest that Option 1 or Option 2 would be somewhat superior to the original model in that the cumulative residuals for these options oscillate closer to the x-axis and stay within the 2σ limits. The Calibrator output indicated that the percentage of points in the CURE plot exceeding the 2σ limits is 29% for the original model, 19% for Option 1, and 13% for Option 2. There is little to choose between Option 1 and Option 2, except that the calibra- tion factor of 1.04 for Option 1 is closer to 1.0 than the calibration factor of 0.63 for Option 2 (Table 21), so, the practitioner may prefer SPF Option 1 despite the slightly larger percentage of points in the CURE plot exceeding the 2σ limits. Based on these results, the practitioner decides that even though none of the SPFs provide reliable estimates, the Option 1 SPF will be used in the short term with due caution. Option 1’s cumulative residuals oscillate closer to the x-axis, with close to 80% of them within the 2σ limits, and the calibration factor is close to 1.00. The practitioner will also search for other SPFs for evaluation.