Developing a Guide to Effective Methods for Setting Transportation Performance Targets (2023)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

35 C H A P T E R 3 Phase II: Developing and Vetting Target Setting Methods Research Approach Phase II of this research involved four components: Task 5: Develop and Select Methods for Target Setting. Task 5 involved developing methods for target setting that state DOTs can adopt for performance management that consider: data requirements; appropriate analytic methods; and how to account for uncertainty, practicality, and practitioner capacity. Building on feedback from the panel, the team developed a condensed list of methods for target setting from the list of possible target-setting methods created in Phase I. The research team documented each of the proposed methods with the details necessary to guide a practitioner in using them for target setting including the following components: • Data requirements • Quantitative analysis method • Qualitative analysis method (if applicable) • Summary of applicability of and rationale for using selected method. Task 6: In-person Meeting with NCHRP Panel. Due to travel restrictions from COVID-19, the research team met virtually with the NCHRP project panel to focus on the proposed target-setting methods and details for application. The meeting also included a discussion of the planned approach to validate the proposed methods with state DOTs. Task 7: Validate Targets with State DOTs. The research team worked to validate the proposed methods for target setting by testing the methods in coordination with state DOTs. The research team advertised the opportunity to pilot the target setting methods by creating a flyer, which American Association of State Highway and Transportation Officials (AASHTO) distributed. Additionally, the research team participated in the AASHTO/Transportation Research Board (TRB) Committees Transportation Asset Management meeting on May 12, 2021, to advertise the pilot opportunity. The research team vetted responses to Figure 2. Flyer used to Solicit Interest in Participating in Pilots

36 the flyer and had discussions with several states to ultimately identify pilot agencies that were interested in testing different target setting methods. The pilot agencies also assessed how easily the method is employed, how the resulting targets compare to the targets set by each state DOT’s previous method, and any potential issues that could result from using the method. Understanding the capabilities and characteristics of state DOTs validating the methods will help frame challenges and related solutions for the Guide to be developed in Phase III. Task 8: Develop Outline for Guide. The research team developed a detailed outline for the target- setting guide to be produced in Phase III. The Guide outline was structured with instructions for applying the proposed target-setting methods and to include relevant background information on the target-setting process and requirements, along with practical applications and considerations for agencies with differing resources, data, and/or polices that may impact target setting. Pilot Testing Overview Table 7 outlines the state DOTs that piloted the target setting methods for each performance measure. Table 7. Pilot State DOTs State DOT Safety (PM1) Infrastructure Condition (PM2) Reliability, Congestion (PM3) Connecticut X Minnesota X X New Jersey X Oklahoma X X South Carolina X Utah X Washington State X X

37 Target Setting Methods Pilot and Results for PM1 Safety Measures Pilot Participants Three state DOTs participated in the piloting of safety target setting methods: Washington State, South Carolina, and Minnesota. Participants from these states had experience with a range of methods for safety target setting and held differing perspectives on how aspirational targets should be. Summary of Methods Tested Based on research and interviews conducted prior to the start of the piloting phase, the research team explored and tested three primary target setting methods with the pilot participants, with one additional just for serious injuries: 1. Historical Trend Forecast – Simple forecasts such as a linear projection using only historical performance data that can be analyzed in Excel software. 2. Statistical Model Forecast – Any more sophisticated forecasting method that requires different software to analyze and that can quantitatively incorporate exogenous explanatory variables. Several different model forms were tested, including: a. ARIMA b. Negative Binomial c. Poisson 3. Forecast + Adjustment for Other Factors – A method to adjust quantitative trends based on factors identified as important to a state (developed for SCDOT only) 4. Serious Injury Ratio – A method for estimating serious injuries as a ratio of fatalities based on a method used by Michigan DOT (applies to serious injuries only) The research team did not develop or test a statistical model for the non-motorized measure because performance results tend to be very “noisy”, and there is limited literature examining the likely influencing factors on non-motorized safety outcomes, which makes developing a model more guesswork than desired. Piloting Process Piloting the safety target setting approaches included two primary elements: 1. An analytical comparison of several different statistical models with linear historical trend forecasts for all states and the District of Columbia, along with analyses for adjustment approaches and the serious injury ratio method; and 2. Discussions with pilot participants on the technical approach, results, preferences, and agency practices surrounding target setting and communication. Analytical Comparison Statistical Model and Linear Trend Methods Analysis Initial discussions with data science and statistical experts on both the research team and among pilot participants indicated skepticism about a simple statistical model’s ability to consistently provide improved predictions over historical linear projections given the infrequent occurrence and seemingly “random” nature of fatality and serious injury crashes. The research team therefore developed an analysis to compare the results of several simple statistical models with linear projection results for all 50 states plus the District of Columbia to assess whether there could be widespread benefits to a model approach.

38 The approach to comparing the different forecast methods was to build several forms of regression models using identified explanatory variables, train them on a subset of historical fatalities data, validate the models against known data for the remaining data points, and compare the error metrics and forecasted values associated with each model against the same error metrics for a linear trend forecast. The research team first ran each statistical model on years 2000-2014, “blinded” the models to the safety results from 2015 - 2018 and predicted “future” performance for these years. This allowed the research team to compare model fit on existing data, but also the models’ ability to forecast values. The predicted values were compared to the actual values for the years 2015 – 2018 based on generated error statistics to determine how accurately each model could forecast future performance results. The team researched the variables most likely to predict fatal crashes through a review of published literature and the practices of states interviewed for this project. The research team chose to include those variables with the least ambiguous results, those for which data could be collected in a short time frame for multiple states, and those which the study’s pilot participants recommended. Table 8 outlines the six variables chosen for inclusion in the final analysis. Table 8. Variables Included in the Analytical Comparison Variable Rationale Source Years of Data Unemployment rates Identified across studies, used in both Michigan and Virginia models Bureau of Labor Statistics 1980 - 2020 Per capita income Proposed by pilot participant (Minnesota) Bureau of Economic Analysis 1980 - 2019 Per capita alcohol consumption Proxy for impaired driving prevalence, used in Michigan model (beer consumption) Inter-university Consortium for Political and Social Research 1977 - 2018 Seatbelt usage A possible measure of drivers’ willingness to assume risk and therefore propensity to engage in risky driving behavior National Highway Traffic Safety Administration (NHTSA), Traffic Safety Facts 2000 - 2019 Population 16-25 Proxy for high-risk drivers Census 2000 - 2019 VMT While research showed mixed results of VMTs correlation with safety outcomes, it is commonly used by practitioners across states (South Carolina) Bureau of Transportation Statistics 1994 - 2019 To create a comprehensive data set to be used for statistical modeling, the research team joined all the data by state and year for years 2000 to 2018, then selected the temporal window so that all data sets could be used with no missing data for any variable in any year. Selecting the right model type is important to get reliable forecast results. The team began with a simple model frequently used in data science for time series data called an auto-regressive integrated moving average (ARIMA). Iowa DOT uses this model to forecast for safety targets, and it is attractive for its relative simplicity while still having an ability to integrate explanatory variables. We also explored traditional regression models like those used by Virginia and Michigan, who both use a negative binomial regression

39 model. Poisson is a related form of generalized linear regression model that a project panel members and the team’s statistical expert suggested exploring. The models considered were assessed based on metrics of how well the data fit the model, along with how well the model could accurately forecast “future” performance results. The research team reviewed model performance using two error metrics: Akaike information criterion (AIC) to assess model fit and mean absolute percent error (MAPE) to assess the model’s ability to forecast future fatality values. The AIC and MAPE values were each aggregated across all states’ results and the mean taken. For all the error metrics presented, lower values are better. • Data Fit – Data fit applies to the linear trend forecast and statistical model. Fit for the methods was measured by the Akaike information criterion (AIC), which estimates prediction errors for a given set of data. This metric is often used to assess the relative quality of multiple models in fitting the data, with the stipulation that all models must be run on the same data set to be comparable. It is often used as a model selection metric. • Forecast Accuracy – In addition to overall data fit, to be useful for setting targets the models need to show how well they predict future values. Since the models were only developed using data between 2000-2014, forecast accuracy was assessed for the years 2015-2018, for which the predicted values from the model could be compared to actual performance values. The MAPE is useful in this context to assess forecast performance because it is percentage-based, which allows it to work comparably across states with varying levels of fatalities and across model types. MAPE indicates the percentage that forecasted values tended to be different from actual values. For example, a MAPE of 9 indicates that forecasts are on average off by 9%. Overall ARIMA models showed a strong fit for the data across model forms. The models with unemployment rate as predictor variables floated to the top using either Poisson, negative binomial, or ARIMA methods in terms of forecast accuracy. This is consistent with other research of explanatory factors in safety outcomes. Of these models, the time series delivers the best performance and is simpler to create and interpret than the negative binomial or Poisson models. While adding unemployment to the ARIMA model improves performance to some degree, users should examine the value in adding this complexity versus using the extremely simple base version ARIMA with no regressors. The research team’s assessment of errors was conducted with years that have known values for both performance and for the explanatory values. For true forecasts, users will need to develop forecasts of any explanatory value used in the model. For a variable such as unemployment that often has national or state forecasts available, this may not be a large barrier. Another alternative is to lag the variable and use current unemployment as a predictor for future safety results. Either approach is likely to increase the error of predictions beyond what was estimated in this analysis. All of the ARIMA models outperformed a simple linear trend forecast on both data fit and ability to forecast performance. The improvement was notable enough to potentially warrant the additional effort involved in setting up the base ARIMA model for agencies wanting an improvement over linear trend analyses as a basis for setting targets. Years of Data Analysis The team also conducted an assessment of linear forecast results with different years of data to inform a recommendation on how many years practitioners should include. To explore the right time period to use in the linear trend projection, the team ran fatality forecasts for three different time periods. The team applied a similar process to that used in the model comparison analysis of fitting a trend line for each of the subsets of the data outlined above, validating the forecasts from that trend on the remaining known data

40 points (2016-2020), and calculating MAPE error metrics based on the comparison to actual results. The MAPEs for each time period was averaged across the three pilot states, and results are summarized below. Table 9. Mean Absolute Percent Error for Fatality Forecasts Made with Different Time Periods Time Period MAPE 1994-2015 22.3% 2000-2015 35.6% 2010-2015 8.3% The results show that for fatalities in the pilot states, using the shorter time period produces far more accurate forecasts than using a long data period to build the linear trend. The notable difference in results could potentially point to the fact that influencing factors and trends change over time and that the factors influencing safety outcomes two decades ago are no longer relevant today. Excluding these periods keeps results most relevant to the influencing factors most likely to still be at work now and in the near future. The team recommended identifying the time period most likely to be relevant for current and near future conditions, which may be a different time period for each state. Particularly important is to identify any possible shocks to the system or data and exclude or account for them if possible. This will be a particular challenge for the impact of COVID-19 in using 2020 and to some degree 2021 data, as well as for serious injuries for the year in which states switched definitions of “serious injury”. Adjustment Method Approaches Early analysis of target setting methods across the country showed that agencies often applied this approach of “adjusting” quantified forecasts to account for other factors deemed important to their state. Of the project’s pilot participants, only South Carolina developed such a forecast that incorporated other factors after an initial trend is established. This is the agency’s preferred approach that they intend to continue into the future. The agency starts with a linear trend in Excel based on the years 2008-2020. Once the linear trend is established and projected forward to the target year, SCDOT uses the agency’s estimates for future VMT to develop a VMT growth rate that is applied to the fatalities and serious injuries trends based on the assumption that they rise at roughly the same rate as VMT. Then the agency develops estimates of project impacts from their planned spending for the year, based on CMF estimates. The final target may be adjusted slightly up or down from this based on the YTD monthly trends, but it is not applied directly to the trend forecast.

41 Serious Injury Ratio Method The team examined historical fatal and serious injury counts to assess the consistency of the relationship between fatal and serious injuries among states and over time, based on Michigan DOT’s approach to setting serious injury targets as a ratio of fatality targets. To do this, the research team collected fatal and serious injury counts for each state plus Puerto Rico and the District of Columbia between 2012 and 2018 from the FHWA’s Transportation Performance Reporting website. The team calculated the number of serious injuries for each fatality in each year and compared the average of the first three years (2012-2014) with the average of the last three years (2016-2018) to get a sense for the stability of ratios over time while smoothing out some of the annual noise. We also took the average of the entire period (2012-2018) and compared among states. The ratios showed extremely wide variation among states. While the average ratio was 5.77 serious injuries for every fatality, the highest ratio was 16.60 and the lowest was 0.98, with a standard deviation of 3.13. Ratios were also not consistent over time for the same state; most states observe the ratio of serious injuries to fatalities decreasing over the 2012-2018 time period, normally by double-digit percentages. Only a handful of states kept ratios that were constant or near constant, including Michigan, North Dakota, and Puerto Rico. This change over time discourages the use of ratios for safety target setting. Results show that while this relationship does exist for Michigan, in most cases the relationship is not strong enough to warrant this approach for other states. Serious injury metrics should therefore be forecasted separately from fatalities using one of the historical trend options Discussion with Pilot Participants The research team engaged agencies in late spring 2021 to participate in the piloting of the safety target setting approaches. The team held initial discussions with each agency separately in early summer to discuss their current approaches to safety target setting, initial impressions of the research team’s proposed methods, data sources the agency may want to contribute, and how to make the results of the research most helpful to their peers. The research team held at least one follow-up discussion with each participant in late summer or early fall to review initial results and discuss the likelihood of the agency adopting any of the tested methods. Each state’s current approach to target setting is outlined in Table 10. Table 10. Pilot Participants' Current Target Setting Methods State Target Setting Method Washington State Targeted Reduction South Carolina Simple Model and Trend with Adjustment Minnesota Targeted Reduction Much of the discussion with participants centered on the philosophical difference of setting aspirational versus data-driven targets. Participants from both Washington State and Minnesota indicated a commitment to continuing their aspirational target setting approach into the future but expressed that there was value in identifying accessible ways to predict annual, statewide crashes more accurately. While the research team solicited additional state-specific data to go into the models from pilot participants, such as safety spending data, none of the participants supplied such data. South Carolina was the most interested in this, as they already adjust their projections of safety results in light of anticipated safety project types but expressed some reservations in supplying spending data directly for use in the models.

42 Minnesota’s participants expressed an interest and willingness to try the ARIMA version given its simplicity, but also acknowledged that they were likely to continue setting aspirational targets as long as they are able to (at the time of the pilot, discussions were underway with the state’s public safety office about pushback form the NHTSA on these aspirational targets and that the approach could change into the future.) Washington State’s participants, who both have PhDs in fields related to building models for safety analysis, are both committed to aspirational targets. In addition, they are skeptical that developing more refined predictions for exact future outcomes is worth the additional effort given the somewhat random and hard to predict nature of both fatalities and serious injuries. The approach they would be likely to apply if they were to set data-driven targets was a linear trend because it is simplest, and any other effort would not add significant value to the prediction. South Carolina has stopped using a formal model that incorporates VMT and have recently moved to running forecasts based on historical data in Excel. They also adjust the forecasted targets based on expected safety investments. This is the preferred approach going forward, though there was some willingness to try the ARIMA as the base forecast to then adjust for investments. The research team interviewed several states regarding their target setting for non-motorized safety that informed the team’s decision making on selected target setting. Several of their key comments are summarized below. Note: this input was obtained in previous phases of research and is reiterated here for completeness of the basis for selecting methods. • Iowa: Iowa DOT has found fluctuations in non-motorized safety performance to be highly random, producing a very “noisy” database and making it infeasible to find reliable explanatory factors. Therefore, the agency has used a moving average to forecast its non-motorized safety. • New Mexico: NMDOT has found that non-motorized safety data is often inaccurately recorded in the field, hindering their attempt to identify meaningful trends. Moreover, the fact that the federal measures combine non-motorized fatalities and serious injuries into a single measure masks the separate trends in each. • South Carolina: SCDOT uses a trendline analysis with a relatively small number of years. The small number of years was selected so that recent trends would have more influence on the prediction than previous trends. • Virginia: VDOT used to use trendlines for non-motorized safety forecasting but has since developed a safety performance model including variables such as weather, socio-economic factors, and spending on infrastructure and behavioral programs. Based on this input, the team proceeded with historical trend projections as the preferred approach for non-motorized target setting, potentially with adjustment for other factors an agency deems relevant.

43 Results The results of the pilots span several factors that inform the ultimate decision of which method to employ: • Performance – How well each method statistically fits historical data and how closely each method predicted future performance • Complexity – How technically difficult each method is • Agency Preference – Despite all the factors outlined above, each agency in the pilot expressed preferences around target setting philosophy, comfort based on previously used methods, and stakeholder preferences that ultimately hold more sway over final decisions going forward than the accuracy of other forecasts. In terms of performance, the ARIMA model with unemployment included as an explanatory variable showed the most accurate forecast results. Based on the results of this study alone, if an agency wanted the most accurate forecast to use as a target setting basis, this method of target setting should be used. However, these models only provide a marginal improvement over the other ARIMA models, increasing accuracy, and the improvement comes at the expense of increased complexity of switching from the familiar environment of Excel to another software capable of running ARIMA models, and the complexity of adding explanatory variable data and forecasts. The research team recommends that practitioners who wish to hone their target predictions beyond trend analysis use only the simplest of the statistical models tested, unless the agency is committed to getting academic and/ or statistical expert assistance to build a robust model specific to their state or region. One of the most persistent issues that arises in discussions with agencies about target setting for safety measures is an entrenched belief in one of two target setting philosophies: 1. That targets should always be aspirational, or 2. That targets should reflect the performance realistically expected from current conditions combined with agency action. Due to the high stakes of safety outcomes, a broad interest by elected officials and the public, and difficulty in consistently predicting impacts by factors outside the agency’s control, this aspect of agency preference continues to influence target setting method selection. Testing results indicate that any of the three primary methods could be recommended for safety performance target setting based on the accuracy of results they provide. Final recommendations based on this piloting process considering all these factors, along with forecasted values for 2022 performance across methods, pilot states, and measures are presented in the following table. Table 11. Pilot Methods and Recommendations Method Recommended? Linear Yes ARIMA Model, base (no variables) Yes ARIMA Model, unemployment Yes, if agency does not mind added complexity ARIMA Model, other No Poisson Model, all No Linear Trend with Adjustment Yes

44 Target Setting Methods Pilot and Results for PM2 Infrastructure Condition Measures Pilot Participants Two state DOTs, Oklahoma and New Jersey, agreed to participate in the pilot testing for the infrastructure performance measures. Summary of Methods Tested Both agencies used some version of time-series analysis to set their current national performance targets and both agencies requested to pilot the use of scenario analysis. Since the scenario analysis and model/system-based methods are very similar the results from this pilot can be used to support guidance for either approach. Both agencies agreed to participate using pavement data and their PMS. No agencies volunteered to participate in the pilot using their bridge data or management systems. Piloting Process The approach to pilot testing performed for each state varied significantly based on the data available in the respective PMSs. Table 12 provides an overview of the differences in the type and extent of data available for analysis within each agency’s PMS. Scenario analysis requires confidence in the management system forecasts but is not reliant on a specific approach to developing the forecasts. Both agencies used the Deighton dTIMS system, but the research team is confident the results are applicable to states with other management systems. Table 12. Comparison of Pilot States’ Pavement Management Data Data Type NJDOT PMS Oklahoma PMS Inventory (extent) Only contains data for state-owned NHS Contains data for the entire NHS Inventory (segmentation) Segmented for likely project locations Segmented for likely project locations Condition Contains rutting, ride quality, and faulting, but not the cracking metric used for national pavement performance measures Contains historic data for all national pavement performance metrics Deterioration Models Contains models for all metrics, but the cracking metric differs from the metric used to calculate the NHP measures Contains models for all national pavement performance metrics Treatments Contains a variety of treatments for preservation, rehabilitation, and reconstruction Contains a variety of treatments for preservation, rehabilitation, and reconstruction Treatment Warrants Contains validated warrants for the selection of treatments Contains validated warrants for the selection of treatments Performance Improvements Performance improvements are modeled for all treatments in terms of all metrics Performance improvements are modeled for all treatments in terms of all metrics

45 Data Type NJDOT PMS Oklahoma PMS Unit Costs Contains up-to-date unit costs for all treatments Contains up-to-date unit costs for all treatments Available Funding Available funding can be input as part of defining a scenario, or “run” Available funding can be input as part of defining a scenario, or “run” Prioritization Algorithm Uses dTIMS incremental benefit calculation Uses dTIMS incremental benefit calculation Both agencies possessed the ability to reliably forecast pavement performance for state-owned highways. Oklahoma DOT (ODOT) had the ability to forecast performance for all NHS pavements because the inventory and condition data for all NHS pavements are included within the PMS software. NJDOT stores data for all NHS pavements in its pavement management database, but only NJDOT-owned pavements are included in the PMS. Neither agency possessed a means to directly transform PMS forecast results from the PMS segmentation to the 0.1-mile segmentation of HPMS. There were additional differences between the agency’s capabilities when it came to the condition metrics included in the respective PMS. ODOT’s PMS includes metrics for rutting, cracking, ride quality, and faulting, in the same terms used to calculate the national pavement performance measures. This meant the only aspect of calculating the national pavement performance measures was transforming the forecasted conditions from project-based segments to 0.1-mile segments. NJDOT’s PMS did not include a performance curve for cracking metric that was similar to the metric used for calculating the national pavement performance measure (i.e., wheel path cracking). This required that another means be used to correlate conditions forecasted in terms of the NJDOT measures to the national pavement performance measures. Figure 3 shows the differences between a pavement management database and a PMS. Pavement inventory and condition data is collected by both DOTs using automated data collection vehicles. This information is processed and stored in a pavement management database at 0.01-mile segments. The granular data in the pavement management database can then be aggregated in different ways to serve various analysis and reporting purposes. For HPMS, the data is aggregated to segments that break arbitrarily every 0.1 miles. This is an easy reporting standard for states to follow. For use in the PMS, the pavement data is aggregated to segments that vary in length based on common pavement attributes and traffic conditions. PMS segments may also be divided at common project start or end points such as intersections. Within the PMS, the pavement data is used in conjunction with other inputs, (e.g., budgets, performance models, treatments, treatment warrants, and prioritization algorithms) to forecast pavement conditions under different scenarios.

46 Figure 3. Relationship between pavement management databases, systems, and outputs The issue of network segmentation is critical to the calculation of the national pavement performance measures. The national pavement performance measures are based on four pavement metrics. Each 0.1- mile segment of the HPMS inventory is assessed based on these metrics and assigned a condition rating of Good, Fair, or Poor, accordingly. The national pavement performance measures are calculated as the percentage of segments that are assigned Good or Poor (23 CFR Part490.313(f)). In most cases, the state DOT PMS base performance modeling on pavement segments that represent homogenous pavement and traffic loading attributes. Within the PMS, the condition metrics are averaged over each of the longer segments. Because of this difference in segmentation, a calculation of the national pavement performance measures cannot be directly made from PMS forecasts. Figure 3 provides a hypothetical example of how the calculation of percent Good and percent Poor can vary based on segmentation. Figure 4. Example of segmentation influencing pavement measure calculations In Figure 4, four miles of pavement are divided in two different sets of segments. The top row shows the conditions of 40 0.1-mile segments. The bottom row shows the conditions of three project-length segments. Table 13 shows how this difference in segmentation impacts the calculation of the NHP measures. This difference is often enough to significantly impact target setting. While the issue of segmentation was relevant for each of the pilot states, different approaches were required to address it due to other differences between the agencies’ PMSs.

47 Table 13. Example of Segmentation Influencing Pavement Measure Calculations Segmentation Percent Good Percent Poor 0.1 mile 27.5 27.5 Project-Length 30.0 32.5 New Jersey The research team met with NJDOT on July 2, 2021, to assess the agency’s PMS capabilities related to forecasting the national pavement performance measures. As described above, the NJDOT PMS had several limitations that make calculating the NHP measures impossible. However, NJDOT feels confident in the ability of the PMS to forecast pavement conditions in terms of the agency’s internal measures and NJDOT had developed a means of correlating its infernal measures to the national pavement performance measures for development of its 2018 TAMP and is documented in Appendix C of the 2019 TAMP. In 2018 and 2019, this correlation process was used to support the development of pavement investment strategies that supported achievement of the pavement condition targets. The research team focused its efforts with NJDOT on: • Updating the correlation process to estimate national performance measures from NJDOT performance measures. • Accounting for NHS pavement not modeled within the PMS or managed by NJDOT. Estimating National Performance Measures Based on Forecasts of NJDOT Performance Measures NJDOT has developed a composite index of pavement condition, Condition Status (CS), to categorize pavement condition as good, fair, or poor. While CS reports pavement conditions in the same terms as the PM2 measures, there are distinct differences in the metrics used and the thresholds used to set the three condition categories. Figure 5 and 6 below, from the New Jersey 2019 TAMP, describe how CS is calculated and compare the metrics used to calculate CS and the PM2 measures. Figure 5. NJDOT Surface Distress Index (Source: New Jersey TAMP)

48 Figure 6. NJDOT and PM2 Performance Measure Comparison (Source: New Jersey TAMP) Due to the differences in underlying metrics between the NJDOT CS and national pavement performance measures, the research team agreed with NJDOT that it would be best to perform a correlation based on the Good/Fair/Poor ratings provided by each approach and not the individual distress metrics. This was possible because NJDOT can report historic CS and national pavement performance measure in 0.1-mile segmentation. NJDOT produced a report from its pavement management database that provided national performance measure and CS related data for every 0.1-mile segment on the NJDOT NHS. Table 14 shows a sample of data that was used to calculate both the national and NJDOT pavement ratings of Good/Fair/Poor for each segment. Table 14. Sample of NJDOT and National Data Begin MP End MP IRI Cracking % Faulting (in.) Rutting (in.) SDI NHPP NJDOT 0.2 0.3 275 17 0 0 4.65 Poor Poor 0.3 0.4 132 24 0 0.11 1.41 Fair Poor 0.4 0.5 122 12 0 0.09 2.10 Fair Poor 0.5 0.6 144 1 0 0.10 3.17 Fair Fair 0.6 0.7 143 15 0 0.12 0.31 Fair Poor 0.7 0.8 88 3 0 0.09 2.80 Good Fair The data in Table 14 shows how ratings can vary between the two systems. Data collected in 2018, 2019, and 2020 were used to develop correlation factors to determine the likelihood of a pavement section being rated Good, Fair, or Poor in the national rating based on its CS rating. The results of this correlation are shown in Table 15 – 18 and indicate that the relationship between the two rating systems is relatively constant from year to year. In these tables the percentages indicate the likelihood that a pavement segment with the CS rating indicated by the row label receives the national rating indicated by the column heading. For example, in 2020, 88.43% of segments rated Good based on CS was also rated Good based on the national performance measure. The biggest disconnect between the two systems can be seen with percent Poor. CS rates far more pavement in Poor condition than the national performance measures. This is primarily because NJDOT considers multiple types of cracking, but the national performance measure only considers wheel path cracking.

49 Table 15. 2018 NJDOT to National Pavement Rating Correlation National Good National Fair National Poor NJDOT Good 89.18% 14.89% 0.00% NJDOT Fair 22.01% 77.97% 0.02% NJDOT Poor 4.78% 87.29% 7.93% Table 16. 2019 NJDOT to National Pavement Rating Correlation National Good National Fair National Poor NJDOT Good 85.11% 14.89% 0.00% NJDOT Fair 19.10% 80.90% 0.00% NJDOT Poor 5.66% 85.75% 8.59% Table 17. 2020 NJDOT to National Pavement Rating Correlation National Good National Fair National Poor NJDOT Good 88.43% 11.57% 0.00% NJDOT Fair 30.12% 69.88% 0.00% NJDOT Poor 8.65% 85.01% 6.34% Given the relative consistency between these annual results, the research team recommended that NJDOT use a three-year average to transform forecast results from CS ratings to national performance measures. These averages are shown in Table 18. Applying these factors to the baseline conditions in the PMS will not result in a replication of the national pavement performance measures reported in HPMS. This is because the PMS only contains data for NJDOT owned pavements, which excludes approximately 40% of the NHS. To account for the NHS owned by others, NJDOT needed to make assumptions regarding how those portions of the systems will perform relative to the NJDOT-owned system. NJDOT made the following assumptions. • Pavements owned by local governments will perform similarly to NJDOT pavements; and • Pavements owned by toll authorities will see constant conditions over the analysis period. Table 18. 2018-2020 Average NJDOT to National Pavement Rating Correlation National Good National Fair National Poor NJDOT Good 87.57% 13.78% 0.00% NJDOT Fair 23.74% 76.25% 0.01% NJDOT Poor 6.36%% 86.02% 7.62% These two assumptions are based on the funding available to manage the different portions of the network. Local-owned NHS pavements are largely managed using federal funds and complete for funding with NJDOT pavements. Toll-authorities have dedicated revenue streams for their pavements and are able to provide a higher overall level of service. The following tables show how this process was applied to the results of a single PMS scenario, with a constrained budget of $320 million per year, to transform the forecasts of NJDOT pavements, in terms of CS, into forecasts of interstate pavements, in terms of the

50 national performance measures. The same process was being followed to calculate the conditions for non- interstate NHS pavements and will be followed for multiple scenarios. The resulting forecasted conditions in terms of CS are shown in Table 19. Table 19. Forecasted NJDOT Pavement Conditions in terms of NJDOT’s Condition Status 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 % Good 27 33 20 14 14 16 16 18 23 22 23 24 % Fair 45 43 52 49 37 22 15 13 11 12 12 15 % Poor 28 23 27 37 49 62 69 70 66 66 65 61 Total 100 100 100 100 100 100 100 100 100 100 100 100 Table 20 shows the conditions of the forecasted NJDOT network transformed from condition status to the national pavement performance measures using the factors from Table 4-8. Table 20. Forecasted NJDOT Pavement Conditions (Percentages) in Terms of the National Performance Measures 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 Good 36 41 32 26 24 23 22 23 27 26 27 29 Fair 62 57 66 71 72 72 73 72 68 69 68 67 Poor 2 2 2 3 4 5 5 5 5 5 5 5 Total 100 100 100 100 100 100 100 100 100 100 100 100 Table 21 shows the forecasted NJDOT pavement conditions adjusted to align with the baseline year conditions for interstate pavements. This adjusts the starting point of the condition trends, but assumes subsequent years perform similarly from that starting point (i.e., the starting point of the curve is adjusted, but the shape of the performance curve remains unchanged). Table 21. Initial Forecasted Interstate Pavement Conditions (Percentages, Not Adjusted for Ownership) 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 Good 62 67 58 52 50 49 48 49 53 52 53 55 Fair 36 32 40 45 47 46 47 46 43 43 42 41 Poor 2 1 2 2 3 4 5 5 5 5 5 4 Total 100 100 100 100 100 100 100 100 100 100 100 100 The forecast in Table 21 does not account for the assumption that toll-authority interstates will remain constant in condition while NJDOT-owned interstates will follow the performance curve. To adjust the results of Table 21 to reflect the different expected performance between jurisdictions, the research team calculated the percentage of interstate pavement segments in each condition rating category owned by each jurisdiction These results are summarized in Table 22. For example, 18.37% of good interstate pavements were owned by toll authorities. The percentages shown in Table 22 were then used to hold conditions for toll-authority pavements constant, while applying the predicted performance to the NJDOT and local

51 governments’ pavements. A final minor adjustment was made to ensure that the totals for each forecasted year sum to 100%. The final forecast of interstate pavements in terms of national pavement performance measures are shown in Table 23. Table 22. Percentage of Base Year Interstate and Miles by Jurisdiction Toll Authorities NJDOT & Local Governments Good 18.37% 81.63% Fair 35.10% 64.90% Poor 38.30% 61.70% Table 23. Final Forecast of Interstate Pavement Conditions (Percentages) in Terms of National Pavement Measures for Budget Scenario $320 Million 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 Good 62 65 59 55 53 53 52 52 55 55 56 57 Fair 36 33 39 43 44 44 44 44 41 41 41 40 Poor 2 2 2 2 3 3 4 4 4 4 4 3 Total 100 100 100 100 100 100 100 100 100 100 100 100 Oklahoma The research team met with ODOT on July 1, 2021, to assess the agency’s PMS capabilities related to forecasting the national pavement performance measures. The ability hinged on three features of the ODOT PMS. 1. ODOT collects and stores the pavement metrics as defined in 23 CFR part 515 in its PMS and has performance models for each metric. This allows ODOT to forecast conditions in terms of the national performance measures. 2. ODOT’s PMS is segmented using the same linear referencing system (LRS) as HPMS. This allows the conditions forecast in the PMS (which uses longer segments) to be transformed to the 0.1-mile segments in HPMS. 3. ODOT’s PMS contains inventory and baseline condition data for the entire NHS pavement inventory so no adjustment needs to be made to forecasts from the PMS to account for local-owned NHS conditions. 4. In the ODOT PMS all forecasting is done in terms of each distress metric. The forecasted distress conditions are then used to calculate measures in each year of the forecast. The primary challenge to using the ODOT PMS to support target setting was to map the 0.1-mile HPMS segments to the project segments. This was somewhat complicated by the fact that the project-length segments were defined with an accuracy of 0.01 miles. Because of this level of precision, the project-length segments did not usually end at the exact location of the 0.1-mile HPMS segments. Often there was an overlap, so some 0.1-mile segments were comprised of two different project-length segments. If a treatment was applied to one of these project length-segments, then that portion of the 0.1-mile segment would improve while the remainder of the 0.1-mile segment would deteriorate. To accommodate this occurrence, weighted averaging was used to calculate the conditions of these transitional segments.

52 Once the different segmentation approaches were mapped ODOT provided the results of four different scenarios. 1. Planned funding 2. Planned funding plus 10% 3. Planned funding minus 10% 4. Current programming ($250 million for preservation and $450 million for capital) The research team transformed the forecasted results for each scenario from project-level to 0.1-mile segmentation and calculated the national performance measures for every year of each scenario. Results The results from these two pilots have been shared with the pilot agencies, and both agencies have expressed an interest in using this approach to provide inputs to the target setting process. It is not expected that either agency will simply select targets based on forecasted conditions but understanding the similarities and differences between different scenarios will enable the agency to assess the sensitivity of different potential programming decisions. This will in turn allow them to understand the risk to meeting any proposed targets. New Jersey Figure 7 and Figure 8 provide a comparison of the original PMS output and the transformed forecast of interstate conditions. Figure 7 shows the forecasted conditions of NJDOT pavements in terms of condition status. Figure 8 shows the transformed results of the PMS output to forecast interstate conditions in terms of the national pavement measures. While the scenario shown indicates a dramatic change in conditions over the NJDOT network in terms of condition status, the national performance measures for interstates are expected to remain fairly steady. This is understood to be due to several factors including: 1. The relative insensitivity of the national pavement performance measures. 2. The generally good conditions of interstate pavements. 3. The stabilizing effect of toll authority pavements.

53 Figure 7. Forecast of NJDOT Pavement Conditions in terms of Condition Status for Budget Scenario of $320 million Figure 8. Forecast of Interstate Pavement Conditions in terms of National Performance Measures for Budget Scenario $320 million

54 These results are encouraging that even states with limited data can develop procedures to effectively forecast the national pavement measures. This pilot demonstrates that states with limited data can still use their PMS to inform target setting, as long as the agency has confidence in the PMS results. Oklahoma ODOT is an example of an agency with a well-developed PMS that can be used to directly forecast the national performance measures with minimal effort. While the calculations for this pilot were performed in Microsoft Excel, the agency could implement changes within the PMS to allow the national performance measures to be provided as direct outputs of the PMS. Figure 9 and Figure 10 show example results that can be used to support target setting. These graphs show the percentage of poor pavement, in terms of the national performance measures, from each scenario, for interstate and non-interstate NHS pavements, respectively. Figure 9. Forecasted Percent Poor Interstate Pavement under Different Scenarios

55 Figure 10. Forecasted Percent Poor Non-Interstate NHS Pavement under Different Scenarios Observations and Applicability The results of the pilots suggest that agencies with a functioning PMS can implement a scenario analysis or system/model-based approach to target setting, even if the PMS does not calculate the national performance measures or model performance based on the HPMS 0.1-mile segmentation. The two agencies in this pilot used very different configurations of the same PMS software to successfully forecast interstate and non-interstate NHS pavement conditions in terms of the national pavement performance measures. ODOT’s PMS was able to forecast the metrics needed for the national performance measures for the entire NHS. This meant that only the issue of segmentation needed to be resolved to calculate the national performance measures. NJDOT’s PMS could not forecast all of the needed metrics, and did not include the entire NHS, the PMS outputs could still be transformed to provide forecasts of NHS pavement conditions in terms of the national performance measures, in a way that NJDOT is confident can support target setting. Table 24 provides a list of major challenges to forecasting national pavement performance measures and approaches for overcoming each. These challenges and resolutions are expanded upon in the Guide to support agencies in using their PMS to support target setting. Agencies can identify the challenges that are relevant to their situation and determine the most appropriate resolution.

56 Table 24. Challenges to Forecasting National Pavement Performance Measures in Standard PMS Challenges Possible Resolutions Differences in segmentation between HPMS and PMS data. Aligning both sets of segments to the same linear referencing system allows them to be mapped to each other. Lack of data or performance models for national performance metrics in the PMS Collection of needed data and development of models. Correlation of state-specific good/fair/poor measures to national performance measures Lack of data for portions of the NHS Expand PMS inventory to include missing data Estimate performance of missing portion based on comparison of baseline HPMS data between those portions and the state-owned portion. Lack of information on planned investments from other owners Gather needed information Estimate based on past investments and apply to model. No agencies volunteered to apply any of the proposed target setting methods for bridges. This is believed to be due to several different factors, including: • State DOT processes for bridge management systems (BMS) to make network level projections is not as mature as the use of PMS for network level projections. • In the short term the national bridge measures are impacted mainly by bridge rehabilitation and replacement projects that change the rating of bridges from poor to good. These projects often take multiple years to construct and for results to be reported. This makes the use of future investments less important for forecasting short-term changes in condition. • Bridges have service lives in excess of 50 years, meaning the typical change in condition over a four-year period is relatively small, minimizing the benefit of using models for short-term forecasts. • Typically, all of the projects that will impact bridge conditions through the first three years of a target setting cycle are already programmed when targets are set. This means that modeling is of limited use prior to the last year of the cycle. • Forecasting the change in component-level conditions in the short term, is likely reasonably accurate using a time-series with a time series trend plus future funding approach. Although no agencies pursued the use of BMS for TPM target setting, many agencies do use their BMS to set longer-term targets such as a 10-year desired state of good repair, which is often established in state TAMPs. From a bridge management perspective, these longer-term analyses are important for establishing strategies and determining if programming (i.e., the balance of funding to different work types) needs to change. Even when changes in bridge management strategies are identified and implemented, it often takes several years to see changes in the mix of projects being delivered, due to the long development and delivery time of bridge projects. For bridges, the Guide elaborates on the benefits of modeling bridge performance long-term. Additionally, the Guide describes how the current list of programmed projects can be used within a BMS to model network level performance over the target-setting period. Drawing from these observations, the Guide demonstrates the benefits to using the agency’s PMS or BMS to evaluate different scenarios as inputs to target setting. These include: 1. Closer connection between preferred life-cycle strategies, actual investment strategies, expected future asset conditions, and targeted asset conditions. 2. Demonstrable sensitivity to changes in both funding levels and investment priorities on achievable conditions, and potential targets.

57 3. Reduced level of effort. Once the BMS and PMS, or supporting tools, are configured to output the national performance measures there is no need for separate efforts to model performance for TAM and TPM, as both results are obtainable from the same analysis runs. 4. Improved communication with internal and external stakeholders by demonstrating that the target setting process was informed by the same models used to establish investment strategies. This helps tie the benefits of achieving the targets to any other benefits used to select the preferred investment strategy from other potential strategies. Target Setting Methods Pilot and Results for PM3 Reliability, Freight, and Congestion Measures Pilot Participants Representatives from five state DOTs participated in the piloting of the PM3 measures. The states included Connecticut, Minnesota, Oklahoma, Utah, and Washington. Summary of Methods Tested The FHWA PM3 rule covers six performance measures, covering travel time reliability of the Interstate and non-Interstate NHS, freight reliability on the Interstates, and the Congestion Mitigation and Air Quality Improvement (CMAQ) measures (annual hours of PHED per capita and non-SOV mode share). For the purposes of the pilot testing of methods, the CMAQ emission reduction measures were not analyzed. The identified methods for PM3 target setting varied from simpler methods such as building the baseline and adjusting the target based on judgment to developing a robust, detailed segment-level statistical model. Table 25 below shows a summary of the proposed methods for the PM3 performance measures. The PM3 performance measures are different from the PM1 and PM2 measures because not all targets are developed for the same geography: while the NHS travel time reliability and the freight reliability targets are set at the state-level, the PHED and Non-SOV measures are developed for the UZAs. Some of the methods listed in the table are not applicable to all the PM3 performance measures for target setting.

58 Table 25. Proposed Methods for PM3 Target Setting Method Interstate and Non-Interstate NHS Reliability Freight Reliability Annual hours of PHED per capita Non-SOV Mode Share Building off Baseline, with Assumptions 0 0 0 0 Time-Series Trend Analysis X X X X Trend plus other Factors X X X X Performance Risk Analysis X X NA NA Segment Risk Analysis X X NA NA Segment Level Statistical Model 0 0 NA NA Travel Demand Forecasting Model 0* 0* 0 0 Note: In the table above NA – means the method is not applicable for the performance measure X – means the method was tested and considered valid as an approach for target setting 0 – means the method was not tested for the pilot (due to data or resource issues or participants’ lack of interest) * This method was suggested for testing in the pilot but was not found to be applied anywhere based on the research conducted for Phase I of this study and was not able to be applied in the pilot Piloting Process: Travel Time Reliability and Freight Reliability Measures FHWA provides the data needed for the travel time reliability and freight reliability measures via the NPMRDS program. State DOTs and other agencies can access the NPMRDS data via the NPMRDS Analytics web tool developed by the Regional Integrated Transportation Information System (RITIS). The NPMRDS analytics web tool allows users to access and download the monthly and annual level performance measure data since 2016. The vendor providing the NPMRDS data had changes in 2017, and hence there may be some differences in the methods in the datasets before 2017 (NPMRDS v1) and after 2017 (NPMRDS v2). Some state DOTs preferred to use the data from NPMRDS v2 to ensure consistency in data. In contrast, some DOTs used the data from 2016 as there is a limited amount of historical data available for statistical analysis. Because of the effects of the reduced travel due to the COVID -19 related lockdown, all states exhibited a regime shift in the performance of the roadways. The use of 2020 performance measure data was a point of consideration, and different states took different approaches, either by disregarding the 2020 data (because of the belief that the 2020 scenario would not occur in the near future) or by taking the 2020 data into consideration for forecasting the performance measures. State DOTs were required to develop the next phase of PM3 targets for submittal by October 2022. Most agencies wanted to use the target-setting pilot to try out new, unfamiliar methods, which could be later used to develop targets in 2022 when the targets are developed for submitting to FHWA.

59 Reliability Methods Tested by the Pilot States and Results The pilot states tested the following methods for target setting for the NHS and Freight reliability performance measure. Time Series Trend Analysis All the state DOTs tried applying the method for one of the performance measures related to NHS or Interstate travel time reliability. ODOT analyzed the monthly trends from 2016 to 2021 for the reliability and freight performance measures. ODOT observed a drastic change in the regime for the monthly Interstate reliability measure once lockdowns due to COVID-19 started in March 2021. ODOT split the monthly Interstate reliability figures into three regimes: pre-COVID (January 2016 to February 2020) and COVID (March 2020 to February 2021) and post-COVID (March 2021 to June 2021). ODOT believes that traffic patterns observed from March 2020 to February 20201 are not representative of what may happen in the future. In fact, the Interstate reliability measure values since March 2021 were observed to go back to the pre-COVID regime. ODOT plans to develop targets using linear forecasting of observed monthly Interstate reliability measure values by excluding the Interstate reliability measure data from March 2020 to February 2021. Figure 11. Time series trend analysis for monthly Interstate reliability measure for Oklahoma Trend Plus other Factors The trend plus other factors method was another popular method tried by most state DOTs. Utah DOT was able to collect data on explanatory factors that could influence the Interstate travel time reliability performance measure. The explanatory factors included VMT, population and employment growth, as well as GDP for the entire region. Unfortunately, UDOT could only gather the historical data, and no future

60 forecast data could be collected during the pilot. Based on the analysis of how the explanatory factors have influenced the historical performance of the Interstate reliability measure, UDOT plans to adjust the targets for the future years. UDOT also plans to collect forecasted data for the explanatory factors, which could help in target setting. Figure 12 shows the Interstate travel time reliability measure values mapped against statewide VMT, GDP and population and employment estimates. The annual Interstate reliability measure values show a clear correlation to some explanatory factors; for example, when the statewide VMT drops, there is a corresponding increase (improvement) in the statewide Interstate travel time reliability measure values.

61 Figure 12. Yearly Interstate travel time reliability measure values mapped with statewide VMT, GDP, population and employment estimates WSDOT compared the historical trend line of TTRI (2017-2020) with relevant other factors, including Interstate VMT, employment and population forecast, historical GDP data. However, the historic TTRI 28000M 30000M 32000M 34000M 80 90 100 2017 2018 2019 2020 2021 2022 VM T In te rs ta te R el ia bi lit y M ea su re VMT % reliable VMT Linear (% reliable) 140,000 160,000 180,000 200,000 80 90 100 2017 2018 2019 2020 2021 2022 GD P In te rs ta te R el ia bi lit y M ea su re GDP % reliable GDP Linear (% reliable) 3M 3M 3M 3M 4M 80 90 100 2017 2018 2019 2020 2021 2022 Po pu la tio n In te rs ta te R el ia bi lit y M ea su re Population % reliable Population Linear (% reliable) 2M 2M 2M 2M 80 90 100 2017 2018 2019 2020 2021 2022 Em pl oy m en t In te rs ta te R el ia bi lit y M ea su re Employment % reliable Employment Linear (% reliable)

62 data did not show any correlation with the VMT or economic indicators such as GDP, Employment or Population totals. Also, WSDOT realized that the 2019 TTRI data provided by RITIS NPMRDS showed some quality issues as the 2019 data showed significant improvement in TTRI from the 2018 values. The observed improvement in TTRI in 2019 was contradictory to the TTRI trend reported by FHWA’s freight mobility data tool as well as the state’s ITS data. On further investigation, it was noted that RITIS is reporting 200 less traffic management center (TMC) miles for 2019 as compared to 2018, which is questionable. CATT lab is currently investigating the issue and thinks that this issue could be because of some issue related to network conflation. Connecticut DOT did not complete the pilot for this measure. Segment Risk analysis Method The segment risk analysis method comprises analysis of the individual segments' performance and the risk that individual segments would shift from reliable to unreliable. FHWA has defined a threshold for the LOTTR (1.5) for the reliability performance measures over which the segments are deemed unreliable. For the Freight performance measure (TTTR), there is no defined threshold over which the segment would be considered unreliable; hence the segments with risk of performance degrading (substantial increase in TTTR) were analyzed. The segment risk analysis methods require additional assessment of the performance on a segment level, allowing for more detailed analysis, which gives additional insights into the performance of the segments, corridors, and the transportation network. Figure 13. Application of Target Setting Segment Risk Analysis Method by MnDOT MnDOT tried developing a draft approach for target setting using the segment risk analysis method for the pilot project. For this draft approach, MnDOT did not take into account the 2020 LOTTR values affected by COVID. The analysis used a subsection of statewide TMCs. In order to apply the trend analysis, the TMCs need to be available for all years (2015 – 2019). The segments were categorized based on the value of LOTTR as below: • Good (<1.4) • Barely good (1.4-1.5) • Barely bad (1.5-1.6) • Bad (>1.6)

63 MnDOT used 2015 to 2019 LOTTR data to perform trend analysis and forecast for year 2024. The 2024 forecasts were based on the TREND function in excel, which forecasts using a linear trend using the method of least squares. Based on the 2024 forecasted LOTTR values by segment, the performance measure was computed again for 2024. Based on the change in the LOTTR value between 2019 and 2024, the segments were categorized as below for visualization (see Figure 11 above). • Remained Reliable (2019 and 2024 LOTTR <1.5) • Remained Unreliable (2019 and 2024 LOTTR >1.5) • Became Reliable (2019 LOTTR > 1.5and 2024 LOTTR < 1.5) • Became Unreliable (2019 LOTTR < 1.5and 2024 LOTTR > 1.5) The computed 2024 performance measure will be used to inform the 2024 target. Washington State DOT (WSDOT) tried the segment risk analysis method. WSDOT wanted to adjust the TTRI values of the segments, which had a risk of worsening due to the presence of construction projects on the segments or corridors. The idea was to adjust the forecasted TTRI values of the segments over which construction was expected to happen in the year for which targets were to be developed. To test how construction projects affect the TTRI values, WSDOT overlaid a layer of construction projects with a layer of TMC segments in a GIS analysis software and identified segments with construction work between 2017- 2019. Each segment was labeled whether there were any construction projects, and if so, in which years. Then change in TTTR between 2017-2018 and between 2018-2019 was examined for those segments with construction work. Specifically, WSDOT evaluated those segments with construction for one year and no construction work for the other year to separate the construction effect. The absolute change in TTTR for those segments in two consecutive years (with and without construction) was negligible. Most segments did not exhibit any change, while a small fraction of segments demonstrated an increase/decrease in performance. The aggregated effect for all segments weighted by length was almost zero. The percentage change in median travel time for those segments in two consecutive years was also negligible with most segments showing no change. The aggregated effect for all segments was examined by calculating average travel speed for the segments and was also found negligible. The result of the analysis was counterintuitive and did not provide a correlation of worsening TTRI values due to the presence of construction on the segments. WSDOT did not pursue the application of the Segment risk analysis method further because of the findings. Performance Risk Analysis Minnesota DOT utilized a variation of the performance risk analysis method by using the monthly data for Interstate Reliability to develop a box and whisker plot using Microsoft Excel, which allows them to visualize the variance in the monthly data over the past years and monitor outliers. The variance in the past monthly data will be used to inform the target setting. Figure 12 below shows the box and whisker plot developed by MnDOT for the Interstate reliability performance measure.

64 Figure 14. Box and Whisker Plot for Monthly Interstate Reliability measure for Minnesota (Data Source: RITIS NPMRDS) *Note: The 2021 Monthly Interstate reliability measure (% reliable) data includes data only up to June 2021 Table 26. Monthly Interstate Reliability Values for Minnesota and Computed Statistics Monthly Interstate Reliability Measure (% reliable) 2017 2018 2019 2020 2021 January 82.5 83.3 90.2 89.3 99.7 February 88.5 86.6 74 89.4 99.3 March 89.4 88.8 85.7 98.6 98.2 April 81.4 82.6 80.3 100 95.4 May 77.1 77.1 79.3 99.7 93.5 June 75.3 77.3 78.3 99.5 92.9 July 81.8 79.5 80 98.7 August 79 79.7 81.6 98.8 September 75.8 79 78.5 98.3 October 81.6 83 79.9 98.8 November 83.7 82.1 83.6 99.4 December 81.9 85 81.6 99.5 Statistics Min 75.3 77.1 74 89.3 92.9 First Quartile 78.5 79.4 79.1 98.5 93.9 Median 81.7 82.3 80.1 98.8 96.8

65 Monthly Interstate Reliability Measure (% reliable) 2017 2018 2019 2020 2021 Third Quartile 82.8 83.7 82.1 99.5 99.0 Maximum Value 89.4 88.8 90.2 100 99.7 Mean 81.5 82 81.1 97.5 96.5 Range 14.1 11.7 16.2 10.7 6.8 Standard Deviation 4.41 3.64 4.07 3.84 2.97 Piloting Process: Non-SOV Mode Share The target for non-SOV mode share performance measure is developed and reported on an UZA scale. All states and MPOs with NHS mileage that overlaps within the applicable UZA must coordinate on a single, unified target and report on the measures for that area. States and MPOs have an option of various data sources, including the Census ACS mode share data, local travel survey, or samples collected in the region. Most States choose the ACS-provided data on the percentage of workers aged 16 and older who commuted to work using a transportation mode other than a single-occupancy vehicle. Only a few state DOTs were interested in developing targets for non-SOV mode share as the MPOs covering the UZA typically lead the target setting, with support from state DOTs. Non-SOV Mode Share Methods Tested by the Pilot States and Results In the pilot, only a few states tried developing the non-SOV mode share targets. WSDOT tried to gather the non-SOV mode share numbers from the regional travel demand model for the Puget Sound region but could not obtain the results from the regional travel demand model for the base and the future year in time for performing the analysis for the pilot study. Finally, WSDOT used the historical data on non-SOV mode share from ACS (2012 to 2016) to forecast the 2022 and 2026 non-SOV mode share targets based on both linear and exponential functions available in Excel. For the pilot, WSDOT developed the targets both for the Seattle area UZA and the entire Washington State. Table 27. Time Series Trend Analysis and Forecasting Method Used by WSDOT for Non-SOV Mode Share Non-SOV Mode Share (%) Linear Forecasts Exponential Forecasts Washington State 5-year Seattle 5- year Washington State 5-year Seattle 5-year 2012 27.6% 31.2% 27.6% 31.2% 2013 27.3% 31.1% 27.3% 31.1% 2014 27.3% 31.3% 27.3% 31.3% 2015 27.4% 31.5% 27.4% 31.5% 2016 27.7% 32.0% 27.7% 32.0%

66 Non-SOV Mode Share (%) Linear Forecasts Exponential Forecasts Washington State 5-year Seattle 5- year Washington State 5-year Seattle 5-year 2017 27.6% 32.0% 27.6% 32.1% 2018 27.6% 32.2% 27.7% 32.3% 2019 27.6% 32.4% 27.7% 32.5% 2020 27.6% 32.6% 27.7% 32.7% 2021 27.7% 32.8% 27.8% 32.9% 2022 27.7% 33.0% 27.8% 33.1% 2023 27.7% 33.2% 27.9% 33.4% 2024 27.8% 33.4% 27.9% 33.6% 2025 27.8% 33.6% 28.0% 33.8% 2026 27.8% 33.8% 28.0% 34.0% Piloting Process: Annual Hours of Peak Excessive Delay (PHED) per Capita The target for the Annual PHED per capita is developed and reported on a UZA-level. Typically, the state DOTs and MPOs covering the UZA develop the targets in coordination. State DOTs obtain the data for the excessive delay covering the NHS typically from the RITIS NPMRDS Analytics tool. In the pilot, only a few state DOTs were interested in developing targets for Annual PHED per capita as the MPOs covering the UZA typically lead the target setting, with support from state DOTs. Table 28. Recommended methods for annual hours of PHED per capita target setting Method Reason for Recommending Building off Baseline with Assumptions This method though quantitative, is easiest to implement and works well for agencies that are not able to gather historical data on past performance or factors that can influence performance. Trend Plus other Factors The method helps understand correlations between the performance measure and the explanatory factors. The adjustments to the targets can be made using judgment, without the use of sophisticated forecasting models Time Series Trend Analysis Time Series Trend analysis allows visualizing small and large shifts especially due to regime shifts (Example: change in regime post-COVID-19) Travel Forecasting Model This method is fully data-driven and helps make a fuller understanding of the cause of outcomes. supports linking the target setting process with decision-making by informing what factors can be influenced

67 PHED per capita Methods Tested by the Pilot States and Results Not many state DOTs chose to set targets for the PHED performance measure as part of this pilot as they are expected to be set by the regional MPOs covering the UZA with support of the state DOTs. WSDOT tried the time series trend analysis method and used the annual hours of PHED per capita data from 2013 to 2020 to forecast 2022 and 2026. WSDOT used the yearly NPMRDS data and developed the 2022 forecasts by including and excluding the 2020 data. Figure 15. Time Series Trend Analysis for Annual hours of PHED per capita for Seattle UZA Connecticut DOT noted that it was planning to coordinate with the MPOs in the region to obtain data on delays for setting targets for the Annual Hours of PHED per capita performance measure. Connecticut DOT was not able to gather the data in time for the pilot. WSDOT tried collecting the Annual Hours of PHED per capita data for the Peugeot Sound region but was not able to obtain the data from the regional travel demand model for the base and the future year. The state DOTs tried using various methods for target setting as outlined in the preceding section. Since the next round of performance targets for reliability were due on October 1, 2022, the state DOTs indicated they would utilize the methods they had tried to support the 2022 reliability performance targets. By the time the states start developing the targets for reporting to FHWA, the states will have additional data, including complete data for 2021 and at least half a year of data for 2022. Having additional data will be all the more important, as travel trends post-COVID may become more apparent. The state DOTs may not use the targets developed using the quantitative techniques at face value and may tweak the target values to better align with the region's goals, policy decisions, and general consensus on changes in travel patterns with stakeholders and partners. Observations and Applicability While applying the trend plus other factors method, the forecasted data on explanatory factors was sometimes not readily available. There were also concerns as some of the forecasts for the explanatory factors were developed pre COVID-19 pandemic, and it was questionable whether the forecasts would hold

68 true, given that the COVID-19 pandemic could not have been predicted. It is generally accepted that the COVID-19 pandemic could have long-term effects change in travel behavior and economic factors, but the extent and the duration of the impact are unknown. Though the segment risk analysis method may help bring some insights into the performance, some state DOTs thought that the level of effort for utilizing the method was excessive and chose to use simpler data- driven methods such as trend analysis and forecasting. Some state DOTs used a simple variation of the performance risk analysis method, analyzing the variance in the monthly data and did not develop the confidence intervals, as it would have required the use of specialized statistical software. None of the agencies tried developing a statistical model for target setting on the reliability performance measures, as developing such a sophisticated model would require a significant level of effort, a period of at least a few months, and sophisticated data analysis. State DOTs questioned the utility for developing such an intense, sophisticated model just for developing targets, given that this method may not bring any additional value as compared to simple analytical methods. Some state DOTs tried gathering data from the regional models for the non-SOV mode share and the annual hours of PHED per capita performance measures but were unable to gather the required data during the pilot. Gathering such data from the partnering agencies could take some time and would have needed to be planned in advance as the regional model may have to be run for the selected base or forecast year for the exact area for the UZA under consideration. If the data from the regional model were not available for the years or the area under consideration, adjustments would potentially need to be made on the model results. Overall Observations from Piloting the Target Setting Methods The piloting of target setting methods involved participants from seven state DOTs, who agreed to work with the research team to attempt to implement promising methods for setting targets. The pilot effort was able to successfully test different target setting methods for each category of performance measure, with the exception that no agency in the pilot chose to analyze bridge condition measures (only methods for pavement condition measures were tested) under PM2. Several broad observations emerged from the overall set of pilot tests. These are highlighted below as possible “takeaways” or lessons learned. Takeaway 1: Data Collection and Coordination Challenges Within the timeframe of this study, all agencies faced challenges in gathering the data necessary to test the full set of identified methods for specific performance areas. Staff time, resource constraints, and coordination with other agencies were challenges for this pilot test and likely will remain challenges for state DOTs in using some of the more complex and data-intensive methods as part of the target setting process. The research team supported efforts to gather data or adapted the approach for testing methods in some cases in response to these challenges. Specifically: • For the PM1 safety target setting that involves development of a statistical model to forecast fatalities and serious injuries, the research team developed a regression model using national data from all states rather than developing specific regression models tailored to each individual state, given the challenges of collecting significant amounts of data for potential explanatory variables individually for each state within the timeframe of this validation exercise.

69 • For the PM2 pavement condition measures, NJDOT faced challenges in collecting all of the pavement inventory data. The pilot effort at NJDOT was led by the pavement management unit. However, data needed to correlate the NJDOT condition measures with the PM2 condition measures was owned by a separate highway inventory unit. An internal data request was needed to provide a custom query of data from the highway inventory unit. This data request proved challenging and more time consuming than anticipated. Although the needed data was eventually provided, time did not allow for evaluation of all the potential investment scenarios. • For the PM3 congestion (peak-hour excessive delay and non-SOV mode share) measures, which are focused on the UZA level, the state DOTs faced challenges in coordinating with MPOs to conduct runs using the regional travel demand forecasting model, which was one of the recommended methods for testing. In general, the lack of consistent historical data is a challenge for the PM3 reliability and freight measures, as well as the peak-hour excessive delay measure. Takeaway 2: Data Quality Challenges In some cases, the analysis revealed concerns about data quality. This was most notable in relation to data from the NPMRDS used for the reliability, freight, and peak-hour excessive delay performance measures. State DOTs identified questionable data from NPMRDS for some roadways or some periods of the day, creating challenges for the analysis. For instance, ODOT saw a large different in the non-Interstate LOTTR values between 2018 and 2019, which made trend line analysis problematic. WSDOT noted a significant improvement in 2019 reliability for both the reliability and freight measures, based on conflation of the network from the RITIS. These results, however, conflicted with the agency’s own travel data being collected from intelligent transportation systems (ITS), as well as the FHWA’s freight mobility data trend. Takeaway 3: Challenges in Assessing Trends due to COVID-19 Impacts For all of the primary performance measures explored in the pilot testing, a simple trend line analysis or trend line analysis with some adjustments were explored as possible target setting methods that are relatively simple and easy to apply. Problems arose, however, with conducting trend analysis due to the dramatic changes in vehicle travel caused by the COVID-19 pandemic for the PM3 reliability, freight, and peak-hour excessive delay measures, as well as the non-SOV mode share measure and PM1 safety measures. Where data were available, trend analysis showed a clear change from the historical trend post- COVID compared to pre-COVID, making the trend analysis not very meaningful if applied in a simple way. The 2020 data could be viewed as an anomaly rather than as part of a trend, and this raised questions about whether the 2020 figures should be thrown out, essentially assuming that trends would go back to pre-COVID patterns, or whether new assumptions should be made about patterns for the future. These are challenging issues given the high levels of uncertainty about how travel patterns may have shifted somewhat permanently (due to increased telework, hybrid work arrangements, and changes in use of e-commerce, etc.). In response, the research team supported the pilot states in conducting analyses that displayed trend lines pre- and post-COVID (to the extend data were available) to help staff consider how the trends differed and consider whether to go back to the historical trend or make alternative assumptions. Pilot states found this analysis to be very helpful and indicated it would be useful in future target setting. Takeaway 4: Preference for Simpler Methods Although more complex methods may provide more robust predictions of future system performance, the transportation agencies in the pilot overall consistently expressed a preference for simpler methods using trend-lines and conducting analyses in Excel, rather than developing more complex statistical models.

70 While the sample of agencies participating in the pilot was small and may not be representative of all state DOTs, those participating in the pilot generally felt that simpler methods would be more efficient and serve the purpose of supporting target setting and preferred this approach rather than investing in more complex regression model development for the purpose of target setting. Developing statistical models requires considerable data associated with explanatory variables, and even when historic data are available on these variables, forecasts for these variables may not be available and may have high levels of uncertainty. Moreover, there is a lack of understanding of all of the factors that influence some measures, like fatalities and serious injuries. As a result, staff were concerned that the additional time and resources for developing more complex, data intensive models would not be worthwhile and that these methods may not actually provide as much improvement in overall forecasts as would be desired. The PM1 pilot testing, which involved development of a statistical model to estimate fatalities and serious injuries, suggested that a historical trend or historical trend adjusted for other factors performed nearly as well for a much lower level of effort than developing a statistical model to account for the many factors that may influence fatalities. The high level of commitment of staff resources to conducting more complex analysis methods was seen as a barrier to their application. It should be noted that for the PM2 pavement condition measures, time-series trend analysis represented the simplest solution in past target setting efforts when the national performance measures could not be forecast with the standard asset management systems. ODOT used this “easier” method during previous target setting. However, once their management systems, or external spreadsheet tools, are configured to calculate the national performance measures from the asset management system outputs, use of the asset management system becomes the simplest solution, and this pilot helped to demonstrate how using the management system could be easy. This results in a win-win for the agency since the full sophistication of their asset management systems can be applied to support the target setting process without extensive staff effort. While this takes some work up-front, it can result in a more informative and easier to apply method for future target setting. Takeaway 5: Target Setting Philosophy Although the focus of the pilot testing was to validate the ability to apply different technical methods, the philosophy associated with target setting – whether it should be fully data-driven or more policy-driven and aspirational – continued to emerge as an issue, particularly for the safety measures. One finding from discussions with transportation agency practitioners is that it is possible to find success from targets regardless of the particular method used to set them. The key element may be whether the target setting process and targets selected lead to engagement across the agency and with external stakeholders on the performance outcomes and underlying causal issues. Proponents of data-driven target setting have found that it is the tension around “unacceptably high” targets for fatalities and serious injuries at the time of target establishment that have led to productive discussions with leadership and elected officials around performance results and how to improve them. Meanwhile, those who stand by aspirational target setting insist that they are able to have frank discussions with leaders and stakeholders when those aspirational targets are missed, and people want to understand why. In both cases, practitioners who find value in targets are leveraging opportunities to have “hard conversations” about outcomes that can lead to new funding, new studies, or new approaches to close the performance gap. Takeaway 6: Success Factors It is difficult to say what constitutes “success” in a pilot effort like this, but participants may view target setting success in relation to a number of factors: Accuracy of forecast, enhanced understanding of

71 underlying causes of performance results, ease of implementation, or helpfulness in engaging with stakeholders, for instance. The methods tested differed in relation to these attributes (although the accuracy of future forecasts cannot yet be discerned), and as noted above, many indicated that ease of implementation is a valued attribute. Takeaway 7. Benefit of Technical Assistance While the pilot was a short-term effort and not directly tied into the required target setting, agencies indicated appreciation for exploring new ways of considering setting targets. They seemed to find that having a menu of a few different approaches to select from and test was valuable. Additional Research to Support Development of the Guide Following the completion of the pilot testing, the research team conducted a limited amount of additional research to help support development of the resulting Guide. This research included a review of the 2020 Mid-Period Performance Reports, using information provided in the FHWA TPM Dashboard; gathering additional information through a review of literature; and organizing a focus group of representatives of state DOTs to gather additional insights on how to make the target setting process more meaningful. Review of the 2020 Mid-Period Performance Reports The review of the 2020 Mid-Period Performance Reports showed that while some states made adjustments to their targets, none of the states reported changing their target setting methods at the midpoint. It is important to note that in many cases, it was difficult to discern from the FHWA TPM Dashboard whether different approaches were tried or utilized. The most common reasons for adjusting targets were incorporation of new data or having met the 4-year target at the midpoint. Review of Additional Literature Through a review of information presented at the Transportation Research Board Annual Meeting in January 2022, the research team identified an additional method for inclusion in the Guide. The method involves using machine learning to take large amounts of segment data to predict the share of roads that will be reliable (whether the segment would meet the reliability threshold); this method is documented in Babiceanu, Simona and Sanhita Lahiri (2022), “Methodology for Predicting MAP-21 Interstate Travel Time Reliability Measure Target in Virginia,” Transportation Research Record, Volume 2676, Issue 8. The Virginia DOT applied the method using Virginia-specific data for a set of independent variables including: roadway geometry information (Number of Lanes, Terrain), urban category (Urban, Rural, Urban Cluster), traffic information (Hourly Volume, Truck Percentage, and Volume/Capacity Ratio), crash information (Equivalent Property Damage Only Rate, Lane Impacting Incident Rate), and operations strategies (Presence of Safety Service Patrol) to predict if a reporting segment is reliable. VDOT subsequently added information on weather, impact of proposed future projects, and improved treatment of incidents to crashes. This information is then used to estimate predicted “Percent of the person-miles traveled on the Interstate that are Reliable.” Focus Group to Gather Additional Insights on Making the Target Setting Process More Meaningful While the pilots generated many interesting findings, the project team felt that the Guide would benefit from more details about how to generate meaningful engagement with the targets and link them to

72 subsequent actions. Therefore, the team set up a virtual focus group discussion with six practitioners to hear what they have found successful to make their targets a meaningful part of the performance management process. Focus group participants are listed in Table 29. Table 29. Participants in Discussion on Making the Target Process Meaningful Agency PM Focus Connecticut DOT All Michigan DOT PM1 Minnesota DOT PM2 Nebraska DOT PM1 Nebraska DOT PM2 Virginia DOT PM3 Core questions to start the focus group discussion included: • What advice would you give to an agency unsure about how to take action to achieve targets? • What aspect of the target setting process have you found to be instrumental in motivating effective action from your agency and partners? • Has there been any difference in how targets are received (by leadership, boards, or partners) depending on the method used to establish the target? • Have you found any value in using a simpler target setting method compared to one that is very complex? • Have you found that the coordination you undertook in the target setting process impacted how that target and necessary actions to achieve the target were received by key stakeholders after the targets were set? • Was there something specific that happened after targets were set that resulted in meaningful actions and/ or performance improvements? The most significant takeaways from the discussion were integrated into the Guide. Data-driven targets tend to push agencies to have conversations about what the targets mean and what should be done to achieve improved outcomes earlier than aspirational targets. Data-driven targets are estimates – if an agency uses a data-driven approach and estimates an outcome, that estimate can be used as the basis for what is and is not within an agency’s control to move the needle toward a more desired outcome. These conversations can still happen with aspirational targets; however, they are more likely to happen after the aspirational target is not met versus when the target is set. A challenge for bridge and pavement and reliability targets is the timeframe being too short compared to when investment decisions are made. The programmed investments that would make a difference in meeting these targets are the result of decisions that are made outside of the target setting timeframe. Additionally, large construction projects seem to impact reliability performance more than overall investments in projects that would improve reliability. The agencies noted the midpoint discussions were more substantive than those during the initial target setting. The comparison of target and actual at the midpoint was helpful for agencies to determine whether they were on track to achieve the target and what the limitations were in terms of data and methods.