Guide to Effective Methods for Setting Transportation Performance Targets (2022)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

78 Measures The third FHWA performance measure rule established six measures for assessing the per- formance of the NHS, freight movement on the Interstate system, and CMAQ. This section of the guide gives an overview of the travel time reliability and freight reliability measures. Targets for performance measures for travel time reliability and freight reliability are developed on a statewide basis. MPOs within the state have an option of either supporting the state targets or establishing their own targets. The performance measures are as follows: • Travel time reliability on the NHS: – Percentage of person miles on the Interstate system that are reliable – Percentage of person miles on the non-Interstate NHS that are reliable • Freight reliability on the Interstate: – TTTR index For the travel time reliability measures, the assessment of reliable travel is based on a metric of LOTTR, which is calculated as the 80th percentile travel time divided by the 50th percentile travel time. It is assessed for each roadway segment for each of four time periods: 6–10 a.m. weekdays, 10 a.m. to 4 p.m. weekdays, 4–8 p.m. weekdays, and 6 a.m. to 8 p.m. weekends. A segment’s LOTTR must be <1.50 for all four time periods in order for the segment to be considered reliable. For the freight reliability measure, the assessment is based on a metric of TTTR, which is calculated as the 95th percentile truck travel time divided by the 50th percentile truck travel time. It is assessed for each roadway segment for each of five time periods: the four time periods used in determining travel time reliability plus the overnight period of 8 p.m. to 6 a.m. The travel time reliability index (TTRI) is computed as follows: the length of each segment is multiplied by its maximum TTTR of the five time periods, and the sum of the length-weighted TTTRs is then divided by the sum of all segment lengths. TTTR index all segment lengths all segments length weighted TTTRs-\ = / / A higher TTRI signifies lower reliability. Challenges Associated with Setting and Revising Targets Data Limitations and Quality Data limitations are the biggest challenge and were cited by every agency interviewed. The NPMRDS has few years of available data from which to conduct statistical analyses. As more data Target-Setting Methods for Reliability

Target-Setting Methods for Reliability 79 become available in the future, this challenge should begin to diminish. For now, it limits states’ ability to conduct robust analyses. During the pilot testing phase of NCHRP 23-07, the analyses revealed concerns regarding data quality, notably in relation to the data from the NPMRDS used for the system performance and freight movement measures. State DOTs identified questionable data from the NPMRDS for some roadways or some periods of the day, which created challenges for the analysis. Finally, many states cited uncertainties related to COVID-19 as a big limitation in setting targets in 2020 and beyond and in terms of making adjustments to targets. Meaningfulness of the Target Many rural states (e.g., Iowa and New Mexico) already have reliability levels of nearly 100%, which makes a statewide target for reliability of limited value and meaning. DOTs also men- tioned that statewide measures of reliability are sometimes too broad to be useful; reliability and related issues often vary broadly for urban and rural areas. It is often more productive to measure reliability on the corridor level. Understanding the Relationship of Planned Actions to Targets Some states noted a limited understanding of the relationship between concrete actions (e.g., building a certain project) and the impacts of those actions on the targets. The relationship could be even more complex for multimodal projects, in which the impact on the reliability target may not be linear. Over time, with more data, this understanding should improve. Summary of Target-Setting Methods for Reliability States use a variety of methods to set reliability and freight targets. While the development of performance measures for travel time and freight reliability relies on segment-level speed data, some of the methods for target setting use aggregate data on performance levels for the system of interest (i.e., Interstate, non-Interstate NHS) to forecast future performance and set targets. Other methods use disaggregate data to assess possible changes in performance at the road segment level, which are then rolled up to calculate an overall performance target level. Data for the measures were provided by FHWA through the NPMRDS, which is a national data set of average travel times on the NHS. Many states rely upon NPMRDS Analytics, a web- based tool provided by Regional Integrated Transportation Information System (RITIS) that enables a simple, easy-to-use analysis of NPMRDS data. Table 11 highlights six primary methods and summarizes strengths, limitations, and other considerations related to each. Group discussion and engagement with stakeholders is also an approach that some states use to set targets. This approach can rely upon data and analysis using any of the approaches listed above; it often builds on simple methods such as adjusting the baseline or analyzing trends and may use collaborative tools in the process. The California DOT (Caltrans), for instance, coordinated with California MPOs via in-person or webcast workshops and other key stakeholder meetings, which involved use of an interactive polling tool called “Poll Everywhere.” Workshop participants were given draft baseline numbers for the perfor- mance measures and then asked to vote on setting targets above the existing baseline, at the existing baseline, or below the existing baseline. Information from these workshops and meet- ings was used to collaboratively establish targets for performance measures (FHWA 2020). Texas DOT staff also reported close coordination with MPOs. A big part of the Texas DOT’s analysis relied on the state’s large MPOs, as those MPOs account for 60% of the state’s total VMT.

80 Guide to Effective Methods for Setting Transportation Performance Targets Method Strengths Limitations Other Considerations Building off the baseline with assumptions Maintaining the baseline level as the target or making an adjustment based on judgement Simple, easy to communicate, and often brings in stakeholders No rigorous analytical methods are used for the adjustments Method for agencies with limited data; agency will need to decide which exogenous factors are relevant Time-series trend analysis Forecast based simply on historical performance trend Simple while still being data driven No insights into causes of outcomes; misses incorporation of new factors that may influence targets in the future May result in a worsening target, which can pose communication challenges; may be useful for agencies with limited data on exogenous factors Trend plus other factors Expands upon trend analysis to account for other factors that may shift future performance Data driven; allows for consideration of additional factors There may still be no rigorous methods for the adjustments—sometimes adjustments may not be data driven May result in a worsening target; agency will need to decide which exogenous factors are relevant Performance risk analysis Uses monthly performance data to calculate a standard deviation and then uses the deviation to assess confidence level to set target Data driven; allows for deeper scrutiny of the observed variation in the past performance; helps to make an informed decision on the possible future range for the target Data may be limited for robust analysis; no insight into causes of outcomes, unless paired with other method; misses incorporation of new factors that may influence targets in the future Using target ranges often seems to lean toward selecting conservative targets for which there is a high likelihood of meeting the target Segment risk analysis Focuses on segment- level data to assess segments that are at risk of shifting across the threshold of a reliable segment Introduces secondary analysis onto the reliability calculation; more customized approach Requires additional, somewhat complex analysis of individual segments May result in a worsening target Multivariable statistical model Regression analysis or tool developed to account for various factors to predict performance; typically applied at the segment level Fuller understanding of causes of outcomes, fully data-driven, and may support linking the target- setting process with decision-making by informing what factors can be influenced Complex, requiring analytical and data skills; harder to communicate the method and nuance to stakeholders; may result in a worsening target A sophisticated model will require significant data gathering and in- depth knowledge of application of statistical models Table 11. Target-setting methods for travel time reliability and freight reliability.

81   What It Is Building off the baseline with assumptions refers to a pivot off the baseline value with some assumptions, either to maintain the baseline level or to adjust on the basis of a consideration of factors that might affect future performance. This is a more qualitative approach to target setting, often selected in recognition of the limited data available for freight and reliability measures. This method may be applied for its simplicity and ease of application, especially when the agency does not have access to historical data or data on external influencing factors. This approach is also useful when it is expected that only a slight variation from the baseline value would be realized by modest improvements to the Interstate system or limited changes in expected vehicle travel. When to Use It This method can work well for agencies with limited data for trend analysis and limited data on external influencing factors. Agencies will need to decide which exogenous factors are relevant. What Is Needed Travel data (e.g., NPMRDS data) are needed to establish the baseline. To determine whether the baseline should be adjusted, data on things such as planned construction projects, nearly completed improvements, socioeconomic trends, and other related data can be used. No special tools are needed for application of this method, and simple spreadsheet tools can be utilized for setting the targets. How to Do It Step 1: Establish the Baseline Establish the baseline (e.g., using NPMRDS data), as required for performance reporting. Step 2: Adjust the Baseline Determine whether to adjust the baseline up or down to set a target based on an assessment of factors that might affect performance according to professional judgement. F A C T S H E E T Reliability Method 1: Building off the Baseline with Assumptions AT A GLANCE Ease of application: Technical robustness: Ease of communication: Allows for policy preference:

82 Guide to Effective Methods for Setting Transportation Performance Targets Advantages This method is simple, easy to communicate, and often encourages coordination within agencies. Limitations No rigorous analytical methods are used for the adjustments when setting the targets. Examples The Hawaii DOT intends to set targets around the baseline with some improvements on the Interstate, recognizing that it is not pursuing major congestion projects but implementing lower-cost types of projects (FHWA 2020). The Michigan DOT established conservative targets due to gaps in probe data and limitations in months of data to evaluate. For instance, while its baseline for reliable person miles traveled on Interstates was 85.2% in 2017, it set a target of 75.0% for 2019 and 2021 (FHWA 2020).

83   What It Is Time series trend analysis is a simple method that can be used to forecast performance measures for travel time reliability and freight reliability. State DOTs can choose to use a linear forecasting function or one of the other functions available in their analysis software (e.g., exponential function) to forecast the target year’s performance. This method is recommended when the agency has access to historical data on the performance measure but not on external influencing factors. When this method is employed, care should be taken to consider significant changes in trend or inconsistencies in the data sources during the analysis period. For instance, recognizing the dramatic change in travel that occurred due to the COVID-19 pandemic, agencies could break the data into pre-COVID, COVID, and post-COVID trends to better understand the trend to support forecasting for targets. Some additional work is required to explore trends and make judgements about which trend is likely to occur in the future. A challenge with this approach is dis- continuity of the data between the NPMRDS V.1 and NPMRDS V.2 data sets. When to Use It This method may be useful for agencies that have historical data available on the perfor- mance metric but limited data about exogenous factors. What Is Needed As with all reliability methods, travel data (e.g., NPMRDS data) are needed to establish the trend line; a spreadsheet application or statistical software package is needed for the analysis. How to Do It Step 1: Collect Historical Data for Selected Years Historical data on monthly or annual reliability or both should be collected with the RITIS NPMRDS analytics tool. This method can be applied on either monthly or annual data, depend- ing on the availability of the data. Step 2: Establish Trends and Projection After the historical data are collected and plotted, a trend line needs to be established. This can be done with a spreadsheet tool (e.g., Excel) or another statistical analysis tool. Agencies can use F A C T S H E E T Reliability Method 2: Time-Series Trend Analysis AT A GLANCE Ease of application: Technical robustness: Ease of communication: Allows for policy preference:

84 Guide to Effective Methods for Setting Transportation Performance Targets a linear function or one of the other functions (e.g., exponential, logarithmic) available in the analysis software to generate the trend line. The projections can be developed by using either simple linear forecasting or nonlinear methods such as exponential smoothing or ARIMA. Following are some of the commonly used forecasting techniques: • Linear trends, which are a continuation of the line of best fit for historical data and are the most common approach for establishing a projection. Forecasts can be established through the FORECAST function in Excel (FORECAST.LINEAR in newer versions) or similar functions in other software. • Exponential smoothing, which weights recent observations more heavily than older ones in estimating future points. These models are particularly useful in handling seasonality in trends. They likely are not helpful for the limited years of data currently available, but they may be a good option once more data become available. Excel has a function for exponential triple smoothing (FORECAST.ETS) that can be applied to historical data. • ARIMA, which uses correlations in past observations as the basis for future trends. These models are likely not helpful for the limited years of data currently available, but they may be a good option once more data become available. Advantages This method is easy to apply while still being data driven. Limitations This method does not provide insights into the causes of outcomes and may result in a worsening target, which can pose communication challenges. If the historical data show dif- ferent patterns of trends, the historical data needs to be studied carefully. For example, agencies can split historical reliability data into pre-COVID, COVID, and post-COVID periods. Judge- ment is needed to determine which trends are likely to occur in the future. Examples The Connecticut DOT used 3 years of NPMRDS V.1 data to forecast trends through time but applied the trend to the NPMRDS V.2 data as the baseline condition due to discontinuities between the two data sets for the reliability measures (FHWA 2020). Tennessee DOT used the INRIX tool and NPMRDS data to calculate reliability measures for the years 2014 to 2017. A trend line was developed to forecast the 2019 and 2021 values (FHWA 2020). As part of the pilot testing of methods for the development of this guide, the Oklahoma DOT analyzed the monthly trends from 2016 to 2021 for the reliability and freight perfor- mance measures. The agency observed a drastic change in the regime for the monthly Interstate reliability measure once lockdowns due to COVID-19 started in March 2020. The Oklahoma DOT split the monthly Interstate reliability figures into three regimes: pre-COVID (January 2016 to February 2020), COVID (March 2020 to February 2021), and post-COVID (March 2021 to June 2021), as shown in Figure 16. The agency believed that traffic patterns observed from March 2020 to February 2021 would not be representative of what would happen in the future, and, in fact, the values for Interstate reliability after March 2021 were observed to return to the

Target-Setting Methods for Reliability 85 pre-COVID regime. The Oklahoma DOT indicated in the pilot test that the agency planned to exclude Interstate reliability data from March 2020 to February 2021 when it developed targets by using linear forecasting of observed monthly Interstate reliability values. The Oklahoma DOT indicated in the pilot test that it planned to develop targets by using linear forecasting of observed monthly Interstate reliability values that exclude the Interstate reliability data from March 2020 to February 2021. 85 87 89 91 93 95 97 99 Ja n- 16 M ay -1 6 Se p- 16 Ja n- 17 M ay -1 7 Se p- 17 Ja n- 18 M ay -1 8 Se p- 18 Ja n- 19 M ay -1 9 Se p- 19 Ja n- 20 M ay -2 0 Se p- 20 Ja n- 21 M ay -2 1 Se p- 21 Ja n- 22 M ay -2 2 Se p- 22 Ja n- 23 M ay -2 3 Se p- 23 Ja n- 24 In te rs ta te R el ia bi lit y M ea su re % reliable- pre-COVID % reliable- COVID % reliable- post COVID Linear (% reliable- pre-COVID) Linear (% reliable- COVID) Figure 16. Oklahoma DOT time series trend analysis for monthly Interstate reliability.

86 What It Is The trend plus other factors method expands on a trend analysis by including additional data, factors, or other inputs that are expected to influence performance beyond trend line performance levels. This can include relevant external factors such as economic indicators, travel projections, and population growth as well as factors such as anticipated con- struction impacts or network improvements that are closely tied to travel time reliability. The trends observed in the forecasted additional factors can guide the adjustments of the targets of the performance measures. This method is recommended when the agency has access to historical data on the performance measure and external influ- encing factors. The analysis might also include more qualitative stakeholder input or policy judgements (such as decisions to adjust the targets to be more conservative). When to Use It This method is recommended when the agency has access to historical data on the performance measure and external influencing factors. What Is Needed As with all reliability methods, travel data (e.g., NPMRDS data) are needed to establish the trend line. Data on economic indicators, travel projections, population growth, construction projects, network improvements, and other factors that may affect performance are also needed. A spreadsheet application or statistical software package is necessary for the analysis. How to Do It Step 1: Collect Historical Data for Selected Years Historical data on monthly or annual reliability or both should be collected with the RITIS NPMRDS analytics tool. This method can be applied on either monthly or annual data, depending on the availability of data. Step 2: Establish Trends and Projection After the historical data are collected and plotted, a trend line needs to be established. The trend line can be established with a statistical analysis tool. One has an option of generating F A C T S H E E T Reliability Method 3: Trend Plus Other Factors AT A GLANCE Ease of application: Technical robustness: Ease of communication: Allows for policy preference:

Target-Setting Methods for Reliability 87 a linear function or one of the other functions (e.g., exponential, logarithmic) available in the analysis software. Step 3: Identify and Analyze Factors In this step, the agency examines factors that may have an impact on travel time reliability or on freight reliability, in particular. These factors may include economic indicators, travel fore- casts, anticipated effects of infrastructure construction projects (which may reduce reliability by leading to work zone delays or other delays during construction), and improvements to the transportation network (which may improve reliability when in operation). Step 4: Adjust Forecasted Trend The trends observed in the forecasted additional factors can guide the adjustments of the targets of the performance measures. This adjustment is typically developed on the basis of judgment of what is reasonable to expect. This step often involves soliciting input from partners to agree upon a reasonable adjustment to the trend line figures. Advantages This method is easy to apply and is data driven. It allows for consideration of exogenous factors that can influence the performance measures. Qualitative inputs can also be taken into consideration when the targets are being adjusted. Limitations The adjustments are based on judgment and may not be completely data driven. There may still be no rigorous methods for adjusting the targets. This method may result in worsening targets, which could be a challenge for communicating the targets. Examples The Delaware DOT noted that historical trends for Interstate travel time reliability revealed a decrease of approximately 2.0% per year. However, beyond these trends, the target-setting pro- cess also considered the extent to which major capital projects would likely influence congestion levels on the basis of when and where multiyear construction activities would occur within the next 2 to 4 years (FHWA 2020). The Indiana DOT analyzed the impacts of new projects. The agency calculated the baseline TTTR index using 2017 travel time data from the NPMRDS. On the basis of the programmed projects, the agency then calculated an exposure factor related to construction projects in miles per day. Indiana used this exposure factor to consider the potential impacts of planned construc- tion, calculating a ratio of future exposure to base year exposure and using that ratio to develop 2- and 4-year targets (FHWA 2020). The South Carolina DOT used trends and other statistical techniques that incorporated several variables. In 2018, the South Carolina DOT considered the impact of paving projects in its analysis. Staff reported that this portion of the analysis did not yield a conclusive result and, therefore, it was omitted. However, it offered an example of a factor that might be considered for inclusion, as other states might find that paving projects resulted in significant impacts on reli- ability. In 2020, for the Mid Performance Period Progress Report, the South Carolina DOT focused

88 Guide to Effective Methods for Setting Transportation Performance Targets on analyzing isolated unreliable hot spots and conducting customized analyses for different regions to account for differing factors and variables unique to specific regions. This allowed for improved, tailored consideration of unique geographic features of different regions. For example, coastal regions in South Carolina are constrained by peninsulas. As part of the pilot testing of methods for the development of this guide, the Utah DOT was able to collect data on explanatory factors that could influence the performance measure for Interstate reliability, including VMT, population, employment, and GDP, for the entire region. Unfortunately, the Utah DOT could only gather the historical data, and no future forecast data could be collected during the pilot test. Figures 17 through 20 show the Interstate reliability measures mapped against statewide VMT, GDP, and population and employment estimates. The annual Interstate reliability measures are clearly correlated with some explanatory factors. For example, when the statewide VMT drops, there is a corresponding increase (improvement) in the statewide Interstate reliability measure. On the basis of the analysis of how the explanatory factors influenced the historical performance of the Interstate reliability measure, the Utah DOT plans to adjust the targets for the future year. The Utah DOT also plans to collect forecasted data for the explanatory factors, which could help in target setting. 28000M 30000M 32000M 34000M 80 90 100 2017 2018 2019 2020 2021 2022 VM T In te rs ta te R el ia bi lit y M ea su re VMT % reliable VMT Linear (% reliable) Figure 17. Interstate reliability mapped against statewide VMT. 3M 3M 3M 3M 4M 80 90 100 2017 2018 2019 2020 2021 2022 Po pu la tio n In te rs ta te R el ia bi lit y M ea su re Population % reliable Population Linear (% reliable) Figure 18. Interstate reliability mapped against population. 140,000 160,000 180,000 200,000 80 90 100 2017 2018 2019 2020 2021 2022 G D P In te rs ta te R el ia bi lit y M ea su re GDP % reliable GDP Linear (% reliable) Figure 19. Interstate reliability mapped against GDP.

Target-Setting Methods for Reliability 89 Figure 20. Interstate reliability mapped against employment estimates. 2M 2M 2M 2M 80 90 100 2017 2018 2019 2020 2021 2022 Em pl oy m en t In te rs ta te R el ia bi lit y M ea su re Employment % reliable Employment Linear (% reliable)

90 What It Is Performance risk analysis is a data-driven approach that uses statistical analysis to explore the variation in performance levels to account for risk in setting a target. Under this approach, data on monthly performance are used to calculate a standard deviation, and then the deviation is used to assess a confidence level for likely future performance levels to account for risk. This methodology is often used for risk analysis of financial assets and stock prices. This method may be particularly helpful when the availability of annual trend data is limited available. Under this approach, data on monthly performance are used to calculate a standard deviation, and then the deviation is used to assess a confidence level for likely future performance levels, to account for risk. The agency selects an appropriate confidence level and then sets the target on the basis of this level. This method helps address the limited amount of data and uses monthly data to establish probabilistic targets. When to Use It This method is applicable when the availability of annual trend data is limited but monthly reliability data are available to perform trend analysis. What Is Needed As with all reliability methods, travel data (e.g., NPMRDS data) are needed to establish the trend line. A spreadsheet application or statistical software package is needed for the analysis. How to Do It Step 1: Calculate the Performance Measure Collect performance measure from the NPMRDS by month on the basis of the available months of data. Step 2: Select a Distribution That Fits the Data Select a distribution that fits the observed historical performance data. Different model fits to the distribution of the monthly data may need to be tested. Step 3: Calculate Standard Deviation and Desired Confidence Level Use statistical software to calculate the standard deviation of the monthly values. Select a confidence level desired to achieve the target (e.g., 75%). F A C T S H E E T Reliability Method 4: Performance Risk Analysis AT A GLANCE Ease of application: Technical robustness: Ease of communication: Allows for policy preference:

Target-Setting Methods for Reliability 91 Step 4: Identify Target Level Use the statistical software to identify the target level according to the confidence level desired. Advantages This method is data driven. It employs additional methods to strengthen the analysis. Limitations Data may be limited for robust analysis. This method does not provide insight into causes of outcomes unless paired with other methods. It misses incorporation of new factors that may influence targets in the future. Examples To determine the percentage of reliable person miles and truck travel time reliability, the Iowa DOT calculated each of these measures by month. For the initial performance targets, the agency calculated the measures for each month in 2017, and for the Mid Performance Period Progress Report, it used the same procedures but extended the data set to reflect additional available data. Data for 2017, 2018, and 2019 were used to calculate the measures for each month. The agency used the statistical software @RISK to analyze the 36 monthly observations, and various statistical distributions were fit to the data and compared. For the percentage of reliable Interstate person miles, the values cannot exceed 100%, and the monthly data were heavily skewed, which resulted in a triangular distribution as a best fit. For the percentage of reliable non-Interstate NHS person miles, although it was not an ideal fit, the software recommended using a normal distribution. For the TTTR index, the software recom- mended using a Pareto distribution, although the lognormal distribution also performed well and has the advantage of making more sense, given the nature of the TTTR index, and so was used, as shown in Figure 21. The Iowa DOT then used the theoretical distribution to obtain target values corresponding to various levels of confidence and chose to use a 75% confidence level. The agency then rounded the value down to the nearest 0.5% for the reliability measures and to the nearest 0.01% for the TTTR index (Iowa DOT 2020b). The Iowa DOT recognized that the relationship between monthly and annual values is not straightforward, as the LOTTR and TTTR are calculated with specific thresholds of reliability for the specific time frame being evaluated. This can result in annual values that are lower than the average of monthly values. Because of this issue, the Iowa DOT analyzed the annual values to see how likely they would be to come from the theoretical distribution of the monthly values. For the share of non-Interstate NHS that is reliable, the agency observed that the annual values were atypical of the theoretical distribution and therefore used the same standard deviation as the monthly data but substituted the mean of the three annual observations, resulting in a shifted distribution. The Mid Performance Period Progress Report yielded no change to the target for non-Interstate NHS reliability (originally set at 95.0%) but resulted in small revisions to the Interstate reliability and TTTR index targets. The Minnesota DOT employed a variation of the performance risk analysis method by using the monthly data for Interstate reliability, shown in Table 12, to develop a box and whisker plot in Excel. This allowed the agency to visualize the variance in the monthly data over the past years and to monitor outliers. The variance in the past monthly data will be used to inform the target setting. Figure 22 shows the box and whisker plot developed as part of the pilot stage of NCHRP 23-07 for the Minnesota DOT’s Interstate reliability performance measure.

Source: Iowa DOT (2020b). © Iowa Department of Transportation. Used with permission. Figure 21. Lognormal distribution fit to Interstate TTTR data. Monthly Interstate Reliability Measure (% reliable) Month 2017 2018 2019 2020 2021 January 82.5 83.3 90.2 89.3 99.7 February 88.5 86.6 74 89.4 99.3 March 89.4 88.8 85.7 98.6 98.2 April 81.4 82.6 80.3 100 95.4 May 77.1 77.1 79.3 99.7 93.5 June 75.3 77.3 78.3 99.5 92.9 July 81.8 79.5 80 98.7 August 79 79.7 81.6 98.8 September 75.8 79 78.5 98.3 October 81.6 83 79.9 98.8 November 83.7 82.1 83.6 99.4 December 81.9 85 81.6 99.5 Summary Statistics Minimum 75.3 77.1 74 89.3 92.9 First quartile 78.5 79.4 79.1 98.5 93.9 Median 81.7 82.3 80.1 98.8 96.8 Third quartile 82.8 83.7 82.1 99.5 99.0 Maximum value 89.4 88.8 90.2 100 99.7 Mean 81.5 82 81.1 97.5 96.5 Range 14.1 11.7 16.2 10.7 6.8 Standard deviation 4.41 3.64 4.07 3.84 2.97 Table 12. Monthly Interstate reliability values for Minnesota and computed statistics.

Target-Setting Methods for Reliability 93 Key 2017 2018 2019 2020 2021 Year Upper Outlier Lower Outlier Median Mean 3rd Quartile 1st Quartile Upper Whisker Lower Whisker Whiskers extend to the minimum and maximum data points within 1.5 times the range from the 1st Quartile to the 3rd Quartile, from the bottom and the top of the box, respectively. Figure 22. Box and whisker plot of the Minnesota DOT monthly Interstate reliability measure.

94 What It Is A segment risk analysis describes an approach to travel time reliability and freight reli- ability measures that relies directly on an analysis of individual segments to identify those that are likely to shift from reliable to unreliable or vice versa. For the travel time reliability measures, FHWA defined the threshold for a reliable seg- ment as an LOTTR of less than 1.50 for all four time periods analyzed, where LOTTR is calculated as the ratio of the 80th percentile travel time to the 50th percentile (normal) travel time. This method involves conducting an analysis of LOTTR for each segment or traffic message channel (TMC) to identify segments that are likely to influence future performance reliability by shifting above or below the 1.50 threshold. For the freight reliability measure, there is no defined threshold over which the segment would be considered unreliable. The performance metric, TTTR, is defined as the ratio of the 95th percentile truck travel time to the 50th percentile truck travel time and is calculated for five time periods for each segment. Since the performance measure is an index, and not based on a specific threshold of reliability, a segment risk analysis approach for TTTR may involve identifying segments for which reliability is likely to increase (e.g., as a result of construction) and increasing the assumed TTTR for those segments. When to Use It This method is applicable when reliability data are available on a segment level. What Is Needed A spreadsheet application or statistical software package is needed for the analysis. The following data are needed: • Performance levels for reliability at the segment level (LOTTR or TTTR or both), either retrieved from RITIS or developed from NPMRDS data; • The associated data on segment length and traffic volumes; and • Data on anticipated construction projects or other segment-level causes of potential change (depending on the analysis approach). How to Do It The process will depend on the specific approach used—whether for LOTTR or TTTR (see the examples below)—but generally will involve the following steps: F A C T S H E E T Reliability Method 5: Segment Risk Analysis AT A GLANCE Ease of application: Technical robustness: Ease of communication: Allows for policy preference:

Target-Setting Methods for Reliability 95 Step 1: Obtain Segment-Level Performance Measures Calculate or obtain the performance metrics (LOTTR or TTTR) for individual segments with the RITIS NPMRDS analytics tool (e.g., for TMCs) for the base year of analysis. Step 2: Identify Segments Likely/Possible to Change Performance A change in performance could be due to new construction projects or to the LOTTR level being very close to the 1.5 threshold for one or more periods. Step 3: Make Adjustments to Those Segments Adjust the LOTTR or TTTR of segments that are likely/possible to change. Step 4: Recalculate the Systemwide Measure Calculate the system-level performance for the measure on the basis of the revised segment- level performance. Advantages This method involves additional assessment of the performance on a segment level, which allows for more detailed analysis, which gives additional insights into the performance of the segments, corridors, and the transportation network. Limitations This method requires additional, somewhat complex analysis of individual segments, and some agencies may not feel that it is worth the level of effort. Examples The Oregon DOT tagged each TMC on the basis of the individual LOTTR calculations as “Good” (1.4 or less), “Barely Good” (1.4–1.5), “Barely Bad” (1.5–1.6), or “Bad” (1.6 and above). Given difficulties of predicting the future, a decision was made to consider a scenario of limited funding where sections rated “Barely Good” could shift to the “Barely Bad” category. The agency then calculated the revised share of person miles traveled with reliable travel times at the system level (for Interstates and non-Interstate NHS) (FHWA 2020). The Arkansas DOT identified significant construction projects on the Interstates and assigned all TMCs within the anticipated project limits an assumed TTTR of 5.0 (reflecting an assump- tion that travel time on the worst day of the week would be five times greater than travel time on average) to account for a potential decrease in reliability on those segments during construction. To account for anticipated growth in truck volumes, the agency also increased the maximum TTTR for each TMC by 5%. These values were then used to calculate the overall TTTR index to set the target (FHWA 2020). The Minnesota DOT tried developing a draft approach for target setting by using the segment risk analysis method for the pilot effort for NCHRP 23-07. For this draft approach, the agency did not take into account the 2020 LOTTR values affected by COVID. The analysis used a subsection of statewide TMCs. The segments were categorized on the basis of the value of the LOTTR, as follows: • Good (<1.4), • Barely Good (1.4–1.5),

96 Guide to Effective Methods for Setting Transportation Performance Targets • Barely Bad (1.5–1.6), and • Bad (>1.6). During the pilot effort, the Minnesota DOT used LOTTR data for 2015 to 2019 to perform a trend analysis and forecast for the year 2024. The 2024 forecasts were based on the TREND function in Excel, which uses a linear trend and the method of least squares. On the basis of the 2024 forecasted LOTTR values by segment, the performance measure was computed again for 2024. Based on the change in the LOTTR value between 2019 and 2024, the segments were categorized as follows for visualization: • Remained Reliable (2019 and 2024 LOTTR < 1.5), • Remained Unreliable (2019 and 2024 LOTTR > 1.5), • Became Reliable (2019 LOTTR > 1.5 and 2024 LOTTR < 1.5), and • Became Unreliable (2019 LOTTR < 1.5 and 2024 LOTTR > 1.5). Figure 23 shows the visualization the Minnesota DOT developed during the pilot effort. The computed 2024 performance measure will be used to inform the 2024 target. The Washington State DOT also tried the segment risk analysis method during the pilot effort for NCHRP 23-07. The agency wanted to adjust the TTRI values of the segments, which were at risk of worsening due to the presence of construction projects on the segments or corridors. The idea was to adjust the forecasted TTRI values of the segments on which construction was expected to happen in the year for which targets were to be developed. To test how con- struction projects affected the TTRI values, the Washington State DOT overlaid a layer of construction projects with a layer of TMC segments in a geographic information system (GIS) analysis software and identified segments with construction work between 2017 and 2019. Each segment was labeled as to whether there were any construction projects and, if so, in which years. Then the change in TTTR between 2017 and 2018 and between 2018 and 2019 was examined for those segments with construction work. Specifically, the Washington State DOT evaluated segments with construction in one year and those with no construction work in the other year to separate the construction effect. The absolute change in TTTR for those segments in two consecutive years (with and without construction) was negligible. Most segments did not exhibit any change, while a small fraction of segments demonstrated an increase or decrease in perfor- mance. The aggregated effect for all segments weighted by length was almost zero. The percentage change in median travel time for those segments in two consecutive years was also negligible, with most segments showing no change. The aggregated effect for all segments was examined by calculating the average travel speed for the segments and was also found negligible. The result of the analysis was counterintuitive and did not show a correlation of worsening TTRI values with the presence of construction on the segments. Because of these findings, the Washington State DOT did not pursue the application of the segment risk analysis method further.

Figure 23. Minnesota DOT’s application of target-setting segment risk analysis method.

98 What It Is A multivariable model is a formal effort to quantitatively forecast performance by using methods more sophisticated than trend analysis based on historical performance; it typically integrates various factors into a forecast. Whereas multivariable models for safety measures are often developed at an aggregate level to calculate total fatalities or serious injuries, multivariable models in the context of reliability and freight measures are often developed to support the calculation of metrics that are applied at the segment level and then used to calculate the systemwide measure. Many agencies experiment with several types of multivariable models or test different variables in the models, analyze results, and select the one that seems to best reflect real-world conditions. At least one agency has also applied a machine learning model. When to Use It The method is technically robust; however, use of sophisticated statistical models requires significant data gathering and in-depth understanding of the application of complex statistical models. This method should be considered when the agency has good prior experience with the application of statistical models and has access to experts who can build, validate, and interpret these sophisticated models. What Is Needed To use this method, agencies will need • Performance levels for reliability at the segment level (LOTTR or TTTR or both) gathered from the NPMRDS, • Associated data on segment length and traffic volumes, • Data on factors that may influence reliability performance (e.g., roadway attributes), • A statistical software package, and • Understanding of the statistical implications of model choices or access to reference material or expertise. How to Do It Step 1: Explore Exogenous Factors This can be conducted through a literature review, exploratory analysis of data, or simply discussion among experts on factors likely to have an impact on reliability. Because complex F A C T S H E E T Reliability Method 6: Multivariable Statistical Model AT A GLANCE Ease of application: Technical robustness: Ease of communication: Allows for policy preference:

Target-Setting Methods for Reliability 99 models will be tweaked with new findings or data, these factors can be removed if there is no statistically significant relationship or high correlation with other factors. Step 2: Collect Data Collecting data can be the most time-consuming part of developing a new model, especially one that is very detailed and that requires disaggregated data. The number of years of data available for each variable can limit the number of years available to use in the model, unless alternative values can be imputed for missing points. Step 3: Select Model Form and Details This step involves selecting the final explanatory factors from the previous steps and doing statistical research to ensure the explanatory variables are accounted for appropriately and do not pose statistical problems such as multicollinearity. Correlation and factor analysis are two examples of ways to assess the different factors and inform the final selection. An agency may also build two or more models and make a final selection on the basis of the results. For example, a model can be binomial, polynomial, or log-based and can be built using raw data, data on the change between periods, data with a lag applied, or other form of data. Often, agencies build several models and select the one that provides the most accurate estimates and greatest explanatory power. Step 4: Validate Model A model can be validated by using it to predict the value of a point for which observed data already exist. A model’s ability to accurately predict the known value is a positive indication of its ability to predict future values. Step 5: Apply the Model Once the statistical model is developed, it can be used to • Predict current performance, • Predict future performance, • Update the current observed metric value for each segment by the predicted difference, and • Aggregate to develop the overall performance measure value. Advantages This method helps agencies have a fuller understanding of the causes of outcomes. It is fully data driven and may support linking the target-setting process with decision-making by informing what factors can be influenced. Limitations This method is complex and requires analytical and data skills. The complexity makes it harder to communicate the method and nuance to stakeholders. It may result in a worsening target. Examples The Alabama DOT used a consultant to develop a model to support analysis of reliability and freight measures. The team first collected data on current capacity and volumes on road- way segments and reliability metrics (LOTTR and TTTR). It then developed a statistical model

100 Guide to Effective Methods for Setting Transportation Performance Targets to associate segment-level LOTTR/TTTR with volume, capacity, and roadway attributes. Next, it created a forecast of future volume on the basis of current growth rates and updated the levels of future capacity for each segment on the basis of planned projects. Recognizing that models are not perfect, the team used the model to predict current performance and then future performance on the basis of estimates of future volumes and capacities. The predicted difference was used to update the current observed metric value for each segment. These data were then used to calculate the updated performance measure to use as a target (FHWA 2020). The Maryland DOT used a similar approach to develop statistical models that relate system reliability metrics (LOTTR and TTTR) to roadway volume/capacity ratios at the segment level. To develop the models, the agency retrieved segment-level LOTTR and TTTR scores from RITIS and calculated peak hour average volumes from the state’s automatic traffic count stations. The models used a logarithm format and estimated reliability metrics on the basis of roadway volume, capacity, and roadway characteristics, including urban/rural designation and functional classification. Although the models for LOTTR and TTTR had limited overall explanatory power (about 25% of the total variation in segment-level performance was explained by the models), they had highly significant coefficient estimates in relation to volumes and capacities that are used for forecasting. Estimates of future traffic volumes were based on growth rates calculated per county and per functional class from HPMS submissions, and traffic volume growth rates were damped for segments with high volume/capacity ratios. Future capacity values were also updated by identifying capacity-enhancing projects, conflating project boundaries to the TMC segments, and then adding lane capacity within the project boundaries after the project com- pletion date. The model was applied to predict current performance and then to predict future performance on the basis of the estimated future volumes and capacities, and the predicted difference was used to update the current observed metric value for each segment. These data were then used to calculate the updated performance measure to use as a target (Maryland DOT 2018). The New Mexico DOT used a consultant to develop log-linear regression models that asso- ciated LOTTR and TTTR for each segment with volume, capacity, and roadway attributes. It then updated forecasted future volumes on the basis of estimated growth rates (from the HPMS and forecasting models), updated future capacity on the basis of planned projects, and used the models to forecast future LOTTR and TTTR with the updated volumes and capacities. The Virginia DOT developed a unique methodology for forecasting reliability targets as part of a research study (Babiceanu and Lahiri 2022). The method uses machine learning to take large amounts of segment data to predict the share of roads that will be reliable (only whether the segment would meet the reliability threshold). The agency used Virginia-specific data for a set of independent variables including roadway geometry (number of lanes, terrain), urban category (urban, rural, urban cluster), traffic information (hourly volume, truck percentage, and volume/ capacity ratio), crash information (equivalent property damage only rate, lane impacting inci- dent rate), and operations strategies (presence of safety service patrol) to predict whether a MAP-21 reporting segment is reliable (Virginia Transportation Research Council 2021). The Virginia DOT subsequently added information on weather and the anticipated impact of pro- posed future projects, incidents, and crashes. This information is then used to estimate the pre- dicted “percent of the person miles traveled on the Interstate that are reliable” (PMTR-IS) with the MAP-21 specified formula. Classification and regression tree (CART) models were used with 1,536 different configura- tions (Babiceanu and Lahiri 2022). The method yielded four models that were close in their configuration and prediction accuracy. Data sets from years 2017 and 2018 were used for training and data sets from years 2019 and 2020 for testing. The predicted and calculated PMTR-IS values were compared, and the error percentage was within 1% for all models for 2019, which can be

Target-Setting Methods for Reliability 101 considered negligible. The 2020 error rate was higher and can perhaps be attributed to unusual reliability because of the impact of the pandemic on travel. Sensitivity analyses revealed that the predicted PMTR-IS values were reactive to capacity increases in unreliable sections at a local level, slower to respond to local increases in annual average daily traffic (AADT), and stable to small statewide AADT oscillations. The Virginia DOT staff believe this methodology is valuable for target setting, since it could be used to assess a project’s statewide impact on reliability, in combination with information on policies and other local influences not captured in the model.