Guide to Effective Methods for Setting Transportation Performance Targets (2022)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

6 Overview of Target-Setting Methods While each of the national performance measures has unique attributes with regard to data sources, calculation methods, availability of tools, policy considerations, and other factors that influence how to set a target, there are some common types of target-setting approaches across the measures. In general, quantitative methods range from very simple to quite complex and generally fall into the following categories: • Policy-based approaches (e.g., annual decrease of 3%), • Historical trends (e.g., based on trend over the past 5 years), • Probabilistic and risk-based approaches (e.g., considering potential variability in performance), • Statistical models (e.g., regression model), and • Other tools and models [e.g., pavement management systems (PMSs)]. These general approaches are briefly described below, and Part II of this guide provides more detailed information on specific methods, steps associated with the methods, and examples as they apply to each performance measure area. It is also valuable to note that some state DOTs and MPOs use multiple approaches to forecast anticipated performance and combine results from multiple approaches to select a target. Policy-Based Approaches: Aligning Targets with Agency Vision For the first round of federally required reporting, some states chose to select a target with relatively limited analysis of trends and factors that influence performance, relying primarily on baseline data and policy-driven decisions on setting a target off of that baseline. This was most common for safety measures, for which some states set a target in the form of a regular annual decrease (e.g., 2% reduction per year), often based on a long-term goal that is annualized. For instance, California set 2019 performance targets of an annual decrease in fatalities of 3.0%, consistent with its Strategic Highway Safety Plan (SHSP) goal of reducing fatalities by 3% annually (FHWA 2020). In the case of infrastructure condition measures, during the initial performance period, some states selected targets that matched their baseline conditions or set conservative targets (at levels below current or forecasted conditions). Some states even set targets at or near minimum condi- tion levels [established under 23 CFR 490.315 for Interstate pavements and 23 CFR 490.411 for National Highway System (NHS) bridges] to ensure that the targets would be met. In the case of travel time reliability and freight reliability measures, several states set targets at their baseline values, reflecting limited historical data with which to assess trends and uncertainties in future directions. This approach was consistent with FHWA guidance, which noted that, due to differ- ences between the Version 1 and Version 2 data sets of the National Performance Management Before We Begin: Target-Setting Foundations

Before We Begin: Target-Setting Foundations 7 Research Data Set (NPMRDS), setting achievable targets based on baseline (2017) figures would be a prudent approach. Historical Trends: A Simple Data-Driven Approach Transportation agencies may conduct analysis of historical performance trends, to the extent data are available, and use the historical trend line to forecast future performance as a basis for setting a target. During the first round of federally required target setting, this was a common approach for safety measures, non-single-occupancy vehicle (non-SOV) mode share, and annual hours of PHED per capita, in particular. Trend line analysis is generally clear and straightforward, although agencies may use statistical analysis tools to explore different ways to fit a trend beyond a simple straight trend line analysis. The sophistication of the statistical analysis depends in part on the number of years of available data on trends. Agencies also may calculate the anticipated trend value and then make adjustments to reflect other factors, such as economic indicators, travel pro- jections, or transportation projects or program efforts that might influence performance. These adjustments are generally based on judgment of what is reasonable and, in some cases, account for policy considerations. For example, in Minnesota, during the initial target setting, the number of fatalities was calculated by using historical data and projecting forward, but slight adjustments were made on the basis of local knowledge gathered from stakeholders to establish targets. The dramatic changes in travel patterns that occurred following the COVID-19 pandemic demonstrated that historical trends do not always hold, and transportation agencies need to consider how to address anomalies when they are conducting a trend analysis (this is discussed further in the section on handling performance disruptions in the chapter “Practical Application”). Probabilistic and Risk-Based Approaches: Considering Possible Ranges A probabilistic or risk-based approach involves using statistical analysis to explore likely varia- tions in performance levels and develop an estimated range of potential future performance to inform setting a target. Confidence intervals are used to develop probabilistic target ranges based on the observed variance in the performance levels in the past. The upper and lower bounds of the target ranges are then considered in developing the targets. A related approach that some states have utilized for travel time reliability and freight reliability measures is to consider the risk that a roadway segment will become either less or more reliable. Under this approach, segments that are likely or at risk of shifting from reliable to unreliable (or vice versa) are identi- fied on the basis of the current level of travel time reliability and information on factors that may shift reliability, such as construction projects. Multivariable Statistical Models: Quantitatively Integrating Influencing Factors A multivariable statistical model involves use of statistical techniques that are able to account for explanatory factors that influence performance in order to calculate anticipated performance levels. These approaches go beyond trend line analysis of performance data and integrate different factors that may influence performance within the forecasting itself, rather than treat those factors as a “back-of-the-envelope” consideration used to adjust the trend analysis result. These approaches typically involve collecting data on different factors that influence perfor- mance to develop a regression equation that best fits the data and functions as a forecasting model. While the ability to include explanatory factors is an important component of these methods, statistical models can also include methods such as univariate time series models that do not incorporate additional variables. This approach has been used by some states to support development of safety, reliability, and freight targets. For instance, for safety targets, the Virginia

8 Guide to Effective Methods for Setting Transportation Performance Targets DOT used monthly data at its district level to develop a model that accounts for the impact of 14 different factors on safety outcomes. For reliability and freight targets, often this type of model development focuses on data at the segment level. For instance, the New Mexico DOT developed log-linear regression models that associated LOTTR and truck travel time reliability (TTTR) for each segment with volume, capacity, and roadway attributes. It then updated forecast future volumes on the basis of estimated growth rates [from the Highway Performance Monitoring System (HPMS)], updated future capacity on the basis of planned projects, and used the models to forecast future LOTTR and TTTR by using the updated volumes and capacities to develop updated segment-level figures for the performance measure calculation (FHWA 2020). Other Tools and Models Finally, other models and tools are used to support target setting, notably for infrastructure and congestion measures. States, for instance, commonly use PMSs that encompass detailed data on pavement conditions along with algorithms that account for deterioration rates and performance prediction, along with life-cycle planning analysis. For the congestion measures (non-SOV mode share and PHED), some regions apply regional travel demand models to forecast mode shares and congestion levels and apply these forecasts to calculate targets. Target-Setting Philosophies While data availability, data quality, access to resources, and other tools are often key factors in selecting a specific method for establishing targets, what appears to most influence agencies’ approach to target setting is the philosophy of staff and leaders on the purpose of the target. On this point, there are three primary schools of thought: 1. Realistic/predictive target setting: Targets should reflect what is attainable in the stated time frame and be established in relation to the performance level that data indicate is most likely to occur. 2. Aspirational target setting: Targets should reflect an agency’s performance aspirations and commitment to seeing improved outcomes. 3. Conservative target setting: Targets should be conservative, to essentially ensure the agency can attain the target. Realistic/Predictive Target Setting The first view is that targets should be realistic for the agency to achieve, even if that target shows worsening conditions. Agencies that take this approach often attempt to predict the anticipated level of performance and set the target at that level. They may build complex models to try to understand the factors that impact performance—both internal agency actions and external factors (e.g., travel demand, population)—and where those factors are likely to be heading in the future. This approach can help to connect the target-setting process with analysis that informs decision-making, and, according to some practitioners, this connection supports a more honest conversation about the reasons an agency met or did not meet its targets. Since realistic targets may in some cases reflect the reality of worsening conditions, agencies must have difficult conversations between their leadership and stakeholders about why the agency’s target—seen by many as similar to a goal or something desired—may not be reached. Aspirational Target Setting Another primary view is that targets are primarily a communication and motivation tool and should reflect a desired direction for performance, even if that is somewhat unlikely to be

Before We Begin: Target-Setting Foundations 9 achieved. The most important elements are simplicity and reflecting desired policy outcomes, with the direction of the target more important than the ability to attain the exact number set. Under this approach, targets should be something that stakeholders and partners can rally around to make improvements, even if the target is more optimistic than current expectations suggest. Agencies that subscribe to this philosophy often conduct data analysis on perfor- mance but do not hold their target to the results of the analysis. Staff at these agencies do not necessarily expect to meet these aspirational performance targets, but they do expect to have discussions with stakeholders about performance relative to targets. This approach avoids the difficult conversation associated with selecting a worsening performance target, but it forces discussion on why targets are not met. It may help to support further discussions about agency actions—including the planning and policy directions adopted by the agency—necessary to achieve desired targets. Conservative Target Setting A third approach is to set targets that are very likely to be achieved by either selecting a target that is at a minimum standard or at a level that has been consistently achieved in the past. Under this philosophy, the most important aspect is meeting the target. Generally, states that have used this approach expressed concern about limited data and experience to be able to realisti- cally predict performance but expected to set more realistic/predictive targets in the future. The philosophy of an agency about target setting can influence the method that it wishes to choose for target setting, specifically, whether to use a more complex method that involves developing statistical models or one that uses other tools and models. What Makes a Target-Setting Method Effective? Each philosophy outlined above has an implicit assumption of what “effective” means. A proponent of realistic target setting might say that a target-setting method is effective if it can accurately forecast anticipated future performance. Adherents to more aspirational targets might argue that an effective target conveys an agency’s commitment to better out- comes and uses the clear statement to communicate with and engage with stakeholders. Those with a more conservative target-setting philosophy might view success as ensuring that the target is attained. Ultimately, the effectiveness of target setting itself may come down not to whether the specific targets are met, but whether the method of target setting influences investment decisions in ways that lead to better long-term results. Each target-setting philosophy could help influence investment decisions under the right conditions, though each through a different mechanism. For agencies that have a more aspirational target-setting philosophy, this philosophy could help galvanize stakeholders to work together and motivate greater collective action that improves outcomes. On the other hand, a realistic/predictive target-setting philosophy may convince stakeholders to take action by highlighting the realities of expected performance under the status quo. Use of a more complex quantitative approach may help to reveal influencing factors (e.g., traveler behavior, socioeconomic trends, multimodal influences or interactions, or changes to agency policy or infrastructure) that are driving performance outcomes, thereby indicating which intervention would be most likely to lead to improved performance. While a conservative target-setting philosophy seems less likely to spur discussions that lead to improved performance, under some circumstances, this philosophy might allow for less time and energy to be put into quantitative analysis and more time for other types of policy and strategy discussion.

10 Guide to Effective Methods for Setting Transportation Performance Targets In each of these scenarios, what matters for effectiveness is the extent to which the target- setting process (1) helps inform investments and strategies by providing greater information about the factors driving performance and the actions most likely to achieve desired outcomes and (2) motivates a broad array of stakeholders and decision-makers to engage in discussions about actions to meet performance targets. While it is difficult to say what target-setting methods are most effective in influencing decisions, given the short time that standardized performance measures and targets have been in place among DOTs and MPOs, a few key characteristics of methods may contribute to effectiveness: • Ease of application: Methods that require less staff time and fewer technical resources can be viewed as effective, since they may allow for more time to be devoted to more in-depth analyses of strategies rather than to the target setting itself. • Technical robustness: Methods that account for different factors affecting performance can be viewed as effective, since the process of setting the target may help to reveal how different strategies may influence performance and help in ultimately shifting decisions. • Ease of communication: Methods that are easy to communicate to stakeholders and that result in targets that are easy to understand (e.g., targets that conform to the desired direction for an outcome) can be viewed as effective, in that they make it easier to communicate the target to elected officials, stakeholders, and the public. • Allowing for policy considerations: Methods that incorporate policy objectives and long- term goals (e.g., moving toward zero fatalities) can be viewed as effective, since they help in communication and consistency across short- and long-range planning efforts. In Part II of this guide, each target-setting method is described with respect to these four char- acteristics to help readers compare methods on the basis of their own definitions of effectiveness.