Traffic Forecasting Accuracy Assessment Research (2020)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

I-46 Reporting Accuracy Results The project team recommends that agencies involved in producing traffic forecasts periodi- cally report the accuracy of those forecasts relative to observed data. It is recommended that this reporting include three components: 1. A short (∼4-page) summary report that provides and updates the overall distribution of forecast error. This summary report would be written for consumers of traffic forecasts so they understand the range of errors to expect. The summary report would be updated approximately every 2 years as new projects open. 2. Estimates created using quantile regression models from local data. This step is optional, but doing this allows the agency to use a more specific range of expected outcomes as a function of the forecast rather than using the defaults produced by this research. 3. Specific deep dives aimed at understanding the sources of forecast error for either typical or important projects. This step should occur as a precursor to travel model improvement projects, as discussed in Part I, Chapter 5. The remainder of this chapter provides guidance on how to go about reporting these results. 4.1 Segment- and Project-Level Observations The research presented in Part II of this report presents results both at the segment level and at the project level. A segment corresponds roughly to a link in a travel demand roadway net- work, and is usually defined as a section of roadway between major intersections. A project can include one or more segments. For example, a road expansion project might include segments that are linearly connected along a corridor. Alternatively, an interchange project might include segments covering both directions of the mainline freeway, on the arterial roadway, and on each of the ramps. Thus, depending on the project, various segments can involve characteristics that are homogenous or that differ considerably. In the context of this research, it was not immediately clear whether segments or projects should be considered an observation. The project team’s preference was to aggregate the segments to projects before reporting the results, although situations may occur that war- rant a segment-level analysis (e.g., if the agency’s interest is in a particular type of facility). The project-level aggregation ensures that projects with more segments do not dominate the results. The project team’s preferred way to aggregate from the segment level to the project level is to sum the vehicle-miles traveled (VMT) across segments that have both counts and forecasts. In the data compiled for this project, the project team did not always have information about the length of the segment, so the team chose to average the traffic volumes instead. C H A P T E R 4

Reporting Accuracy Results I-47 4.2 Summary Reports Several metrics are used to report accuracy results. A popular metric used to determine the accuracy of traffic forecasts is the half-a-lane criterion. This criterion specifies that the forecast is accurate if the forecast volume varies from the measured volume by less than half a lane’s capacity based on the constructed facility’s capacity. If the forecast is more than half a lane less than the facility’s capacity, the facility could have been constructed with one fewer lane in each direction. If the forecast is more than half a lane greater than the facility’s capacity, the facility needs one additional lane in each direction. Calculating whether a forecast falls within half a lane of the constructed facility’s capacity requires making several assumptions, such as the share of the daily traffic that occurs during the peak hour. Details of how the project team calculated the metric in this research are included in Part II, but the project team recognizes that agencies may wish to vary those assumptions based on local practice. A brief discussion of the calculations used by the project team is provided here for reference. Past research has commonly reported the percent error (PE), defined as: = − ∗PE Forecast Counted Volume Counted Volume 100%, (I-6)i where PEi = the percent error for Project i. A limitation of this standard metric is that it uses the counted volume as the reference point and the counted volume is not known until after the project opens. For this reason, as discussed in the section on “Large-N Analysis: Data and Methodology,” the project team preferred to report the percent difference from forecast (PDFF), as measured in Equation I-1 (repeated here): = − ∗PDFF Counted Forecast Volume Forecast Volume 100%, (I-1)i where PDFFi = the percentage difference from forecast for Project i. Using either PE or PDFF, a perfectly accurate forecast will produce a value of 0%. The absolute magnitude of the percentages describes the magnitude of inaccuracy. With PDFF, the value will be negative if the actual outcome is lower than forecast, and the value will be positive if the actual outcome is higher than forecast. In this way, a project team can use forecast accuracy metrics to show the expected distribution of PDFF, and can use that distribution to understand the uncer- tainty surrounding a forecast. This approach aligns better with the quantile regression model reports, for which the team will want to know whether the actual value is likely to be higher or lower than what has been forecast at the time of the forecast. As a measure of the average bias in forecasts, the authors of this report recommend report- ing the average PDFF. As a measure of the average error in outcomes, the authors recommend reporting the mean absolute percent difference from forecast (MAPDFF), which is analogous to the more commonly used mean absolute percent error (MAPE). Both measures prevent positive and negative values from canceling each other out. As was discussed in the section on Large-N analysis, the MAPDFF is calculated as shown by Equation I-2: ∑( ) = ∗ =Mean Absolute Percentage Difference from Forecast MAPDFF 1 n PDDF , (I-2)ii 1 n where n = the total number of projects. The project team also suggests reporting the 5th and 95th percentiles of the PDFF distribu- tion, which provides a bound within which 90% of projects fall. For multiple projects, the project

I-48 Traffic Forecasting Accuracy Assessment Research team recommends showing the entire distribution of the selected metric(s) in a frequency chart because using only the average or mean error rates can mask important fluctuations. Figure I-1 (reproduced here for convenience) shows the distribution of PDFF at the project level from this research. The project team recommends that the summary report include the following metrics: • The distribution of PDFF (similar to Figure I-1); • The mean and median PDFF; • The mean absolute PDFF; • The 5th and 95th percentile PDFF; and • The percentage of forecasts that fall within half a lane of the constructed project’s capacity. Agencies also have the option to report these same metrics stratified across certain project characteristics if they wish to understand how the metrics vary in relation to those characteris- tics. These details are of greater interest to technical staff, and if explored they are best included in an appendix to the summary report. Project characteristics that agencies may wish to explore include: • Traffic volume, • Functional class, • Improvement type, • Area type, • Forecast method, • Forecast horizon, • Comparisons to previous summary reports, and • Comparisons to the default data that were used in this research. For categorical variables, the project team recommends both a tabular reporting of the results, such as that shown in Table I-7, and the use of violin plots, such as the plot shown in Figure I-9. A violin plot shows the distribution of the PDFF on the vertical axis, providing a simple way of visualizing the differences across categories. Appendix G (in Part III of this report) provides a more detailed anatomy of a violin plot. PDFF Forecast ∗ 100 (Actual – Forecast) Figure I-1. Distribution of PDFF (project level).

Reporting Accuracy Results I-49 4.3 Updating Quantile Regression Models As discussed in Part I, Chapter 2, quantile regression models can be used to estimate the uncertainty window around a forecast. A set of default models is provided with this guidebook; however, if an agency has collected data on traffic forecast accuracy, the quantile regression models can be re-estimated using the local data. Doing this is advantageous because it is based on data that are likely more similar to the types of forecasts that the agency will continue to perform. The task is to estimate a model of the actual traffic as a function of the forecast traffic. The estimate provides a model that can be used to predict the range of expected traffic if the forecast is known. The recommended model form can be expressed as follows: = α + β + γ + γ + + γ + εˆ ˆ ˆ . . . ˆ , (I-7), ,1 1, ,2 2, , , ,y y X y X y X yq i q q i q i i q i i q N N i i q i where yq,i = qth percentile of the actual (expected) traffic on Project i, ŷi = the forecast traffic on Project i, and eq,i = a random error term. Functional Class Observations Mean Absolute PDFF Mean Median Standard Deviation 5th Percentile 95th Percentile Interstate or Limited-Access Facility 434 12.32 -9.21 -8.48 13.58 -27.81 10.44 Principal Arterial 837 16.95 -9.63 -10.89 19.38 -37.51 23.95 Minor Arterial 404 18.92 -8.26 -10.24 24.54 -41.50 29.26 Major Collector 258 20.67 -10.81 -11.10 26.92 -51.11 23.85 Minor Collector 19 22.53 -12.74 -8.66 24.30 -41.43 28.58 Local 1 46.67 46.67 46.67 46.67 46.67 Unknown Functional Class 1958 32.42 10.69 2.68 53.67 -48.75 86.21 Table I-7. Forecast inaccuracy by functional class (segment-level analysis). Figure I-9. Distribution of PDFF by functional class (segment-level analysis).

I-50 Traffic Forecasting Accuracy Assessment Research In Equation I-7, αq is an estimated constant and βq is an estimated slope; X1,i through XN,i are descriptive variables associated with Project i, and γq,1 through γq,N are estimated model coefficients associated with those descriptive variables and those quantiles. Each variable is multiplied by ŷi, which makes the effect of that variable scale with the forecast volume (i.e., change the slope of the line) rather than be additive (i.e., shift the line up or down). Consider a median model in which α is 0, β is 1, and there is a single descriptive variable, X1,i, which is a binary flag (1 if the forecast is for a new road, and 0 otherwise). If γ1 has a value of –0.1, then it means that the median actual value would be 10% lower than the forecast. If γ1 has a value of +0.1, then it means that the median actual value would be 10% higher than the forecast. In this example, eq,i is a random error term. At a minimum, models should be estimated for the 5th, 50th, and 95th percentiles, with the option to include additional percentiles. The main difference between understanding a standard regression model and understanding a quantile regression model is that for each coefficient (αq, βq and γq,1 through γq,N) there is one estimated coefficient for each percentile considered (q), rather than a single coefficient. For consistency, the project team recommends the same model specification across all the percentiles, even if the coefficients are insignificant for some percentiles. Some guidance on expected coefficient values is provided by the discus- sion in the following bullet points. For simplicity, these points refer only to the 5th, 50th, and 95th percentiles, although the expectations are similar for other percentiles. • If the median is unbiased, α50 should be zero, α5 should be negative, and α95 should be positive. All values are in the same units as the forecast variable (ADT in this research). • If the median is unbiased, β50 should be 1, β5 should be less than 1, and β95 should be greater than 1. Each value serves as a scaling factor on the forecast. Values closer to 1 indicate a narrower range, and values farther from 1 indicate a wider range. • If a descriptive variable Xn,i has no effect, γq,n should be zero. • If γq,n has the same sign for all the percentiles, it indicates that the effect of that variable is to shift or bias the results in that direction. Alternatively, it is possible that only the median estimate is biased, as would be indicated by the value of γq,n • If γ5,n is negative and γ95,n is positive, it indicates that the variable serves to expand the range of expected forecasts, and therefore the variable is associated with more uncertainty. • If γ5,n is positive and γ95,n is negative, it indicates that the variable serves to narrow the range of expected forecasts, and therefore the variable is associated with less uncertainty. The default model specifications can be used as a starting point for local models. Additional variables can be tested and evaluated based on the logic of the resulting coefficients and their statistical significance. Detailed guidance on the art of model specification is beyond the scope of this guidebook. Various statistical packages can be used to estimate quantile regression models. As an open- source package that works across a variety of platforms, R was used for the models developed in this research. Python scripts for estimating basic quantile regression models also were included as a starting point for future analyses. It is important that projects used to develop the quantile regression equations be (1) suffi- cient in quantity to produce statistically significant coefficient estimates and (2) representa- tive of all the types of forecasts made. If an agency does not have a sufficient sample of local projects to support model estimation, the agency can and should supplement the local data with data from projects at peer agencies. The data provided with this report also can be used. The project team also recommends that, to the extent possible, agencies use a census of all projects rather than a sample. Including all projects will avoid the risk of “cherry picking”

Reporting Accuracy Results I-51 forecasts that are highly accurate or inaccurate. The next section in this chapter describes scripts that are provided to compile forecast accuracy data in a format suitable for quantile regression estimation. 4.4 Using the Forecast Archive and Information System Summarizing forecast accuracy metrics is straightforward if the data are available in a clean format, meaning that the data are provided in a table with one record for each observation (project or segment), and at a minimum with one column for the forecast traffic volume and one column for the actual traffic volume. Other fields can be included if they are of interest for segmenting the data, or as descriptive variables in quantile regression. The data should only include projects that are open and have both forecast and actual traffic volumes. The data also should avoid duplicates, and they should be filtered to include only the forecasts of interest (e.g., opening-year forecasts if design year is to be excluded). Once the data are in this format, standard statistical software or a spreadsheet can be used to calculate the metrics discussed. As with many statistical analyses, the bulk of the effort is in assembling and cleaning the data. Chapter 3 in Part I described the Bronze archiving standard and associated actual data, and described a forecast archive and information system to facilitate the storage of these data using the forecast cards structure. If agencies follow the Bronze standard, they will acquire the data necessary to assess forecast accuracy locally. If agencies also share their data, the assessments can combine local and external information. By separating projects into separate folders and the related data into separate CSV files, the forecast cards data structure is optimized to make it easy to store and share the data. However, separate forecast cards are not well suited to analyz- ing the data—for that purpose, a single combined file with the relevant fields performs better. To accommodate such analysis, the project team has developed Python scripts to convert data from forecast cards into a flat file format. The main data_validation_and_preparation script follows the process that was used to clean the data for this research. The process includes the following steps: 1. Get and validate the forecast cards. This step reads the forecast cards from a designated re- pository and checks that they conform to the data specification and have the required fields coded appropriately. 2. Combine data from multiple sources. In this step, data can be combined from multiple repositories. The repositories could include local data from a hard drive and data pulled from a shared public repository. 3. Combine and format data for estimation. The data are combined into a single table with one record per observation and the appropriate fields. 4. Filter records. The data are filtered to exclude projects that have not yet opened, do not have observations, or otherwise are problematic. 5. Explore the estimation dataset. In this step, summary statistics are printed for the combined dataset. As an extension to this process, the resulting data can be used to estimate new quantile regres- sion models, as opposed to using the default models provided with this research. Users may want to use this process either to (1) update the models with new data as more projects open, or (2) estimate models from local data, reflecting the accuracy of their own agency’s forecasts. Estimating models based on local data may result in uncertainty windows that are narrower or wider than the defaults obtained from this research. Once the data have been formatted with the data_validation_and_preparation script, the process of estimating new quantile regression models is similar to estimating linear

I-52 Traffic Forecasting Accuracy Assessment Research regression models where the actual value is the dependent variable and the forecast value is the descriptive variable. An estimate_quantiles script is provided to estimate a basic model with just this one variable. The spreadsheet that implements quantile regression models can be updated with new estimation results to test their effect. Instructions for running the data_validation_and_preparation and estimate_ quantiles scripts are available at https://github.com/uky-transport-data-science/forecastcards. 4.5 Deep Dives A deep dive is a case study of the accuracy of a forecast for a particular project. It can only be performed after a project has opened to traffic, so it requires the passage of time from the original forecast. The goal of a deep dive is to investigate the sources of forecasting error or reasons for forecasting accuracy of a project, using a retrospective look. The project team’s recommended method involves 7 steps: 1. Record all information about the forecast. This step should be completed shortly after the forecast has been finalized, as described in Part I, Chapter 3. For a deep dive to be successful, the forecast should be archived at either the Silver level or the Gold level. 2. Collect data about the forecast itself and the inputs and assumptions behind the forecast. The data plan specified in the Silver-level annotated outline (presented in Appendix B) can be executed after project opening. 3. Compare project forecasts with actual values. This step is taken to identify accurate and inaccurate forecasts. The comparison of project forecasts and actual volumes should account for any differences between the assumed and actual opening years in order to avoid focusing on errors caused by construction or funding delays. 4. Compare inputs and assumptions with actual values. This step is taken to categorize the accurate and inaccurate assumptions behind the forecast. An initial comparison can include typical or ordinary inputs and the assumptions that are common in most traffic forecasts (e.g., population estimates, auto fuel prices, and changes in land use, travel times, and other variables). Given that these variables are commonly used in traffic forecasts, it is reasonable to assume that errors in these assumptions will contribute to the inaccuracy of the forecast. Once the initial comparisons are complete, a secondary comparison can be made with proj- ect-specific assumptions such as the model’s grasp of travel markets at the time the forecast is generated and the model’s grasp of expected travel markets. The results of both comparisons should be documented. 5. Adjust the inputs and assumptions and generate adjusted forecasts. This step involves re- running the full model or using elasticities. The best method for rerunning the model to assess the causes of forecast inaccuracy is to cor- rect the inputs and assumptions and reproduce the forecast using the same method—usually a travel demand model—that was used to develop the original forecast. In this way, the cumu- lative impacts from the corrected inputs will be estimated, resulting in an adjusted forecast. Ancillary forecasts, which correct just one input, can be produced to enhance understanding of the causes of error. The development of ancillary forecasts is not an option for forecasts that have been made using traffic count or population trendlines. For these situations, the only comparison that can be made is computing the forecasting error. Forecasting error from trend lines can be compared to the error from other methods to determine the best methods for certain project types. A thor- ough analysis will include the differences in costs and resources needed to generate the forecasts along with the forecast error to produce a more complete picture.

Reporting Accuracy Results I-53 Elasticities measure the change in the traffic forecast given a known change in an input or assumption to the forecast. Elasticities are usually expressed as ratios. An elasticity of +0.3 means that the traffic forecast would increase by 3% for every 10% increase in the input value. Conversely, an elasticity of -0.2 means that the traffic forecast would decrease by 2% for every 10% increase in the input value. The equations for adjusting the forecast are: = −)( )∗Effect on Forecast 1, and (I-8)Elasticity (1+Change in Valueexp ln ( )= + ∗Adjusted Forecast 1 Effect on Forecast Actual Forecast Volume. (I-9) Elasticities are general rules of thumb and assume that the forecast changes entirely due to the change in one variable. For the elasticity analysis to be effective, the forecasts must be changed cumulatively for every variable. Figure I-10 shows an example of how the computations can be applied. 6. Compare and analyze the adjusted forecasts with actual values. This step is taken to determine the major sources of error or accuracy. Some key questions that might need to be addressed in the analysis are: – Would the project decision have changed if the forecast accuracy or reliability were improved? – How useful were the forecasts in terms of providing the necessary information to the planned process? – Was risk and uncertainty considered in the forecast? – How were risk and uncertainty communicated in the forecast? Based on the analysis, changes to the forecasting method or the model validation can be suggested for future forecasting efforts. 7. Summarize the findings. The summary can be produced using a memo format so that the results can be easily recalled when needed. An annotated outline for deep dive analyses is provided in Part III, Appendix C. This outline references several interactive Excel tables. Customizable files for the outline and the companion Excel tables can be downloaded from the NCHRP Research Report 934 webpage at www.trb.org. Seg# Items Actual Value Forecast Value Change Required in Forecast Value Elasticity Effect on Forecast Actual Forecast Volume Adj. Forecast Volume Remaining % Error for Adj. Forecast 1 Employment 38,801 48,312 -20% 0.30 -6% 10,262 9,609 13% 1 Population/Household 78,576 80,854 -3% 0.75 -2% 9,609 9,405 11% 1 Car Ownership 54,603 56,084 -3% 0.30 -1% 9,405 9,330 10% 1 Fuel Price/Efficiency $2.340 $1.820 29% (0.20) -5% 9,330 8,872 5% 1 Travel Time/Speed - - 0% (0.60) 0% 8,872 8,872 5% 1 Original Traffic Forecast 8,474 10,262 21% N/A N/A 1 Adjusted Traffic Forecast N/A N/A N/A 10,262 8,872 5% Figure I-10. Example of adjusting forecasts using elasticity computations.