Traffic Forecasting Accuracy Assessment Research (2020)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

II-3 1.1 Research Objective The Fixing American’s Surface Transportation Act of 2015 (FAST Act) provides $41.5 billion each year in roadway and bridge funding (U.S. DOT n.d.). Traffic forecasts are used in part to decide on how these public dollars are invested, through environmental studies, capital cost estimations, and cost-benefit analyses. At the same time, however, “the greatest knowledge gap in U.S. travel demand modeling is the unknown accuracy of U.S. urban road traffic forecasts” (Hartgen 2013). A relatively small set of empirical studies have examined non-tolled traffic forecasting accuracy in the United States. A need exists for research to expand the assessment and documentation of traffic forecasting experiences around the country to improve future modeling and forecasting applications, with the goal of ensuring that transportation funding dollars are invested wisely. The objective of NCHRP Project 08-110 has been to develop a process to analyze and improve the accuracy, reliability and utility of project-level traffic forecasts. 1.2 Overall Approach The research team’s review of past studies of forecast accuracy revealed two main methods of evaluating the accuracy of forecasts: deep dives (case study analysis) and Large-N studies. Deep dives are examples in which a single project is analyzed in detail to determine what went right and what went wrong in the forecast. Individual before-and-after studies from the FTA’s Capital Investment Grant Program are classic examples of deep dives. These studies often involve custom data collection before and after the project using methods such as onboard transit surveys. The sources of forecast errors—such as errors in inputs, model issues, or changes in the project definition—are considered and identified. The advantage of deep dives is that they allow thorough investigation of a complex set of issues. They also reveal the importance of the assumptions made by modelers in relation to data and the particular models that were used. The disadvantage of this method is that it is often unclear whether the lessons from one deep dive project can be generalized to others. Large-N studies consider a larger sample of projects in less depth. Flyvbjerg (2005) extols the virtues of Large-N studies as the necessary means of coming to general conclusions. Often, Large-N studies include a statistical analysis of the error and bias observed in the forecasts compared to actual data. Flyvbjerg et al. (2006) considered a Large-N analysis of 183 road and 27 rail projects, and Standard and Poor’s conducted a Large-N analysis with a sample of 150 toll road forecasts (Bain and Polakovic 2005). Other examples of Large-N studies included C H A P T E R 1 Overview

II-4 Traffic Forecasting Accuracy Assessment Research the Minnesota, Wisconsin, and Ohio analyses (Buck and Sillence 2014; Giaimo and Byram 2013; and Parthasarathi and Levinson 2010). The two approaches are not mutually exclusive, and the research for this NCHRP project has used both methods in a complementary manner. The Large-N analysis included compiling a database of forecast and actual traffic for about 1,300 projects from six states and four European countries. Currently, the forecast accuracy database is the largest known database for assessing forecast accuracy, and it allowed the NCHRP project team to statistically analyze the relation- ships between the projects’ actual traffic volumes, their forecast traffic volumes, and a variety of potentially descriptive variables. The project team also conducted a series of five complete deep dive analyses in an attempt to understand the reasons for forecast inaccuracy for specific projects. For several of those deep dives, the project team was able to reproduce the original travel model runs, which allowed testing of the effects of improving specific aspects of the forecasts. The focus on process requires that this research make recommendations about how to go about the analysis. The project team’s approach to doing so involved three related components: 1. Learn from others. Past efforts at evaluating forecast accuracy have yielded mixed successes, and important lessons can be learned from those efforts. For example, the lack of data availability has been identified as the biggest obstacle to progress in the field (Nicolaisen and Driscoll 2014). When conducting Large-N analyses, the analysis of outliers is very important, with large errors commonly occurring due to a mismatch between the forecast location and the count location (Byram 2015). The literature review (reproduced as Part III, Appendix F, in this report) includes a section that focuses on methods of evaluation. This section identifies approaches used by others to conduct such analyses for the purpose of providing a menu of options to this study. 2. Try it ourselves. One thing travel forecasters have learned is that the details matter. As a means of working through those details, no substitute exists for trying a proposed approach. This study has involved just that: conducting a set of deep dive analyses (five complete and one partial analysis) and conducting Large-N analysis on the data compiled from the 1,300 projects. 3. Ask stakeholders what works for them. For any forecasting process to be effective, it must be implemented by the actors who are involved in generating traffic forecasts. Therefore, it is important that the recommended process fit with the needs and priorities of the DOTs, MPOs, and other agencies (non-federal) responsible for generating those forecasts. The research was divided into two phases, with Phase 1 encompassing Step 1 and Step 2, and Phase 2 encompassing Step 3. Phase 1 involved analyzing existing data, focusing on what could be learned by conducting the analysis. This phase included both the statistical analysis of the Large-N data, and the deep dives into specific case studies. Phase 2 involved establishing the process that would best serve analysis of future data, and focused on engaging stakeholders to find out what works for them. This phase included a stakeholder workshop to review the Phase 1 findings, and the project team solicited input prior to making the final set of recommendations that appear in this NCHRP report. Phase 2 also included developing a working traffic forecast archive and information system to support the data collection necessary in order for the recommendations to be implemented. This research recognizes that there is value not only in establishing the process, but also in the findings of the analysis itself. These findings establish a baseline understanding of forecast accuracy that can later be updated as the process is applied. The project team addressed key questions with respect to both the analysis and the process.

Overview II-5 1.2.1 Analysis Questions Three largely descriptive issues were addressed by the Large-N analysis: 1. What is the distribution of forecast errors across the sample as a whole? 2. Can statistically significant bias be detected in the forecasts? If so, is that bias a function of specific factors such as the type of project, the time between the forecast and the opening year, or the methods used? 3. After adjusting for any bias, how precise are the forecasts? Is the precision a function of specific factors such as the type of project, the time between the forecast and the opening year, or the methods used? Taken together, the Large-N analysis and the answers to these questions provide a means of describing the range of forecast errors that have been observed historically for certain types of projects, albeit with some limitations on the types of data collected. An important caveat to such a result is similar to the axiom, “Past returns do not guarantee future performance.” Although it is useful to describe the historical performance of traffic forecasts for the purpose of establishing a track record, legitimate reasons may cause future accuracy to differ. These reasons could be positive—better models and data could produce more accurate forecasts—or negative—if all the cars start driving themselves, this change to the transportation could overwhelm other sources of inaccuracy. One can say little empirically about events that have not yet happened, so researchers must settle in this case for understanding past performance. Descriptive measures are useful and can shed light on certain factors associated with forecast errors, but they do not shed light on why the forecasts may be in error. To explore further, the deep dives focused on addressing the following questions: • What aspects of the forecasts (e.g., population forecasts, project scope) can be identified clearly as being accurate or inaccurate? • If the inaccurate aspects of the forecast had been accurate, how much would each have changed the traffic forecast? The researchers’ goal was to attribute as much of the error as possible to known factors. The remaining error would be labeled as occurring for “unknown reasons” and the project team would be able to say little about it beyond the fact that it was not due to the aspects that had been identified and quantified. 1.2.2 Process Questions Conducting these analyses also provided the opportunity to evaluate the process itself. The second set of research questions focused on establishing an effective process. Four questions were of particular interest: 1. What information should be archived from a forecast? 2. What data should be collected about actual project outcomes? 3. Which measures should be reported in future Large-N studies? 4. Can a template be defined for future deep dives? The project team compiled data from a number of varied sources, which meant differences occurred in the data that were available from those sources and in the ways those data had been structured. These differences limited what the project team could do with the data, but they also provided an opportunity to demonstrate what can and cannot be done with differing amounts and types of data.

II-6 Traffic Forecasting Accuracy Assessment Research Starting with the Large-N analysis, the project team made a significant effort to define a common set of fields in the forecast accuracy database and to make the definitions as consistent as possible. For example, the various agencies that supplied data may have coded their project types using different coding systems. The project team had to interpret what each system meant and establish common definitions so that the data across multiple agencies could be analyzed and compared. Moreover, in the final database, the project team observed that each agency had provided data that completed about two-thirds of the fields, but it was not the same two-thirds across all agencies. For model estimation, this necessitated that the project team create and include dummy terms for missing data on each field to avoid having to drop the records altogether. Having made these adjustments, if it was found that a particular term had been useful or insightful, the team could recommend that all agencies begin to collect information about that field. The same considerations were applied to the available data on actual project outcomes. Differing levels of detail were available for each of the six deep dives. Full model runs were available for three of the deep dives, which allowed the project team to test changes in inputs or in certain dimensions. Detailed traffic forecasting reports were available for another two deep dives, and for the last deep dive the project team relied on publicly available documents (i.e., the project’s environmental impact statements (EISs)). The research output includes a recommended deep-dive process that can be applied for future updates (see Part III, Appendix C). 1.3 Technical Report Contents Part II of this report documents the technical findings of the NCHRP 08-110 project team based on their analysis of the existing data and the process used to reach those findings. The issues identified in the analysis paved the way for a set of recommendations for forecasters to adopt in practice, along with recommendations for evaluating forecast results. Those recom- mendations are presented in Part I of this report. The remainder of Part II is structured as follows: • Part II, Chapter 2: Large-N Analysis. This chapter presents an analysis of the overall forecast accuracy of roughly 1,300 projects that were included in the final project database and included data for both forecast and counted average daily traffic (ADT). • Part II, Chapter 3: Deep Dives. This chapter analyzes in detail what was right and wrong with a set of five specific forecasts. • Part II, Chapter 4: Conclusions. This chapter summarizes the key findings from both portions of the analysis and re-visits the analysis questions and process questions presented in this overview chapter.