Traffic Forecasting Accuracy Assessment Research (2020)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

II-7 Large-N Analysis 2.1 Introduction to Large-N Analysis The assessment of traffic forecasting accuracy in NCHRP 08-110 builds upon past efforts. Several researchers have assessed the accuracy of traffic forecasts, although most of the research projects have focused on toll roads. The inspiration for this may be that toll road forecasts have a bearing on investor expectations, and that is why their accuracy is considered more important. The Australian Government (2012) cited “inaccurate and over-optimistic’’ traffic forecasts as a threat to investor confidence. Three lawsuits now underway challenge the forecasts for toll road traffic that subsequently came in significantly under projections (Bain 2013). Inaccuracy of toll road traffic forecasts has been investigated in both international contexts (Bain 2011a; Bain and Polakovic 2005; Flyvbjerg et al. 2006; Gomez et al. 2016; Kriger et al. 2006; and Odeck and Welde 2017) and U.S. contexts (Kriger et al. 2006). These researchers identified that, in most cases, actual traffic volumes have been less than predicted. This error in the forecasts has been variously attributed to: • Less toll road capacity (when opened, compared with the forecast capacity); • Elapsed time of operation (roads that have been opened longer have higher traffic levels); • Time of construction (longer construction times delay traffic growth and increase the error); • Toll road length (shorter roads attract less traffic); • Cash payment (modern no-cash payment methods increase traffic); and • Fixed/distance-based tolling (fixed tolls reduce traffic). Bain (2011b) identified the toll culture (i.e., prior existence of toll roads, toll acceptance) and errors in data collection, as well as unforeseen microeconomic growth in the locality, as sources of inaccuracy. Flyvbjerg et al. (2006) attributed the errors to the uncertainties in trip generation and land-use patterns. From 2002–2005, Standard & Poor’s publicly released annual reports on the accuracy of toll road, bridge, and tunnel projects worldwide. The 2005 report (Bain and Polakovic 2005)—the most recent report available publicly—analyzed 104 projects. This report found that the demand forecasts for those projects suffered from optimism bias, and that the optimism bias persisted into the first 5 years of operation. Despite the assessments of forecast errors for toll roads, little comparable research has been done for non-tolled roads. A few recent studies have examined the accuracy of non-tolled roadway forecasts: Buck and Sillence (2014) demonstrated the value of using travel demand models in Wisconsin to improve traffic forecast accuracy and provided a framework for future accuracy studies. Parthasarathi and Levinson (2010) examined the accuracy of traffic forecasts for one city in Minnesota. Giaimo and Byram (2013) examined the accuracy of more than 2,000 traffic forecasts produced in Ohio between 2000–2012. They found the traffic forecasts slightly high, but within the standard error of the traffic count data. In their study of 39 road C H A P T E R 2

II-8 Traffic Forecasting Accuracy Assessment Research projects in Virginia, Miller et al. (2016) reported that the median absolute percent error of all studies was about 40%. (For a more detailed review of past assessments of traffic forecast accu- racy, see Part III, Appendix F.) The first phase of NCHRP 08-110 involved conducting a similar analysis using data on forecast and actual traffic for a combined data set of about 1,300 projects. The data for this analysis was obtained from six states and four European countries. The remaining sections of this chapter detail the data and methods adopted for this portion of the NCHRP 08-110 research and present a high-level description of forecast inaccuracy from the project team’s analysis. The quantile regression model is presented in this chapter, as are the general findings from the analysis. (For a more detailed discussion of the analysis results examining forecast inaccuracy as a function of several descriptive variables, see Part III, Appendix G.) 2.2 Data and Methodology The analysis discussed in this chapter used a database that was compiled as part of the NCHRP 08-110 project. The forecast accuracy database contains traffic forecast and actual traffic information for road projects in six U.S. states and four European countries. The records were compiled from existing databases maintained by the state departments of transportation (DOTs); from traffic forecasting reports; and from project reports, traffic impact statements, or EISs, and databases from similar research efforts. The forecast accuracy database contains information on the project itself (unique project ID, improvement type, facility type, location, and length), the forecast (year forecast produced, forecast year, forecast method), and the actual traffic count information. The forecast traffic and actual traffic for a project on a target year were compared, and several metrics were calculated to ascertain the level of inaccuracy in the traffic forecast. 2.2.1 Data The data in the forecast accuracy database includes information from Florida, Massachusetts (one project), Michigan, Minnesota, Ohio, and Wisconsin, and from Denmark, Norway, Sweden, and the United Kingdom. Additional data were made available to the project team from Virginia and Kentucky, and these data may be incorporated into the database at a future date. Table II-1 presents a short summary of the available information. In the table, the state names have been replaced by agency codes to protect anonymity. Agency All Projects Opened Projects Number of Segments Number of Unique Projects Number of Segments Number of Unique Projects Agency A 1123 385 425 381 Agency B 12 1 12 1 Agency C 38 7 5 3 Agency D 2176 103 1292 99 Agency E 12413 1863 1242 562 Agency F 463 132 463 132 Agency G 472 120 472 113 Total Segments 16697 2611 3911 1291 Table II-1. Summary of available data.

Large-N Analysis II-9 A segment is a unique portion of roadway for which a forecast is provided. The forecasts for an interchange improvement project may thus contain separate segment-level estimates for both directions of the freeway, for both directions of the crossing arterial, and for each ramp. Some of the projects for which data were provided had not yet opened, some of the segments did not have actual count data associated with them, and others did not pass the project team’s quality control checks for inclusion in the statistical analysis. Although all records have been retained for possible future use, the Large-N analysis discussed in this report is thus based on a filtered subset of 1,291 projects and 3,911 segments. The filtering process is discussed in a later section of this chapter. A range of transportation project types was included. Project opening years varied from 1970 to 2017, with about 90% of the projects opening in the year 2003 or later. Although the exact nature and scale of each project was not always known, inspection revealed that the older projects were more likely to be major infrastructure projects, whereas the newer projects were more likely to be routine work for the DOT (e.g., resurfacing works on existing roadways). Almost half of the projects were design forecasts for repaving. Such differences were driven largely by data availability. Some state agencies have begun tracking all forecasts as a matter of course, but the records that do so rarely go back more than 10–15 years. Data for the older projects were derived from someone going back to study and enter information from paper reports or scans of paper reports, and the availability of documentation and the interest in spending the effort to examine them was higher for bigger projects. Thus, the forecast accuracy database does not represent a random sample of projects, and notable differences exist not only in the methods used across agencies, but also in the mix of projects that were included in the database. This is an important limitation that readers should bear in mind to better understand and interpret the results of this NCHRP project. 2.2.2 Database Structure The forecast accuracy database assembled by the NCHRP 08-110 project team provided a starting point for the Large-N analysis. The data initially were collected and examined using a Microsoft Access database; however, to better incorporate the data archiving recommendations developed as part of this project and facilitate future research, the data were subsequently moved to an online repository. (For more information about the Forecast Cards and Forecast Cards Data repository, see Part I, Chapter 3, and Part III, Appendix A.) Three tables in the forecast accuracy database collect information that has been classified as: • Project information, • Forecast information, or • Actual traffic count information. The project information table contains the information that specifies each project’s character- istics. Fields include the project or report ID (unique to each project), project description, year when the project or report was completed, type of project, city or location where the project took place, state, construction cost, and so forth. The forecast information table contains the data related to the traffic forecast: the forecast itself, along with who made the forecast, at what time, and for what year. This table also includes the type of forecast year (opening, mid-design, or design year), the method used to create the forecast, whether any post-processing was or was not done, and similar information. The actual traffic count information table includes the actual traffic volume counted for a particular segment, the year of observation, and the project opening year. The key fields in the database are shown in Table II-2.

II-10 Traffic Forecasting Accuracy Assessment Research To ensure fair comparisons, the count units and the forecast units should be the same. Commonly used count units include: • Average Daily Traffic (ADT): The average number of vehicles that travel through a specific point of a road over a short-duration time period (FHWA 2018). • Annual Average Daily Traffic (AADT): The mean traffic volume across all days for a year for a given location along a roadway (FHWA 2018). • Average Weekday Daily Traffic (AWDT): The mean traffic volume, specifically on weekdays for a given location along a roadway. Whether AWDT is an annual average is not always clearly defined in the project documentation. • Typical Weekday Traffic: The traffic volume on a typical weekday, often defined as non-holiday weeks when school is in session. This count unit is a common output from travel models. Because reporting practices differ by state, the specific count units used in the data avail- able for this study varied by the source. Therefore, the project team elected to use the more general term average daily traffic (ADT) throughout this report even though the actual units were maintained as originally reported in the source data, and as documented in Part III, Appendix D. 2.2.3 Methodology Briefly, the goal of the Large-N analysis was to answer the question, “How close were the forecasts to observed volumes?” (Miller et al. 2016). To facilitate that, researchers have generally looked at two sets of similar data: one during the base year and the other one in the forecast year. Name Description Brief Description Brief written description of the project Project Year Year of the project or construction year or the year the forecast report was produced Length Project length (in miles) Functional Class Type of facility (e.g., Interstate, ramp, major/minor arterial) Improvement Type Type of project (e.g., resurfacing, adding lanes, new construction) Area Type Functional Class Area type where the facility lies (e.g., rural, urban) Construction Cost Project construction cost State State code. Internal Project ID Project ID, report ID, or request ID County County in which the facility lies Toll Type What kind of tolls are applied on the facility (e.g., no tolls, static, dynamic). Year of Observation Year the actual traffic count was collected Count Actual traffic count Count Units Units used to collect count information (i.e., AADT, AWDT) Station Identifier Count station ID or other identifiers for count station Traffic Forecast Forecast traffic volume Forecast Units Units used to forecast traffic (i.e., AADT, AWDT) Forecast Year Year of forecast Forecast Year Type Period of forecast like opening, mid-design or design period Year Forecast Produced The year the forecast was produced/generated Forecasting Agency Organization responsible for this forecast Forecast Methodology Method used to forecast traffic (e.g., traffic count trend, regional travel demand model, project-specific model) Post-Processing Methodology Any post-processing or alternative method used Post-Processing Explanation Explanation, as warranted, if a post-processing method was used Segment Description Description of the segment for which this forecast was done AADT = annual average daily traffic; AWDT = average weekday daily traffic Table II-2. Key Fields in the forecast accuracy database.

Large-N Analysis II-11 From the database and project reports, however, it became apparent to the project team that traffic forecasts are usually done for three analysis years: 1. The opening year, 2. A mid-design or interim year (usually 10 years after opening), and 3. The design year (usually 20 years from opening). This research evaluates the accuracy of opening-year forecasts for the practical reason that, for the vast majority of projects, the interim and design years have not yet been reached. Where necessary (i.e., in cases for which the opening-year traffic counts were unavailable), the project team resorted to taking the earliest traffic count available after a reasonable estimated year of completion and compared it with the scaled-up forecast volume. The project team scaled the forecast to the year of the first post-opening count so that both data points would be in the same year. The project team did this by linearly interpolating the forecast traffic between the forecast opening year and the design year. Notably, the data for the European projects was obtained from Nicolaisen’s PhD dissertation (Nicolaisen 2012) and had already been scaled to match the count year using a 1.5% annual growth rate. The NCHRP project team maintained this logic for the European projects, but for the U.S. projects did the interpolations between each project’s opening year and the design year as needed. One of the differences in methodologies in previous Large-N studies relates to how they define errors. Miller et al. (2016), CDM Smith et al. (2014), and Tsai, Mulley, and Clifton (2014) define error as the predicted volume minus the actual volume such that a result with a positive value is an overprediction. Odeck and Welde (2017), Welde and Odeck (2011), and Flyvbjerg, Holm, and Buhl (2006) define error the other way, such that a positive value represents an underprediction. A popular metric used to determine the accuracy of traffic forecasts is the half-a-lane criterion. This criterion specifies that the forecast is accurate if the forecast volume varies from the actual measured volume by less than half a lane’s capacity. If the forecast traffic volume is more than half a lane less than the facility’s actual capacity, the facility could have been constructed with one fewer lane in each direction. If the forecast traffic volume is more than half a lane greater than the facility’s actual capacity, the facility needs one additional lane in each direction. Calculating whether a forecast falls within a half a lane of the facility’s actual capacity requires several assumptions, such as the share of the daily traffic that occurs during the peak hour. Researchers have evaluated the accuracy of project-level traffic forecasts by comparing them with the actual traffic counts. Again, two schools of thought can be followed. Errors can be presented as (1) a percentage over the actual traffic (Tsai, Mulley, and Clifton 2014; Miller et al. 2016) or (2) a percentage over the forecast traffic (Flyvbjerg, Holm, and Buhl 2006; Nicolaisen and Næss 2015; and Odeck and Welde 2017). An advantage of the former approach is that the percentage is expressed in terms of a real quantity (observed traffic); an advantage of the latter approach is that—even though the actual (observed) value is unknown—a level of uncertainty can be expressed when the forecast is made (Miller et al. 2016). Besides these two methods, Bain (2009) and Parthasarathi and Levinson (2010) have evaluated forecast performance by taking the ratio of actual and forecast traffic. In this NCHRP study, the project team chose to continue in the convention described by Odeck and Welde (2017), who expressed the percent error as the actual count minus the forecast volume divided by the forecast volume. The project team recognizes that the Odeck and Welde approach differs from the standard convention of expressing percent error with the actual observation in the denominator, but found it more useful for understanding to express the error as a function of the forecast volume because the forecast volume is known at the

II-12 Traffic Forecasting Accuracy Assessment Research time the project decision is made (whereas the actual volume is not known until much later). As an example, if a planner knows that a 10% difference can be expected, then the planner has the option to apply that 10% to the forecast volume. To make this distinction clear, the project team chose to express forecast errors as the percent difference from forecast (PDFF): PDFF Actual Count Forecast Volume Forecast Volume 100%, (II-1)p= −i where PDFFi = the percent difference from forecast for Project i. Using this equation, negative values indicate that the actual outcome is lower than the fore- cast (an overprediction), and positive values indicate that the actual outcome is higher than the forecast (an underprediction). The appeal of this equation is that it expresses the error as a function of the forecast, which is known early enough to influence planning. The distribution of the PDFF values over the dataset can be analyzed to assess the systematic performance of the traffic forecasts in that dataset. As for expressing the error over the dataset, different researchers have varied in their use of mean percentage error (MPE) and mean absolute percentage error (MAPE). The MAPE has been acknowledged to “allow [researchers] to better understand the absolute size of inaccuracies across projects” (Odeck and Welde 2017) whereas when using the MPE the positive and negative errors tend to offset each other. The project team chose to continue in this tradition, but again translated it into the language of PDFF: Mean Absolute Percent Difference from Forecast MAPDFF 1 PDFF , (II-2) 1 p ∑( ) = =n ii n where n = the total number of projects. When assessing the forecast accuracy for a transportation project, an important question arises: What constitutes an observation? A typical road project is usually divided into several links or segments within the project boundary. The links usually occur on portions of the road that have differing alignments or carry traffic in different directions. To uniquely identify each project in the forecast accuracy database, a column was specified, titled “Internal Project ID.” Cells in this column typically were filled with the unique Financial IDs of each project. If a project did not have a unique Financial ID, a comparable unique ID (e.g., a report number, control number, or other identifying number) could be used. Using the same Internal Project ID, forecast and traffic count information for the different segments could be recorded by using unique Segment IDs. Analysis could be done on two levels, as follows: • Segment-level analysis, assessing the accuracy of the forecast for each different segment or link, and/or • Project-level analysis, assessing the total accuracy of the forecast for each project in the database. The limitation of presenting accuracy metrics at a segment level is that the observations are not independent. Consider, for example, a project with three segments connected end-to-end. It is reasonable to expect that the PDFFs on these segments are correlated—perhaps uniformly high or low. Whether one treats them as one combined observation or as three independent

Large-N Analysis II-13 observations, the average PDFF could be expected to be roughly the same. There would be a difference, however, in the measured t-statistics, wherein the larger sample size from a segment- level analysis could suggest significance where a project level analysis would not. Segment-level analysis is not without merit, however, given that a few measures of inaccuracy are better repre- sented at the segment level. For example, when assessing forecast inaccuracy over roadways of differing functional classes, segment-level results present a better representation than aggregated results over the entire project. Project-level analysis seems to be free of the correlation across observations described, but the question remains—how to assess the accuracy for a project? In the Virginia Study by Miller et al. (2016), each project consisted of links ranging from 1 to 2,493 in number. The researchers took the MAPE over the segments or links for individual projects and then used the mean to express the level of accuracy. Nicolaisen (2012) measured accuracy by taking the sum of the forecast and actual traffic volumes on the segments in a project. Another method, also described in Miller et al. (2016), is taking the weighted traffic volume: ∑ ∑ ( ) ( ) ( ) = = = Weighted Traffic Volume Volume on Link i Length of Link i Length of Link i (II-3)1 1 i n i n p In NCHRP 08-110, the issue with using the weighted traffic volume (forecast and actual) was the absence of length data in most of the records. In addition, taking the total traffic as suggested by Nicolaisen (2012) would not be able to show the relation between forecast accuracy and project type by vehicles serviced. Taking these issues into consideration, the project team chose to measure inaccuracy at the project level using average traffic volumes and giving equal weight to each segment within a project. Where relevant, the project team reported the distribution of PDFF at both the project level and the segment level. For project-level analysis, the average of the traffic volumes was taken and the error statistics were measured by comparing the average forecast and average actual traffic volume. Aggregating the counts and forecasts across the segments and links was done using the unique identifiers listed in the Internal Project ID column of the database. The variables for analysis also were aggregated by the same unique identifier, albeit with different measures for maintaining unifor- mity. For any given transportation project, the improvement type, area type, and functional class were taken to be the same as the most prevalent type or class among the segments. For example, if most of the segments in a project were categorized as Improvement Type 1 (i.e., resurfacing/ reconstruction/no major improvement), the project was considered to be of Improvement Type 1. Similarly, the forecast method was considered to be the same across the segments for a project, as were unemployment rates and years of forecast and observation. The means of these values were taken for the project-level analysis. Based on the nature of the forecast accuracy database, it was possible to select some vari- ables that might dictate future adjustments in the forecasts. These variables were the type of project (called “Improvement Type”), the method used (“Forecast Method”), the roadway type (“Functional Class”), the area type (“Area Type Functional Class”), and the forecast horizon (the difference between the year the forecast was produced and the year of opening of the facility). Odeck and Welde (2017) employed an econometric approach to determine the bias and the efficiency of the estimates by regressing the actual value as a function of the forecast value using the following equation: ˆ , (II-4)= α + β + εy yi i i

II-14 Traffic Forecasting Accuracy Assessment Research where yi = the actual traffic on Project i, ŷi = the forecast traffic on Project i, ei = a random error term, α and β = estimated terms in the regression, and here α = 0 and β = 1 implies unbiasedness. Li and Hensher (2010) conducted ordinary least squares (OLS) analysis and used the random effect linear regression model to explain the variation in the error forecast as a percentage over the explanatory variables (e.g., year of opening, elapsed time since opening). Miller et al. (2016) performed the ANOVA (analysis of variance) test on the median absolute percentage error on a limited number of explanatory variables (e.g., the difference between forecast year and opening year, forecast method, duration of forecast, and number of recessions between the base year and the forecast year). Both researchers found their models to be a good fit to explain the errors. The goal of such analysis is to present the range of errors of the forecast based on several variables, such as when the project was opened or the difference between the forecast year and the base year. The research for NCHRP 08-110 followed the Odeck and Welde (2017) structure, but introduced additional terms as descriptive variables: ˆ , (II-5)= α + β + γ + εy y Xi i i i where Xi = a vector of descriptive variables associated with Project i, and γ = a vector of estimated model coefficients associated with those descriptive variables. To consider multiplicative effects as opposed to additive effects, the regressors could be scaled by the forecast value: ˆ ˆ . (II-6)= α + β + γ + εy y X yi i i i i In such a formulation, γ = 0 indicates no effect of that particular term, whereas a positive value would scale up the forecast by that amount and a negative value would scale down the forecast by that amount. In addition to the estimation of biases, the project team was interested in how the distribution of errors relates to differing descriptive variables. For example, forecasts with longer time horizons might remain unbiased but have a higher spread, as measured by the MAPDFF. To examine this, the team extended the above framework to use quantile regression instead of OLS regression. Whereas OLS predicts the mean value, quantile regression predicts the values for specific percentiles in the distribution (Cade and Noon 2003). In past transportation research, quantile regression has been used for applications such as quantifying the effect of weather on travel time and travel time reliability (Zhang and Chen 2019), wherein an event may have a limited effect on the mean value but increase the likelihood of a long delay. Quantile regression also has been used to estimate error bounds for real-time traffic predictions (Pereira et al. 2014), an application more analogous to this project. The project team chose to estimate quantile regression models of the actual count as a function of the forecast and other descriptive variables. This was done for the 5th percentile, the 20th percentile, the median, the 80th percentile, and the 95th percentile. These estimates established an uncertainty window in which the median value provided the expected value, essentially an adjusted forecast. The 5th and 20th percentiles provided lower bounds on the expected value, and the 80th and 95th percentiles provided upper bounds.

Large-N Analysis II-15 2.3 Results 2.3.1 Overall Distribution Generally speaking, traffic forecasts have been found to be overpredicting: actual traffic volumes after project completion are lower than what has been forecast, as shown by the right- skewed distribution in Figure II-1. The 3,911 unique records/segments in the database reflected 1,291 unique projects. The project team noticed a general overestimation of traffic volumes across the projects. The distribution of PDFF shown in Figure II-1 is heavier on the negative side (i.e., actual volumes are generally lower than the volumes in the traffic forecasts). As seen in Table II-3, the MAPDFF is 17.29% with a standard deviation of 24.81. The results of the kernel density estimation display an almost normal distribution, albeit with long tails. On average, the traffic forecasts for a project were off by 3,500 vehicles per day. Overpredictions should be expected. In many cases, these traffic volume forecasts are used in design engineering. A design based on overpredicted traffic volumes will be overbuilt and will not see that extra capacity utilized (at least not immediately). On the other hand, if an under- predicted traffic volume is used as a basis for design, the road may open without adequate capacity for the actual traffic. In this situation, a subsequent project will likely be required to meet the demand—adding time, effort, and costs that could have been avoided. 2.3.2 Forecast Volume Figure II-2 reports the PDFF as a function of forecast volume at the project level. An interesting observation from the figure is the low percentage values as the traffic volumes increase. This is PDFF Forecast (Actual – Forecast) ∗ 100 Figure II-1. Distribution of PDFF (project level). Observations MAPDFF Mean Median Standard Deviation 5th Percentile 95th Percentile Project Level 1,291 17.29 -5.62 -7.49 24.81 -37.56 36.96 Table II-3. Overall PDFF (project level).

II-16 Traffic Forecasting Accuracy Assessment Research understandable, since the percentages were taken as a ratio over the forecast volume. Unless the actual traffic differs by a large margin, the PDFF values will not have risen to a big amount. When using the half-a-lane standard, the project team found that 95% of forecasts reviewed were “accurate to within half of a lane.” Table II-4 shows descriptive measures of PDFF by volume group for segments and projects. The measures represent the spread of the PDFF in the forecasts and include the mean, standard deviation, and the 5th and 95th percentile values. The MAPDFF value for each category presents how much the actual traffic volumes deviated from the forecast volumes. The mean values show the central tendency of the data. The standard deviation and the 5th and 95th percentile data represent the spread of the distribution. Some 90% of the data points fall between the 5th and 95th percentile values. One observation from Table II-4 is that as the forecast volume increases, the distribution of the PDFF involves smaller spreads in addition to the MAPDFF values getting smaller. For example, for forecast volume between 30,000 and 40,000 ADT, the PDFF values for 90% of the projects lie between -39.54% and 12.26% with absolute deviation of 16.17% on average. PDFF Forecast ∗ 100 (Actual – Forecast) Figure II-2. PDFF as a function of forecast volume (project level). Traffic Forecast Range (ADT) Observations MAPDFF Mean Median Standard Deviation 5th Percentile 95th Percentile 0–3000 133 24.59 -1.85 -5.75 42.15 -45.01 75.17 3001–6000 142 20.53 -0.37 -4.64 29.74 -36.50 50.33 6001–9000 125 16.75 -5.68 -8.80 21.94 -35.29 36.67 9001–13000 145 15.59 -4.66 -7.29 19.99 -31.34 34.45 13001–17000 143 17.41 -6.20 -6.53 21.61 -37.76 30.65 17001–22000 113 17.98 -5.65 -8.31 25.47 -41.62 37.85 22001–30000 133 19.54 -5.65 -8.47 25.36 -40.31 41.75 30001–40000 115 15.56 -9.78 -10.26 18.23 -39.54 12.26 40001–60000 137 13.18 -8.95 -7.68 16.01 -34.44 7.49 60000+ 105 10.20 -8.96 -7.90 9.90 -24.50 3.68 Table II-4. Forecast inaccuracy by forecast volume group (project level).

Large-N Analysis II-17 2.4 Quantile Regression Results The uncertainties involved in forecasting traffic call for assessing the risks and subsequently developing a range of forecasts of the traffic volumes that can be expected on a project. Considering the forecast accuracy dataset to be representative (i.e., a “national average”), the project team developed several quantile regression models to assess the biases in the forecasts related to the variables described in the previous chapter. The models were developed on the 5th, 20th, 50th (median), 80th, and 95th percentile values. Another goal of these econometric analyses was to obtain the range of actual traffic as a function of the forecast traffic and other project-specific criteria. Table II-5 explains the variables used in the quantile regression analysis. In the first model, the project team regressed the actual count on the forecast traffic volume. The structure for the regression followed Equation (II-4), which is described in the section on “Methodology.” The quantile regression parameter estimates the change in a specified quantile of the response variable produced by a one-unit change in the predictor variable. This allows comparison of how some percentiles of the actual traffic may be more affected by forecast volume than other percentiles. The effect is reflected in the change in the size of the regression coefficient. Variable Name Explanation AdjustedForecast Forecast ADT value for a segment/link or project. AdjustedForecast_over30k Variable to account for links with ADT value greater than 30,000. Defined as: If Forecast > 30,000 then value = Forecast – 30,000. Scale_UnemploymentRate_OpeningYear Unemployment rate in the project opening year. Scale_UnemploymentRate_YearProduced Unemployment rate in the year the forecast was produced. Scale_YP_Missing Binary variable to account for missing information in the year the forecast was produced. Column in the NCHRP database. Scale_DiffYear Difference between the year the forecast was produced and the forecast year (i.e., the forecast horizon). Scale_IT_AddCapacity Binary variable for projects that add capacity to an existing roadway. Reference class is the resurfacing/repaving/minor improvement projects. Scale_IT_NewRoad Binary variable for new construction projects. Scale_IT_Unknown Binary variable for projects of unknown improvement type. Scale_FM_TravelModel Binary variable for forecasts done using the travel model. Reference class is the forecasts done using traffic count trends. Scale_FM_Unknown Binary variable for forecasts done using an unknown method. Scale_FA_Consultant Binary variable for forecaster. Reference class is state DOTs. Scale_Agency_BCF Binary variable for projects done under the jurisdiction of Agency B, Agency C, or Agency F. Reference class is Agency A. Scale_Europe_AD Binary variable for European projects. Scale_OY_1960_1990 Binary variable for projects that opened to traffic before 1990. The reference value for the opening year is 2013 and later. Scale_OY_1991_2002 Binary variable for projects that opened to traffic from 1991 to 2002. Scale_OY_2003_2008 Binary variable for projects that opened to traffic from 2003 to 2008. Scale_OY_2009_2012 Binary variable for projects that opened to traffic from 2009 to 2012. Scale_FC_Arterial Binary variable for forecasts on major or minor arterials. Interstate or limited access facility is kept as the reference class. Scale_FC_CollectorLocal Binary variable for forecasts on collectors and local roads. Scale_FC_Unknown Binary variable for forecasts on roadways of unknown functional class. Table II-5. Descriptive variables for regression models.

II-18 Traffic Forecasting Accuracy Assessment Research Table II-6 presents the regression statistics (coefficients or α and β values and the t value to assess the significance). For the median, the project team observed that the intercept is not significantly different from 0 (zero), but the slope (the forecast volume coefficient) is signifi- cantly different from 1 (one), which can be interpreted as a detectable bias. In addition to detecting bias, these quantile regression models can be applied to obtain an uncertainty window around a forecast as follows: • 5th percentile estimate = –827 + 0.62 p forecast, • 20th percentile estimate = –434 + 0.81 p forecast, • Median estimate = 37 + 0.94 p forecast, • 80th percentile estimate = 1,396 + 1.05 p forecast, and • 95th percentile estimate = 2,940 + 1.42 p forecast. If a forecast is 10,000 ADT on a road, one would expect that the median number of vehicles to actually show up on the facility will be 9,437 ADT (37 + 0.94 * 10,000), which can be referred to as the median estimate, or alternatively as an expected value or adjusted forecast. One also would expect that for 5% of forecasts, the actual traffic will be less than 5,373, and that for 5% of forecasts, the actual traffic will be more than 17,140 ADT. The 20th and 80th percentile values can be calculated similarly. Table II-7 and Table II-8 give the ranges of the actual traffic volume and PDFF over the forecast traffic volume, respectively. 5th Percentile 20th Percentile 50th Percentile 80th Percentile 95th Percentile Pseudo R-Squared 0.433 0.619 0.723 0.750 0.748 Coef. t value Coef. t value Coef. t value Coef. t value Coef. t value Intercept -826.73 -10.55 -434.03 -5.06 37.15 0.54 1395.74 6.59 2940.45 6.50 Forecast Volume 0.62 30.68 0.81 89.56 0.94 148.10 1.05 76.12 1.42 42.26 Table II-6. Quantile regression results (actual count modeled as a function of the forecast volume). Forecast Forecast Window (Estimate) 5th Percentile 20th Percentile 50th Percentile 80th Percentile 95th Percentile 0 -827 -434 37 1,396 2,940 5000 2,294 3,612 4,742 6,670 10,047 10000 5,415 7,658 9,448 11,944 17,153 15000 8,536 11,705 14,153 17,218 24,259 20000 11,656 15,751 18,859 22,492 31,365 25000 14,777 19,797 23,564 27,766 38,471 30000 17,898 23,843 28,269 33,040 45,578 35000 21,019 27,890 32,975 38,314 52,684 40000 24,139 31,936 37,680 43,588 59,790 45000 27,260 35,982 42,385 48,862 66,896 50000 30,381 40,028 47,091 54,136 74,002 55000 33,502 44,075 51,796 59,410 81,109 60000 36,622 48,121 56,501 64,684 88,215 Table II-7. Range of actual traffic volume over forecast volume (actual count modeled as a function of the forecast volume).

Large-N Analysis II-19 Applying the coefficients as an equation, the project team constructed ranges of actual traffic and PDFF for differing forecast volumes (Figure II-3). The lines that depict various percentile values can be interpreted as the range of actual traffic over a forecast volume. For example, it can be expected that 95% of all projects with the forecast ADT of 30,000 will have actual traffic below 45,578. Only 5% of the projects experienced actual traffic less than 17,898. Not considering other variables, this range (45,578 to 17,898 for a forecast volume of 30,000) holds for 90% of the projects. Forecast Forecast Window: PDFF 5th Percentile 20th Percentile 50th Percentile 80th Percentile 95th Percentile 0 5000 -54% -28% -5% 33% 101% 10000 -46% -23% -6% 19% 72% 15000 -43% -22% -6% 15% 62% 20000 -42% -21% -6% 12% 57% 25000 -41% -21% -6% 11% 54% 30000 -40% -21% -6% 10% 52% 35000 -40% -20% -6% 9% 51% 40000 -40% -20% -6% 9% 49% 45000 -39% -20% -6% 9% 49% 50000 -39% -20% -6% 8% 48% 55000 -39% -20% -6% 8% 47% 60000 -39% -20% -6% 8% 47% Table II-8. Range of PDFF as a function of forecast volume (actual count modeled as a function of the forecast volume). Figure II-3. Expected ranges of actual traffic (base model). 0 10000 20000 30000 40000 50000 60000 0 10000 20000 30000 40000 50000 60000 Ex pe ct ed A D T Forecast ADT Perfect Forecast 5th Percentile Median 95th Percentile 20th Percentile 80th Percentile