Traffic Forecasting Accuracy Assessment Research (2020)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

I-25 Forecasting is a challenging activity that always includes an inherent level of uncertainty, yet forecasters traditionally have been asked to provide what is called a point forecast—usually a single estimate of the traffic volumes on a project. As provided, the point forecast is devoid of any recognition of uncertainty. Consequently, any errors in the forecast will be blamed on the forecaster and the travel model. The project team recommends that forecasters use a range of forecasts to communicate this uncertainty. Effectively communicating uncertainty to forecast recipients has the following benefits: • Those who apply the forecast are made aware that the actual traffic volume may deviate from the primary point forecast, which encourages them to account for uncertainty in their decisions. • The uncertainties that are part of the inputs and the assumptions behind the forecasts are passed along to the customers of the forecast. The inputs and assumptions that go into a traf- fic forecast, such as projections of population and employment growth, incorporate inherent uncertainties. The travel forecaster typically “absorbs” these uncertainties when providing point forecasts, which—by their nature—render the uncertainties invisible. • Counted volumes that fall within the range of uncertainties will be deemed accurate, relieving forecasters of the need to defend even minuscule deviations from their point forecasts. • The general acknowledgement that travel models and forecasts cannot be perfectly accurate is reinforced. This reinforcement helps maintain, in the eyes of both decision makers and the public, the more realistic view that forecasting models are a tool used to provide general information about project demand. This chapter describes how to use measured forecast accuracy to communicate uncertainty. 2.1 Quantifying Uncertainty One method that can be used to quantify uncertainty is to vary the forecast inputs and assump- tions to reflect their uncertainty ranges and rerun the travel demand model with multiple inputs. This process can be repeated many times, so that all primary inputs can be varied by their (mini- mum and maximum) extreme values individually and collectively. The result is a distribution of outcomes that reflect the specified range of inputs and assumptions. This method is less pragmatic if the travel model has long running times, if project schedules are constrained, or if a simple trend line extrapolation was used to produce the point forecast. These situations commonly occur in traffic forecasting, so an alternative method is needed. Using quantile regression lines derived from archived forecasts offers an alternative method. Quantile regression is like linear regression. Instead of computing the standard errors based on C H A P T E R 2 Using Measured Accuracy to Communicate Uncertainty

I-26 Traffic Forecasting Accuracy Assessment Research the sample mean, the errors are based on a specified quantile of the sample (e.g., the 10th per- centile, 20th percentile, and so forth). Figure I-4 illustrates the difference between linear regression and quantile regression. This figure represents a set of project forecasts in relation to the corresponding ADT values from actual counts. Each point represents one project, with the horizontal position determined by the forecast ADT value (which was predicted several years prior to each project’s opening year) and the vertical position determined by the actual ADT value (which was measured after each project opened). If all of the points fell perfectly on the diagonal, it would indicate that all the forecasts were perfect predictors of the actual outcomes. Figure I-4 represents a more typical scenario in which the points do not fall perfectly on the diagonal, indicating some inaccuracy in the forecasts. Estimates made using standard linear regression allow a line to be drawn through the middle of the cloud of data (the central diagonal line in the figure). If the regression line has a slope of 1 and an intercept of 0, it is taken to indicate that the forecast ADT values provided an unbiased estimate of the actual ADT values. However, even if the forecasts are unbiased, not all forecasts will be perfectly accurate. By contrast, quantile regression can be used to draw lines at the edges of the cloud (in Figure I-4, the top and bottom blue lines). Like the central blue line, the top and bottom lines reflect estimates based on the cloud of data—but rather than the mean or average of the cloud, quantile regression yields lines that show the range within which most of the points fall. The top and bottom lines in Figure I-4 do not capture all the project points (outliers will always exist); however, the range of values between the top and bottom lines clearly captures most of the points. Applying this technique provides a means for forecasters to clarify their forecasts by supplying upper and lower forecast bounds. A ct ua l A D T Forecast ADT Figure I-4. Sample forecasting accuracy data.

Using Measured Accuracy to Communicate Uncertainty I-27 2.2 Introduction to Quantile Regression Quantiles, or percentiles, are breakpoints that divide a frequency distribution into intervals with the specified probability. For example, the 5th percentile (quantile 0.05) is the value for which there is a 0.05 probability of a value drawn randomly from the distribution being lower than the specified value. At the 95th percentile, a 0.05 probability exists that a value drawn ran- domly from the distribution will be higher than the specified value. A range of quantiles can be used to express a range of likely outcomes. Given a historical dataset of forecast and actual traffic volumes one can estimate a model: = α + β + εˆ , (I-4)y yi i i where yi = the actual traffic on Project i, ŷi = the forecast traffic on Project i, ei = a random error term, and α and β = estimated terms in the regression. Here, α = 0 and β = 1 implies unbiasedness. Whereas linear regression would estimate a single α and single β, quantile regression instead estimates one α for each quantile of interest and one β for each quantile of interest. Such a model must be estimated based on historical data—using forecasts that were made in the past for projects that have since opened, such that actual data can be collected. OLS and random effect linear regression may be used to explain variations in error forecasts as functions of explanatory variables (such as year the project opens or elapsed time since opening). To compare the effects of potential explanatory variables, the model used was: = α + β + γ + εˆ , (I-5)y y Xi i i i where Xi = a vector of descriptive variables associated with Project i, and γ = a vector of estimated model coefficients associated with those descriptive variables. Taking another look at Figure I-3 (reproduced here for convenience), the expected ranges of actual traffic illustrate the results of quantile regressions. The lines in this figure were estimated from a sample dataset of project forecasts and actual volumes. In the figure, the x-axis represents the forecast ADT, and the y-axis represents the quantile regressions that were made based on the actual ADT in the year for which the forecast was made. From top to bottom, the chart’s six lines represent specific aspects of the forecast: 1. The orange line represents estimates of actual volumes based on the lowest 95% of forecast error values; 2. The green line represents estimates of actual volumes based on the lowest 80% of forecast error values; 3. The gray line represents the perfect forecast (i.e., a 45° line depicting a situation in which the actual ADT always matches the forecast ADT), and is used for reference; 4. The yellow line represents the 50th percentile (median) of forecast error values; 5. The dark blue line represents estimates of actual volumes based on the lowest 20% of forecast error values; and 6. The light blue line represents estimates of actual volumes based on the lowest 5% of forecast error values.

I-28 Traffic Forecasting Accuracy Assessment Research In general, the quantile regression lines (lines 1, 2, 4, 5 and 6) represent estimated changes in the actual or measured volumes, based on the nth percentile of error values, given changes in forecast volume. Consequently, lower percentile lines (percentiles less than 50%) more strongly reflect overforecast values and higher percentile lines (percentiles greater than 50%) more strongly reflect underforecast values. Error ranges (the differences between the 80th and 95th percentile lines and the 20th and 5th percentile lines) generally widen as the forecast values increase. A point on the quantile regression lines represents an estimate of the actual volume, based on the nth percentile of error values, for a selected forecast volume. If the yellow line representing the median aligns perfectly with the gray line, it means that the forecasts are unbiased. The lower and higher quantile lines provide the range within which actual ADT falls. For example, 60% of actual outcomes are expected to fall between the 20th and 80th percentile lines, and 90% of actual outcomes are expected to fall between the 5th and 95th percentile lines. If the 5th and 95th percentile lines are closer to the diagonal, it indicates that actual outcomes fall in a narrower range around the forecast. Although quantile regression models such as these must be estimated using data obtained after the project opens, they can be applied to future forecasts that are made before the opening of other, similar projects. In this way, they can be used to provide a range of expected actual values. The wording of such statements would be, “Based on historic accuracy, if we have a fore- cast of X, we would expect that 90% of actual outcomes to fall between the range of Y5 and Y95.” 2.3 Default Versus Local Quantile Regression The next section provides examples that apply the quantile regression models that were esti- mated for this research using data from several agencies. These models are best viewed as a default, and should be used in the absence of other data. It is important to recognize that the specific types of projects considered by and forecasting methods used by an agency may differ 0 10000 20000 30000 40000 50000 60000 0 10000 20000 30000 40000 50000 60000 Note: Figure I-3 shows an example of quantile regression results. Ex pe ct ed A D T Forecast ADT Perfect Forecast 5th Percentile Median 95th Percentile 20th Percentile 80th Percentile Figure I-3. Expected ranges of actual traffic (base model).

Using Measured Accuracy to Communicate Uncertainty I-29 from these defaults. It is therefore preferable to compile local forecast accuracy data using the methods described in Chapter 3, and to use those data to estimate local quantile regression models as described in Chapter 4. Doing so is expected to provide a better view of the uncer- tainty window relevant to a particular agency. If an agency develops a track record of producing accurate forecasts, then the resulting quantile regression models will have a narrower range, supporting more certainty in project decisions. 2.4 Applying Quantile Regression Methods Quantile regression lines that have been developed from a historical dataset of project traffic forecasts and measured or actual traffic provide uncertainty ranges based on past experience. In conjunction with Figure I-3, the information in Table I-1 can provide an example. Assume that Figure I-3 was created by an agency that recently developed quantile regression lines using its historical accuracy dataset. The agency has produced a traffic forecast of 40,000 ADT for a project and wishes to use quantile regression to provide an uncertainty range. In this example for a 40,000 ADT forecast, the percentile estimates are shown in Table I-1. The agency has the option of providing multiple uncertainty ranges, depending on their confidence in the forecast and historical track record on forecast accuracy: • A narrower range would involve the 20% and 80% percentile estimates. Based on the agency’s accuracy track record, about 60% (6 of 10) of past projects would have produced an actual traffic volume between 32,000 and 44,000. The 60% value is derived by subtracting the lower percentile value from the higher percentile value (80% - 20% = 60%). • A wider range would involve the 5% and 95% percentile estimates. Based on the agency’s accuracy track record, about 90% (9 of 10) of past projects would have produced an actual traffic volume between 25,000 and 60,000. These ranges can be provided to decision makers, agency colleagues and/or the public in lieu of the 40,000-point forecast. • The use of a narrower or wider band would depend on the agency’s tolerance for risk. The authors of this report recommend the 5th and 95th percentiles as a default range. These regression equations can include many variables. The project team assembled three ver- sions of quantile regressions using the traffic database as discussed in the technical report (Part II of this report). The three versions are: 1. A simple version that includes only the forecast volume and an intercept as variables. 2. A forecasting version that includes selected variables. This version best represents a practical application because the included variables should be known at the time the forecast is made. 3. An inclusive version that includes an extensive set of variables. This version is best used for applied research purposes because it includes variables that may only be known after the project opens. Percentile Actual Volume Estimate 5% 25,000 20% 32,000 80% 44,000 95% 60,000 Table I-1. Actual volume estimates given 40,000 ADT forecast (example).

I-30 Traffic Forecasting Accuracy Assessment Research Table I-2 lists the variables used in each version. This table shows forecasters a range of possible variables for their individual use. Agencies may choose to include other variables. The project team recommends that such other variables be included subject to three conditions: 1. The variables are important to the agency at the time the forecast is communicated to decision makers, 2. The empirical and statistical significance is reviewed prior to inclusion, and 3. Data exists to support the analysis. The purpose of the variables is to widen or narrow the quantile estimates based on project characteristics, forecast assumptions, and conditions in place at the time the forecast is made. For each project, the forecaster can input the variables to generate the quantile regression lines. Then the forecaster can estimate the upper and lower bounds of the range based on (a) the forecast volume and (b) the percentile line desired for the uncertainty range. Two further examples are provided to demonstrate the application. In both of these exam- ples, an agency has developed five quantile regression equations for 5th, 20th, 50th, 80th and 95th percentiles. The variables and their magnitudes for each of the five quantiles are provided in Table I-3. For the first demonstration, assume that Agency B recently used a travel model to generate a forecast of 50,000 ADT for a new freeway. The opening year will be 10 years from today. The Variable Simple Forecasting Inclusive Intercept √ √ √ Forecast volume √ √ √ High forecast volume (if forecast volume 30k+) √ √ Unemployment rate (for year of forecast) √ √ Unemployment rate (anticipated for opening year) √ Forecast horizon (years between year of forecast and anticipated opening year) √ √ Forecast horizon is unknown √ New roadway √ √ Project adds capacity √ Unknown improvement type √ Travel model used to produce forecast √ √ Unknown method used to produce forecast √ Forecast was developed by consultant (outside agency) √ Forecast was produced between 1960 and 1990 √ Forecast was produced between 1991 and 2002 √ Forecast was produced between 2003 and 2008 √ Forecast was produced between 2009 and 2012 √ Forecast is for arterial roadway √ √ Forecast is for collector or local roadway √ √ Forecast is for an unknown roadway √ Note: A checkmark (√) denotes the variable is included in the quantile regression equation. Table I-2. Variables used in the quantile regression analysis.

Using Measured Accuracy to Communicate Uncertainty I-31 unemployment for the region was 4% at the time of the forecast. After the forecast is finalized, the forecaster enters the information from Table I-4 into a spreadsheet. The spreadsheet is the basis for computing the resulting regression lines and producing the chart shown in Figure I-5. Agency B decides to communicate uncertainty using the 20th and 80th percentile lines as the lower and upper bounds of the range. With the 50,000 ADT forecast, the agency provides the forecast range of 45,000 to 54,000 ADT. Now assume that Agency C recently used a travel model to generate a forecast of 15,000 ADT for a widening of a local roadway. The opening year will be 2 years from today. The unemployment for the region was 8% at the time of the forecast. After the forecast is finalized, the forecaster enters the information from Table I-5 into a spreadsheet. The spreadsheet allows the forecaster to compute the resulting regression lines, producing the chart shown in Figure I-6. Agency C decides to communicate uncertainty using the 5th and 95th percentile lines as the lower and upper bounds of the range. With the 15,000 ADT forecast, the agency provides the forecast range of 7,500 to 17,500 ADT with a 90% certainty that the counted traffic will fall between the upper and lower bounds. The 50th percentile should be considered the “most likely” or expected value and may be an adjustment from the forecast. Pseudo R-Squared Coef. t value Coef. t value Coef. t value Coef. t value Coef. t value (Intercept) - -182.27 -1.77 154.578 3.08 255.551 4.67 287.909 3.94 976.786 4.79 AdjustedForecast 0.70464 15.97 0.73181 36.19 0.89089 45.20 1.02667 44.19 1.25361 23.88 AdjustedForecast_over30k 0.02375 0.57 0.05735 3.05 -0.0042 -0.22 -0.1902 -8.30 -0.4132 -9.89 Scale_UnemploymentRate_YearProduced -0.0058 -1.41 0.00487 2.77 0.00164 0.87 0.00693 2.76 0.00999 1.87 Scale_YearForecastProduced_before2010 -0.0067 -5.64 -0.0051 -5.23 0.00022 0.27 0.00384 3.91 0.00324 2.36 Scale_DiffYear 0.00586 2.81 0.00898 6.68 0.00759 5.62 0.0142 8.23 0.0196 10.50 Scale_IT_NewRoad 0.09326 4.34 0.00948 1.10 -0.0081 -0.90 -0.036 -1.93 -0.0901 -4.29 Scale_FM_TravelModel 0.06756 3.31 0.0136 1.63 -0.0076 -0.52 -0.0185 -1.25 -0.1006 -7.36 Scale_FC_Arterial -0.1495 -5.24 -0.061 -4.86 -0.0621 -5.17 -0.084 -5.96 -0.1163 -5.88 Scale_FC_CollectorLocal -0.2121 -4.03 -0.1114 -4.79 -0.1255 -5.21 -0.2008 -5.78 -0.3214 -2.36 5th Percentile 50th Percentile 95th Percentile 0.475 0.739 0.830 20th Percentile 0.631 80th Percentile 0.804 Table I-3. Example of quantile regression variables and coefficients. User can change the yellow cells to see the effect. Pseudo R-Squared Coef. t value Coef. t value Coef. t value Coef. t value Coef. t value Values 5th Percentile 20th Percentile 50th Percentile 80th Percentile 95th Percentile (Intercept) - -182.27 -1.77 154.578 3.08 255.551 4.67 287.909 3.94 976.786 4.79 -182 155 256 288 977 AdjustedForecast 0.70464 15.97 0.73181 36.19 0.89089 45.20 1.02667 44.19 1.25361 23.88 0.705 0.732 0.891 1.027 1.254 AdjustedForecast_over30k 0.02375 0.57 0.05735 3.05 -0.0042 -0.22 -0.1902 -8.30 -0.4132 -9.89 Scale_UnemploymentRate_YearProduced -0.0058 -1.41 0.00487 2.77 0.00164 0.87 0.00693 2.76 0.00999 1.87 4 -0.023 0.019 0.007 0.028 0.040 Scale_YearForecastProduced_before2010 -0.0067 -5.64 -0.0051 -5.23 0.00022 0.27 0.00384 3.91 0.00324 2.36 - - - - - - Scale_DiffYear 0.00586 2.81 0.00898 6.68 0.00759 5.62 0.0142 8.23 0.0196 10.50 10 0.059 0.090 0.076 0.142 0.196 Scale_IT_NewRoad 0.09326 4.34 0.00948 1.10 -0.0081 -0.90 -0.036 -1.93 -0.0901 -4.29 1 0.093 0.009 -0.008 -0.036 -0.090 Scale_FM_TravelModel 0.06756 3.31 0.0136 1.63 -0.0076 -0.52 -0.0185 -1.25 -0.1006 -7.36 1 0.068 0.014 -0.008 -0.018 -0.101 Scale_FC_Arterial -0.1495 -5.24 -0.061 -4.86 -0.0621 -5.17 -0.084 -5.96 -0.1163 -5.88 - - - - - - Scale_FC_CollectorLocal -0.2121 -4.03 -0.1114 -4.79 -0.1255 -5.21 -0.2008 -5.78 -0.3214 -2.36 - - - - - - 5th Percentile 50th Percentile 95th Percentile 0.475 0.739 0.830 Contribution to Equation 20th Percentile 0.631 80th Percentile 0.804 Table I-4. Demonstration 1, applying quantile regression.

I-32 Traffic Forecasting Accuracy Assessment Research Figure I-5. Demonstration 1, quantile regression lines. User can change the yellow cells to see the effect. Pseudo R-Squared Coef. t value Coef. t value Coef. t value Coef. t value Coef. t value Values 5th Percentile 20th Percentile 50th Percentile 80th Percentile 95th Percentile (Intercept) - -182.27 -1.77 154.578 3.08 255.551 4.67 287.909 3.94 976.786 4.79 -182 155 256 288 977 AdjustedForecast 0.70464 15.97 0.73181 36.19 0.89089 45.20 1.02667 44.19 1.25361 23.88 0.705 0.732 0.891 1.027 1.254 AdjustedForecast_over30k 0.02375 0.57 0.05735 3.05 -0.0042 -0.22 -0.1902 -8.30 -0.4132 -9.89 Scale_UnemploymentRate_YearProduced -0.0058 -1.41 0.00487 2.77 0.00164 0.87 0.00693 2.76 0.00999 1.87 8 -0.047 0.039 0.013 0.055 0.080 Scale_YearForecastProduced_before2010 -0.0067 -5.64 -0.0051 -5.23 0.00022 0.27 0.00384 3.91 0.00324 2.36 - - - - - - Scale_DiffYear 0.00586 2.81 0.00898 6.68 0.00759 5.62 0.0142 8.23 0.0196 10.50 2 0.012 0.018 0.015 0.028 0.039 Scale_IT_NewRoad 0.09326 4.34 0.00948 1.10 -0.0081 -0.90 -0.036 -1.93 -0.0901 -4.29 - - - - - - Scale_FM_TravelModel 0.06756 3.31 0.0136 1.63 -0.0076 -0.52 -0.0185 -1.25 -0.1006 -7.36 - - - - - - Scale_FC_Arterial -0.1495 -5.24 -0.061 -4.86 -0.0621 -5.17 -0.084 -5.96 -0.1163 -5.88 - - - - - - Scale_FC_CollectorLocal -0.2121 -4.03 -0.1114 -4.79 -0.1255 -5.21 -0.2008 -5.78 -0.3214 -2.36 1 -0.212 -0.111 -0.126 -0.201 -0.321 5th Percentile 50th Percentile 95th Percentile 0.475 0.739 0.830 Contribution to Equation 20th Percentile 0.631 80th Percentile 0.804 Table I-5. Demonstration 2, applying quantile regression.

Using Measured Accuracy to Communicate Uncertainty I-33 Figure I-6. Demonstration 2, quantile regression lines.