Addressing Uncertainty about Future Airport Activity Levels in Airport Decision Making (2012)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

120 Chapter 4 summarizes the state of the practice in incorporat- ing uncertainty into aviation demand forecasting. This appen- dix provides more detailed information from the research that was conducted in this area. The research involved primarily a review of industry publications and scholarly journal articles but also leveraged the collective knowledge and experience of the project team regarding the treatment of uncertainty. In particular, it covers recent applications of risk analysis tech- niques for decision support in the aviation industry. The contents of Appendix D include: • Examples of the use of approaches for incorporating uncer- tainty, which are fairly standard in aviation activity fore- casting: high/low forecasts, what-if analysis, and sensitivity analysis. • Discussion of more advanced, data-driven procedures for incorporating uncertainty into forecasting. • An evaluation of both the standard and advanced method- ologies for addressing uncertainty in forecasting. Examples of the Use of Standard Approaches for Incorporating Uncertainty High/Low Forecasts Roberts Field, City of Redmond, Oregon: Aviation fore- casts for commercial service, air cargo, military service and general aviation are presented in the airport master plan (Coffman Associates, Inc., 2005). In particular, time-series regression analysis (with population, income, and employ- ment as explanatory variables), together with a market share analysis, were used to examine trends in passenger enplane- ments and growth. To account for demand uncertainty, constant and increasing market share scenarios were used for projec- tions between 2008 and 2023. The constant market share scenario assumed that Roberts Field’s share of total U.S. domestic enplanements would remain at its 2003 level of 0.031%. This translates to an additional 118,500 enplane- ments by 2023 (from a base of 181,100 enplanements). A second market share scenario assumed that the airport’s share would increase steadily from 0.031% in 2008 to 0.035% in 2023. This assumption produced an increase of 151,200 enplanements by 2023. Memphis International Airport, Memphis, Tennessee: As part of the airport master plan (Jacobs Consultancy, 2010), high- and low-growth forecasts were developed in addition to a baseline. A combination of time-series regression analy- sis, travel propensity analysis, airline schedule analysis, and professional judgment were used to develop the baseline. The factors driving higher passenger demand included faster than projected growth in population, employment, and per capita income at the local, state, and national levels, and an increase of about 10% in airline services at the airport. These additional assumptions produced an annual growth rate of 2.2% per annum in annual enplanements from 2007 to 2027, 0.3 percentage points higher than the baseline average annual growth rate. Under the low-growth scenario, slower economic growth is assumed and capacity constraints are imposed (reductions of 10% to 15% relative to existing ser- vices). These assumptions produced an annual enplanement growth rate of 1.1%, 0.7 percentage points lower than the baseline average. For air cargo, various carrier-specific assumptions regard- ing frequency of services were used. Averaging the estimates throughout the forecast horizon, the baseline produced an average annual growth rate of 2.2% in terms of freight ton- nage. The high-growth scenario assumed a higher level of international services provided by FedEx and one additional all-cargo airline to begin operations in 2010 (and another in A p p e n d i x d Further Information on Approaches for Incorporating Uncertainty into Demand Forecasting

121 was used to forecast passenger demand between 2005 and 2020. Demand projections developed using this model could account for uncertainty in service growth resulting from additional air carrier operations. In particular, the model was calibrated for weighting seating features (e.g., jets with 56 seats versus 100+ seats) that can affect specific services (e.g., meal offerings, seating comfort) and for con- nections provided by aircraft serving both hub and non- hub markets, as well as by regional jets. The what-if options considered in the plan included operations of reconfigured aircraft and penetrations of new hub and non-hub mar- kets. For example, the scenario that produced the high- range forecast included a total of 26 new flights servicing two new hub markets and five new non-hub markets; the scenario that produced the low-range forecast included only 13 new services. Other options considered in the network scenarios include changes in market shares due to different timing of entry into new markets, or congestion conditions at a competing airport (such as DFW). Sensitivity Analysis Department for Transport (DfT), United Kingdom: The UK DfT produces demand forecasts for air travel at UK airports to inform and monitor long-term strategic air traf- fic policy and wider government policy on climate change. In the past, these forecasts have also been used as inputs into the appraisal of proposed airport developments. At the time of writing, the most recent forecasts available were pro- vided in a January 2009 report entitled “UK Air Passenger Demand and CO2 Forecasts.” These forecasts were devel- oped in two stages. First, unconstrained air travel demand was forecasted using a time-series econometric model with explanatory variables such as national income, exchange rates, or oil prices (the National Air Passenger Demand Model). Second, constrained demand forecasts were pro- duced with an airport choice model that took into account the effect of airport capacity constraints (the National Air Passenger Allocation Model). Forecasts were presented for a central case and under a set of sensitivity test assump- tions. The latter included alternative economic activity trend growth (for GDP, consumer spending, and trade), changes in oil prices, revisions in the structure and rates of government taxes, and changes in the fuel efficiency of new aircraft. The outcomes of the sensitivity analysis were sum- marized in a table providing 2030 demand forecast under the central case and all sensitivity tests, as well as the vari- ance from the central case in both value and percentage. The largest relative change is obtained under the “high-high” oil price test (increase from U.S.$38 to U.S.$136 per barrel by 2030), resulting in a 10% reduction in constrained terminal passenger demand. 2012). This scenario produced a 3% average annual growth rate in tonnage transported. In the low-growth scenario, FedEx was assumed to allocate some air cargo to surface modes (although no estimates are mentioned in the report). It was further assumed that other carriers experience slower service growth (relative to the baseline) and that there is no new all-cargo carrier until 2027. These translated to a 1.7% average annual rate in air cargo tonnage. What-If Analysis European Organization for the Safety of Air Navigation, EUROCONTROL: The EUROCONTROL Statistics and Forecast Service (STATFOR) prepared short-term (2 years), medium-term (7 years), and long-term (through 2030) fore- casts of instrument flight rules (IFR) aircraft movements (EUROCONTROL, 2008 and EUROCONTROL, 2010). The long-term forecast was developed by growing baseline traffic using a model of economic and industry development, taking into account factors related to passenger demand, economic growth, prices, air network structure, and fleet composition. The model was calibrated to provide traffic forecast by O/D pair under four scenarios: 1. Global growth, 2. Business as usual, 3. Regulation and growth, and 4. Fragmenting world. These scenarios were defined by a variety of characteris- tics, including economic conditions, environmental regula- tions, fuel prices, and demographics. An impact analysis was provided as “another way of looking at the importance of the forecast factors.” The what-if options considered in the analysis included much higher oil prices, use of current air- craft fleet, full auctioning of allocations under the Emission Trading Scheme by 2020, no Emission Trading Scheme (zero CO2 costs for aviation), extension of the high-speed rail net- work to all links within 400 km air distance, flat ticket prices, and no aging population. The effects of these options were expressed in terms of passenger demand, and were estimated under all four forecasting scenarios. Dallas Love Field (DAL), City of Dallas, Texas: In the 2000 Airport Impact Analysis/Master Plan, a quality of service index (QSI) model was used to simulate and evalu- ate the effects of service changes on DAL market shares, traffic, and passenger flows by terminal. QSI points were assigned to each service offering (defined in terms of jet equipment type and type of service provided), and market shares were computed based on the share of QSI points in each city-pair market. (This is equivalent to dividing up traffic according to service choices available.) The model

122 pendent variables or occurrence of rare events) are typically not addressed. Prediction intervals can be derived with most statistical software packages; yet, based on our research, they are rarely reported along with point forecasts derived through time- series methods. An important limitation of this general approach to incor- porate uncertainty is that time-series methods are best suited for short-run forecasting. When applied to longer-term hori- zons, both the point forecast and prediction intervals may become unrealistic. Another limitation, as mentioned pre- viously, is that the uncertainty in the independent variables themselves is typically ignored by postulating that the future values of those variables are known. Advanced Data-Driven Procedures for Incorporating Uncertainty into Forecasting Section 4.2 identifies three classes of methods where the incorporation of uncertainty relies exclusively on the analysis of historical data: (1) time-series methods, (2) distribution fitting and simulations, and (3) extrapolation of empirical errors. All three methods—sometimes described as “frequentist”—have been used to some degree for demand forecasting in aviation and other transportation industries. In all three methods, past observed variations in aviation activity are used to specify a probability distribution for future activity. In other words, inferences and forecasts rely on some form of probability distribution for the underlying activity— even though that distribution is not always presented in its entirety. Prediction Intervals from Time-Series Methods Time-series methods are based on the assumption that historical values of the variable of interest have been gen- erated by means of a statistical model, which also holds for the future (Keilman, 2002). These methods include extrapolative methods, which are based solely on iden- tifying data patterns in the variable of interest—such as autoregressive moving average (ARMA) and autoregres- sive integrated moving average (ARIMA) modeling—and explanatory variable methods (or time-series regression analysis), where causal variables are introduced to explain and forecast the variable of interest. The UK DfT’s National Air Passenger Demand Model is an example of a time-series regression model (UK Department for Transport, 2009). Similarly, the FAA mentions the use of regression analysis techniques in the production of its TAFs (FAA, 2010). The Terminal Area Forecast text box describes the FAA’s TAFs in more detail. An application of ARIMA modeling to forecast air transport demand can also be found in Andreoni and Postorino (2006). Most time-series methods recognize the uncertainty asso- ciated with model specification through the inclusion of an error term and stochastic parameter values. By imposing dis- tributional restrictions on the error structure, they allow esti- mation of a prediction interval—an interval in which future individual observations will fall within a certain probability. In other words, the application of time-series methods allows the production of a statistical high and low range and a distribu- tion of demand forecasts around a point forecast or expected value. However, this distribution only reflects uncertainty in the model specification and parameter values—statistical uncertainty. Other forms of uncertainty (e.g., stochastic inde- FAA’s Terminal Area Forecasts The FAA produces and maintains a database of airport-specific annual historic aviation activ- ity, as well as airport-specific demand forecasts known as Terminal Area Forecasts. Included in the TAF database are enplanements, itiner- ant operations (for air carriers, commuters and air taxis, general aviation, and military aircraft), local operations (for civil and military aircraft), and terminal radar approach control (TRACON) operations (for aircraft operations under radar control). As of September 2008, the data included 3,368 FAA towered airports, federal contract towered airports, terminal radar approach control facilities, and non-FAA airports. In developing the passenger demand forecast, the FAA analyzes the historical relationships between airport activity and local and national economic indicators (such as income and employment) and/or aviation industry-specific factors (such as growth of originating and connecting traffic and airfares) using statisti- cal trend analysis. Regression models are then applied to produce the forecast, based on the growth rates and projections of relevant model drivers. As for the hub forecast, additional fac- tors such as seating capacity and load factors of commercial aircraft are included. The forecast for military operations is much less involved. The FAA assumes that activity levels remain constant unless the Department of Defense announces changes in Air Force activity. Similarly, unless

123 tory variables (X) and numerically integrating out their joint densities. To provide a numerical approximation of Feldstein’s formulation, McCullough introduces a semi-parametric boot- strap method, where the bootstrapped forecast error is formed by resampling from a uniform distribution. An additional advantage of McCullough’s approach is that it allows for non- symmetric intervals (McCullough, 1996). The approach is summarized in Figure D-1. Based on our research, there are no applications of this approach in aviation demand forecasting. However, an application to traffic and revenue forecasting for toll roads can be found in Vilain and Muhammad (2009). In this working paper, the authors use statistical approxima- tions to develop prediction intervals for traffic and revenue, based on an econometric model. After identifying the prob- ability distributions that the explanatory variables (includ- ing income, fuel prices, inflation, and population growth) may follow, the authors use Monte Carlo methods to sim- ulate and combine these distributions with the model’s variance–covariance matrix. Additionally, forecast errors are simulated to reflect growing uncertainty with respect to time. The resulting forecasts thus incorporate the sampling error, errors in the explanatory variables, and the random error—the three types of errors outlined in Feldstein’s and McCullough’s papers. Distribution Fitting and Simulation Under this group of methods, a probability distribution is defined on the basis of past growth rates or activity levels, and simulation techniques are used to combine multiple realiza- tions of this distribution over time in order to estimate prob- able growth paths. Bhadra and Schaufele (2007) introduced an application of this method to forecast traffic at the top 50 commercial air- ports in the United States. The process outlined in the paper comprises three steps: 1. Historic annual growth rates of total operations are used to identify a distribution for each airport through goodness-of- fit evaluation tests; 2. Monte Carlo simulations are run to produce the entire dis- tribution of possible growth rates over the forecasting hori- zon, using the distribution function identified in step 1; and 3. The simulated growth rates and associated probabilities are converted into an annual traffic forecast for each airport. Simulation results for Hartsfield-Jackson Atlanta Inter- national Airport are presented in the paper as an illustra- tion. A Gumbel distribution was identified as the best fit to the annual growth rates of total operations at that airport (Bhadra and Schaufele, 2007). Approaches to accounting for uncertainty in the future value of explanatory variables when developing prediction intervals have been explored in seminal papers such as Feldstein (1971) and McCullough (1996). Motivated to overcome the limita- tions of forecasts generated by treating exogenous future val- ues as known constants, Feldstein formulated an analytical solution to derive the forecast error variance (used to produce prediction intervals) with probabilistic explanatory variables. Building on Feldstein’s work, McCullough provided statisti- cal approximations of the variance of the forecast error since Feldstein’s approach was computationally cumbersome, even under the simplest distributional assumptions. Both papers examined forecasting errors when the explan- atory variables themselves are unknown or characterized by some degree of uncertainty. Feldstein shows that the stan- dard error of the forecast is a function of the forecasted values of the explanatory variables (Xˆ), the regression coefficients (bˆ), the covariance matrix of (Xˆbˆ), and the variance of the regression residuals (Feldstein, 1971). Prediction intervals are derived assuming a maximum width (based on Tchebychev inequality), rather than relying on parametric assumptions for the residuals and explana- otherwise specified by local or regional FAA officials, activity levels for all FAA and federal contract towered airports and non-FAA facilities are assumed to be constant. Forecast uncertainties are not incorporated into the TAFs, unlike the national-level FAA Aero- space Forecasts. The Aerospace Forecasts (FAA, 2012) account for uncertainty in income (gross domestic and disposable), consumption, price level (consumer price index), and unemployment in domestic and international aviation activ- ity (aggregate, not airport specific). Baseline estimates of passenger enplanement, available seats, revenue miles, miles flown, number of departures, and nominal passenger yield (per mile per passenger fare) are reported, along with high (optimistic) and low (pessimistic) forecast ranges. Similar to the TAFs, the Aerospace Forecasts are produced using econometric analysis. In addi- tion, the forecasts and assumptions are discussed among and presented to aviation associates and industry experts and staff. The resulting com- ments and suggestions are then incorporated into the analysis.

124 As noted by the authors, although it was assumed that the distributions of annual growth rates are independent, this assumption should be formally tested in future applications, and correlation factors should be introduced where needed. The authors also question the time-invariant property of the distribution of annual growth rates resulting—mechanically— in a widening of the range of probable traffic levels over time. They argue that the proportionality of the uncertainty may instead remain fairly constant. Finally, as traffic growth at each airport is being simulated separately, the method ignores network dependencies (i.e., competition and inter- actions across airports). But the most important limitation of this type of approach is that the sources and the nature of uncertainty remain unknown, making the interpretation of possible outcomes and the use of the forecasts difficult. The MITRE Corporation has been producing simulation- based performance assessments of the National Airspace System (NAS) for the FAA’s Operational Evolution Plan (OEP). The Figure D-1. Formulation of forecast interval (Feldstein, 1971 and McCullough, 1996).

125 2015 assessments described in Baden et al. were based on arrival delays, forecast on the basis of NAS-wide demand and capacity (Baden et al., 2007). Sources for data include individual airport traffic schedules from the Official Airline Guide, non-scheduled traffic from the Air Traffic Activity Database System (ATADS) during known good weather days, and baseline demand fore- cast from the FAA’s TAFs. The 2015 NAS-wide demand simu- lations were developed for good and bad (as a portion of the good) weather scenarios (although not explicitly stated, these are essentially high and low scenarios), together with or without an OEP in place. Each of the four weather/capacity scenarios was simulated based on the 2006 TAF airport-specific baseline forecast, as well as 22 additional forecasts that were generated through variations on individual airport’s growth. The 2006–2015 TAFs for each of the 35 OEP airports were used to establish baseline trends for each airport along the fore- cast horizon. To generate demand growth variations, twenty- two 10-year trend lines were computed based on samples starting from 1976. The differences between the historical data and the trend within their respective periods were extrap- olated to generate 22 different sets of deviations from the TAF baseline trend (which can be interpreted as residuals from a regression line). This error-sampling (or more appropriately, deviation sampling) method incorporated uncertainty under the assumption that historic peaks and troughs in demand are cyclical. The resulting 2015 demand levels were then used to calculate the demand growth for each airport from 2006, which were ultimately input in the simulation model. The model produced a range of 11-min to 18-min annual average delay per flight in 2015 under the OEP, and a wider range of 17 min to 36 min otherwise. The model results sug- gested that OEP not only enhances the NAS performance, it also increases the likelihood that the system will be operating at a predefined efficiency level (in terms of arrival delay minutes). Our research leads us to conclude that, other than the few cases presented herein, distribution fitting and simulation are generally not used in aviation demand forecasting. On the other hand, some applications can be found in demography, and the approach is gaining in popularity in the analysis of project cost and cost escalation uncertainty. Extrapolation of Empirical Errors This general approach consists of developing ranges of possible forecast values based on observed errors from his- torical forecasts. Based on our research, its applications in aviation demand forecasting remain limited. Examples of applications in other sectors are presented in the following. Keilman et al. explore methods to develop probabilistic forecasts of population growth. They explain that a variety of methods, formal or informal, may be used to predict errors for current forecasts on the basis of past errors. They also argue that this general approach is often used, informally, in combination with others to derive population forecasts. Two important problems are identified. First, time series of historical errors are usually short, limiting the applicability of the approach to long-term forecasting. Second, extrapolation is often difficult because errors may have diminished over successive forecast rounds as a result of better forecasting methods (Keilman et al., 2002). An example of application in demography is that con- ducted by the National Research Council, which analyzed the distribution of past errors in population forecasts by the United Nations over two decades and, by way of stochastic simulations, produced predictive intervals for the current UN projections: The approach assumes that the accuracy of current forecasts will be closely related to that of past forecasts. We estimate that a 95-percent prediction interval for world population in 2030 would extend from 7.5 to 8.9 billion, and a similar interval for world population in 2050 would extend from 7.9 to 10.9 billion. The intervals are asymmetric around the UN medium projection of 8.9 billion in 2050. This indicates that, based on the record of previous projections, a greater risk exists of a large under- statement of future world growth than of a large overstatement. (Bongaarts and Bulatao 2000, p.10) Flyvbjerg et al. recommend the use of reference class forecasting to address optimism bias and general uncertainty in demand forecasting for public works (Flyvbjerg et al., 2005). Reference class forecasting for a specific project involves the following steps: 1. Identify a group of past, similar projects—the reference class. 2. Using data from projects within the reference class, estab- lish a probability distribution for the variable of interest (e.g., demand). 3. Compare the specific project with the reference class dis- tribution in order to establish the most likely outcome for the specific project. There are, to our knowledge, no formal applications of reference class forecasting for aviation demand. Applica- tions in the transportation sector include guidance on dealing with optimism bias in project cost estimates for the UK DfT. Another example of error extrapolation methods is Butts and Linton’s Joint Confidence Level approach to correcting optimism bias in project cost and schedule estimates for the National Aeronautics and Space Administration (Butts and Linton, 2009). The approach consists of developing probabil- ity distributions for project costs and schedule based on his- torical project performance. Essentially, a “fat tail” is added to the right side of the distribution to allow for cost or schedule

126 increases due to unknown-unknown events. That adjustment is reduced—along with the probability of cost growth—as the project progresses and more risks are being recognized. Important to this approach is that corrections to the initial cost estimates are applied probabilistically and adjusted over time. As in reference class forecasting, there is no need to identify and forecast the impact of specific events. Or in the words of Flyvbjerg et al.: The outside view is established on the basis of information from a class of similar projects. The outside view does not try to forecast the specific uncertain events that will affect the particu- lar project, but instead places the project in a statistical distribu- tion of outcomes from this class of reference projects. (Flyvbjerg et al., 2005, p.140) As noted earlier, we have found no formal applications of reference class forecasting or similar approaches in avia- tion demand forecasting. But informal uses of extrapolation methods are likely since errors from past predictions may be used to adjust current forecasts. Evaluation of the Approaches for Incorporating Uncertainty into Demand Forecasting The approaches presented in this document are sum- marized in Table D-1. The table includes a brief descrip- tion of the procedure, identifies the specific questions being addressed, and assesses the extent to which the procedure is being used for aviation demand forecasting. Based on Yokum and Armstrong, Table D-2 evaluates each procedure against a set of criteria, defined as follows (Yokum and Armstrong, 1995): 1. Ease of use for airport applications; 2. Ease of interpretation—whether the outcomes of the pro- cedure can be easily understood and interpreted; 3. Flexibility—whether the procedure can be applied to a wide range of conditions and airports; 4. Ability to identify the nature and sources of uncertainty; 5. Ability to consider multiple risks and sources of uncer- tainty in combination; Procedures Brief Description Specific Questions Being Addressed Current Usage In Aviation Transportation St an da rd P ro ce du re s High/low forecasts All assumptions are modified in the same direction to produce an optimistic and a pessimistic forecast. How low (high) could demand fall (rise) if all circumstances turn for the worst (best)? Widely used Widely used What-if analysis The impact of a single event is estimated relative to a baseline, most likely forecast. How will demand be affected by a specific, foreseeable event? Frequently used Frequently used Sensitivity analysis Forecasting assumptions are modified one at a time, in various degrees. How robust are the forecasts? What are the critical variables or risk factors? Sometimes used Frequently used M or e Ad va nc ed P ro ce du re s U se d in F or ec as tin g Prediction intervals from time-series methods An interval reflecting uncertainty in model specification and coefficient values is derived using a formula. How accurate are the demand forecasts given the specific data and model at hand? Sometimes used Sometimes used Distribution fitting and simulation A distribution is fitted to historical growth rates and used to produce probabilistic growth paths through simulation. How likely may alternative demand trajectories be given past, observed variations in annual growth rates? Generally not used Generally not used Extrapolation of errors Past, observed forecasting errors are analyzed and are used to adjust current forecasts. How may future demand deviate from forecasted values, given errors observed in similar settings? Generally not used Gaining in popularity Judgmental methods (e.g., Delphi) Experts are engaged in a formal setting to review and adjust point forecasts and prediction intervals or to help determine the probable value of forecasting assumptions. What is the experts’ view on future aviation demand and associated uncertainties? Sometimes used Generally not used Poor man’s Bayesian analysis Forecasting models and forecasts are adjusted by practitioners based on judgment or prior examples from the literature. How may demand forecasts be adjusted to account for all available evidence when statistical modeling alone performs poorly? Generally not used Sometimes used Risk analysis elicitation or similar Probability distributions are specified for all independent variables and model parameters and combined through simulation techniques. May involve consensus building through stakeholder engagement. What is the likelihood of alternative demand forecasts given perceived uncertainties in the forecasting assumptions? Sometimes used Sometimes used R ar e Ev en ts Scenario planning Critical future uncertainties are identified, and plans are defined accordingly, with no attempt to assign probabilities. What is the best approach to dealing with rare/high-impact events? Sometimes used Generally not used Table D-1. Overview of the approaches for incorporating uncertainty into demand forecasting.

127 6. Ability to assign a probability to different outcomes; 7. Ability to account for, and properly represent, conditional probabilities and correlations between sources of uncer- tainty and risk; 8. Whether the procedure combines objective probability with judgment-based probability or only relies on one or the other; and 9. Whether the procedure can be easily updated in light of new pieces of information or planning decisions. The ratings shown in the table were developed, somewhat subjectively, by the project team. The first three criteria are rated on a scale of 1 to 4 (stars), with 4 representing the easiest or most flexible procedure and 1 the least. Ease of Use for Airport Applications Ease of Interpretation Flexibility Identify Sources of Uncertainty Consider Multiple Risks at Once Assign Probability to Different Outcomes Account for Correlations Between Risks Combine Objective Probability with Judgment Update with New Pieces of Information or Decisions Impact or what-if analysis   Yes No No No Yes No High/low scenarios  Occasionally Yes Occasionally Occasionally Yes No Sensitivity analysis  Yes No No No No No Prediction intervals from time-series methods  No Yes Yes Yes No No Distribution fitting and simulation  No Yes Yes Yes No Requires re-estimation Extrapolation of errors  No Yes Yes Yes Yes No Judgmental methods (e.g., Delphi)  Yes Yes, generally Not precisely Not precisely No Yes (Poor man’s) Bayesian analysis  Yes Yes Yes Yes Yes Yes Quantitative risk analysis and risk analysis elicitation  Yes Yes Yes Yes Yes Yes Scenario planning                     Yes Yes No Yes, indirectly Yes Yes Note: Four stars represent the easiest or most flexible procedure; 1 star represents the least ease or flexibility. Table D-2. Ratings of the approaches for incorporating uncertainty into demand forecasting.