Being Prepared for IROPS: A Business-Planning and Decision-Making Approach (2014)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

44 This chapter describes lessons learned during the development of the decision-making approach and accompanying decision support tool. The focus is on lessons that have applica- bility to users of the guidebook and IRIS, although some avenues for future research are also noted. ACRP Project 10-14 included a substantial outreach component, to ensure that the objec- tives and implementation matched the needs of the airport community. The outreach effort consisted of the following components: • A stakeholder survey distributed electronically to a broad range of airports. • Structured one-on-one telephone interviews with a select group of representative airports. • A demonstration project that fielded a prototype version of IRIS at three airports, including one-on-one familiarization sessions. Feedback received from the outreach effort is the primary source of the conclusions and recommendations described here. The findings also reflect the outcome of internal testing of IRIS. Additional information about the research involved in ACRP Project 10-14 is available on-line at www.trb.org by searching “ACRP Project 10-14.” 6.1 IROPS Business-Planning Needs As part of the survey effort, airports were asked to rate their need for a formal business- planning process for IROPS-related investments. As shown in Table 12, the survey responses confirmed the need for an IROPS business-planning approach. The perceived need varied by airport size, however, with larger airports more likely to express a need for formal methods (see Figure 13). At least in part, this result appears to be due to the tendency of smaller airports to favor low-cost solutions, including for tasks such as coordination and planning activities involving an airport’s own staff. Notice that even for mitigation initiatives that rely entirely on the use of airport staff time, IRIS can be used to prioritize investments. Examples include the development of contingency plans, coordination with airport tenants, planning studies, and negotiations with external providers of equipment such as shuttle buses, food, rental equipment, and so forth. In such cases, the cost associated with the initiative would be the marginal cost of the labor hours used. As an approximation, this cost could be estimated as the average hourly salary multi- plied by the expected number of hours required. The evaluation of effectiveness would be measured as normal and would not differ from the case of an initiative involving construction or acquisition. C H A P T E R 6 Lessons Learned

Lessons Learned 45 6.2 Impact of Funding Availability As described in Chapter 5, for the purposes of funding eligibility, IROPS are considered exceptional conditions. Consequently, IROPS-related investments are generally not eligible for AIP or PFC funding. To study this issue, the ACRP Project 10-14 outreach effort included sur- vey questions on the effect of funding availability on IROPS-related investment decisions. The results are shown in Table 13. Approximately two-thirds of survey recipients acknowledged “Some Effect” or a “Strong Effect” of the availability of AIP or PFC funds. To address this issue, the evaluation of effective- ness in IRIS takes into account the impact of the availability of funding. The category “Strategic Challenges” in the decision hierarchy includes “Funding Availability” as a criterion. For airports Table 12. Survey responses on need for IROPS business- planning approach. Non-Hub/ General Aviation Airport Small Hub Airport Medium Hub Airport Large Hub Airport Total No need 3 1 0 0 4 Little need 13 3 1 2 19 Some need 3 2 3 4 12 Strong need 0 1 0 1 2 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Small/Non Hub Large/Medium Hub Sh ar e of Ai rp or ts in Ca te go ry No/Little Need Some/Strong Need Figure 13. Need for IROPS business-planning approach by airport size. Non-Hub/ General Aviation Airport Small Hub Airport Medium Hub Airport Large Hub Airport Total No effect 2 0 0 1 3 Little effect 7 2 0 0 9 Some effect 6 3 2 4 15 Strong effect 4 2 2 2 10 Table 13. Survey responses on effect of availability of funding.

46 Being Prepared for IROPS: A Business-Planning and Decision-Making Approach that wish to reflect that funding availability has a strong impact, this criterion should be rated as important relative to other criteria in the pairwise evaluation. The model uses the question, “How accessible will funding be for this initiative?” to quantify this effect. The responses for each initiative under consideration are incorporated in the ranking of the IROPS investment portfolio. The extent to which funding availability contributes to the final result is determined by the relative importance assigned to this criterion in the pairwise evaluation. 6.3 Need for an IROPS Investment Decision Support Tool Measuring the perceived need for and value of an IROPS investment decision support tool was of particular interest to this research project. Airport stakeholders’ perceived need for such a tool was assessed both prior to and after the development of IRIS. The survey, which occurred early in ACRP Project 10-14, included a description of the basic features of IRIS. Airports were asked to rate their perceived need for the described decision support tool. The results are summarized in Table 14. The assessment of the demonstration project also included several questions intended to evaluate the perceived value of IRIS. The responses to those questions are shown in Table 15. More than two-thirds of the survey respondents indicated “some need/value” or a “strong need/value” for a decision support tool for IROPS investment analysis. Responses from the structured interviews suggested that having empirical data to support investment decisions could help diminish political and subjective appeals in the decision-making process. The value of the tool was confirmed by the feedback received during the formal assessment of the post- implementation demonstration project. The demonstration project only included three air- ports but all three confirmed the need for the tool. Specifically, the assessment demonstrated that the airports found IRIS to be usable and that the results were pertinent to their IROPS business-planning needs. Non- hub/General Aviation Airport Small Hub Airport Medium Hub Airport Large Hub Airport Total No 2 0 0 0 2 Little need/value 5 2 0 2 9 Some need/value 9 4 3 2 18 Strong need/value 3 1 1 3 8 Table 14. Survey responses on need for/value of decision support tool. Strongly Disagree Disagree Neutral Agree Strongly Agree This tool would add value to the IROPS business-planning needs of my airport. 2 1 The tool provides useful results. 3 Overall I was satisfied with the usability of this tool. 3 I would likely use a tool like this for future IROPS business planning. 2 1 Table 15. Demonstration project feedback—overall value of IRIS.

Lessons Learned 47 6.4 Implementing Pairwise Comparisons A component of AHP and its implementation in this project is the pairwise comparison of evaluation criteria. The purpose of these comparisons is to elicit the user’s preferences to ensure that the decision is weighted toward factors that are deemed important. The advantages of using pairwise comparisons are described in Chapter 3. In summary, their use assists in deriving clear preferences by reducing bias and by providing better differentiation. A challenge with the use of pairwise evaluations is that they must be logically consistent for the AHP methodology to function correctly. Consider, for example, the following three criteria in the category “Tactical Complexity” (TC): • TC1: Disruption level during implementation. • TC2: Execution response time. • TC3: Policy and regulatory compliance. If the user specifies that TC2 is more important than TC1 and that TC3 is more important than TC2, then it can be inferred that TC3 is more important than TC1. If, however, the user specifies in IRIS that TC3 is less important than TC1, the comparisons are not logically consis- tent and an error is generated. In practice, the check for logical consistency is not absolute. Because each pairwise com- parison includes a score indicating relative importance, the validation test in IRIS is based on a consistency ratio computed from these scores. This ratio quantifies the extent to which the pairwise comparisons are logically consistent. The consistency ratio cannot exceed a specific error threshold. If the error threshold is exceeded, the user is prompted to repeat the pairwise evaluation. Even though the pairwise comparison is meant to reveal the user’s real preferences, inter- nal testing and the airport demonstration project showed that errors in logical consistency occurred periodically. If the pairwise comparison involves a relatively small number of criteria, it is a relatively easy task for the user to review the choices and manually correct the incon- sistency. This is the case in the example provided using TC1, TC2, and TC3, which involved only three pairwise comparisons. If the number of criteria is large, however, it can be relatively difficult to scan the choices and identify the error. This is the case for the categories “Strategic Challenges” and “User Benefits,” which require six pairwise comparisons each. To prevent a frustrating user experience, changes to the IRIS decision support tool were implemented during the demonstration phase to improve the validation of the pairwise comparisons. Multiple criteria are compared against each other, so it is not possible to have the software identify which particular pair is causing a logical inconsistency. Instead, a feature was added to allow for an adjustment of the user’s inputs to meet the consistency ratio threshold. This feature is implemented using the “Suggest Values” button. Activating this feature starts an iterative process in which the values are incrementally adjusted until the threshold is met. The user then reviews the suggested values and confirms their use. In addition, the consistency ratio threshold was relaxed slightly to reduce the frequency of logical inconsistencies. Although the changes described in this section were found to improve the user experience, they come at the price of potentially losing fidelity in the assessment of user preferences. For this reason, the “Suggest Values” feature is intended to be used as a last resort. Other methods for improving the pairwise evaluation process should be investigated for future application of pairwise comparisons in this or similar decision support tools.

48 Being Prepared for IROPS: A Business-Planning and Decision-Making Approach 6.5 Evaluating Non-Economic Benefits A distinct strength of the AHP-based methodology presented here is its ability to quantify benefits and compare them against costs without assigning monetary values to the benefits. In investment decisions evaluated using traditional benefit-cost analysis, it is common to have both benefits that can be monetized and those that cannot; however, the latter can usually only be described qualitatively by including a narrative describing the benefits. This is because the benefit-cost analysis depends on comparing monetized benefits against lifecycle costs. Both benefits and costs must be expressed in monetary terms. An example of benefits that generally cannot be monetized is the reduction of greenhouse gas (GHG) emissions. If a proposed enhancement would result in fuel savings over the baseline legacy case, GHG emissions would also be reduced. Although the reduction in the amount of GHG can usually be modeled, there is no FAA-approved methodology for monetizing the resulting benefit. The modeled reduction in GHG could be included as part of the benefits narrative, but it would not affect the calculation of the benefit-cost ratio or the other common metrics used to evaluate the business case. Only the cost of the fuel saved would be included on the benefits side of the benefit-cost ratio. In an AHP-based methodology, however, GHG emissions could be one of the criteria included in the hierarchy. This would allow for a quantitative comparison of the proposed enhancement against the baseline case. It may be possible to incorporate AHP as a supplemental methodol- ogy when conducting benefit-cost analyses, as this may allow for a broader range of criteria to be incorporated in the business case. 6.6 Conclusions The feedback from the airport community obtained during the course of ACRP Project 10-14 confirms the original research need: Demand exists for a business-planning and decision- making approach to prioritize IROPS-related funding. Although best practices have been established for business case analysis, including some airport applications, business planning for IROPS mitigation has not been adequately addressed. This project has resulted in the ini- tial development of such an approach. This approach addresses the special challenges of busi- ness planning for IROPS events, including the combination of potentially severe impacts and unusually high levels of uncertainty. The approach presented in this guidebook draws on decision theory, including the AHP methodology. As implemented in the decision support tool IRIS, this methodology has several strengths: • It allows for rapid business case analyses of IROPS mitigation initiatives. • It requires no background or training in decision theory or business case analysis. • It combines subjective evaluations with objective business case analysis metrics. • It uses pairwise evaluations to break down complex decisions into more manageable ones. • It is able to quantify intangible benefits that usually are included only as a qualitative narrative. At the same time, the development of the approach unearthed several challenges. Notable among these is the implementation of pairwise evaluations in a user interface like the one devel- oped for IRIS. These challenges serve as the starting point for areas suitable for future research. Potential research topics include: • Managing bias and differentiation in decision support techniques that rely on subjective input. • Applying decision theory and related methodologies like AHP to other areas of airport plan- ning and administration.

Lessons Learned 49 • Applying decision theory to supplement traditional business case analysis methodologies, with a focus on AHP’s ability to quantify intangible benefits. • Increasing the utility of the IRIS application without sacrificing the analytical underpinnings of the approach, including improving the process for conducting pairwise comparisons. • Investigating alternative methodologies for IROPS-related business planning. The demonstration project confirms the utility and perceived value of the IRIS decision sup- port tool. IRIS could, however, be evolved and upgraded, using feedback from airports as they begin to use the tool to assist in their investment planning process. Updating the tool would further improve its effectiveness by incorporating knowledge gained from a broader range of users and investment portfolios.