Performance Measures for State Aviation Agencies (2020)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

6 Context and Fundamentals This chapter links PMs to strategic planning, discusses the context of performance measuring by public agencies, provides definitions used in the guidebook, and outlines the general funda- mentals of successfully measuring performance. A more in-depth summary of some of these concepts can be found in Appendix C: Annotated Review of TRB Resources. 2.1 Linking Performance Measures to the Strategic Plan Although this guide focuses on measuring the performance of state aviation agencies, performance measurement has an inherent link to strategic planning (i.e., setting the organization’s mission, vision, and goals). Before measuring performance, the agency ideally should formulate its mission, which is likely established in state law. ACRP Report 20: Strategic Planning in the Airport Industry (Ricondo & Asso- ciates, Inc., et al., 2009) provides guidance on setting a mission statement for an airport that can be adapted to an agency. But in general terms, a mission statement should include a clear, concise declaration of why an organization exists. In addition to a mission statement, the agency should also determine what its vision for the future is. A vision statement focuses on the future and is goal oriented, while a mission statement focuses on the present and is operationally oriented. ACRP Report 20 can also be used as a resource for a vision statement; generally a vision statement should briefly describe what an organization’s aspirations are. Once the mission and vision are established, then goals and objectives can be developed to help achieve them. Strategic planning and performance measuring go hand in hand. Strategic planning looks at how an organization functions and the goals it wants to achieve, while performance measure- ment assesses its success in those areas. Therefore, it cannot be emphasized enough that PMs must be aligned to the agency’s mission, vision, and goals. Measuring things that are outside of an agency’s mission or control provides no value to the agency. If an agency’s mission, vision, goals, and objectives have not been set, stop and set those first before defining or implementing specific PMs. 2.2 Context of Performance Measures Because state aviation agencies often fall within a larger transportation organization that is dominated by highways, they are under increased pressure to measure their performance and justify their relevance. The majority of the public and elected officials use and understand the highway mode of transportation and, therefore, inherently appreciate its value. Unfortunately, C H A P T E R 2 Strategic planning looks at how an organization functions and the goals it wants to achieve, while performance measurement assesses its success in those areas.

Context and Fundamentals 7 this is not necessarily the case for airports outside of their use by the airlines or by large cargo companies like FedEx and UPS. As a result, agency officials are constantly educating, explaining, and justifying their value, funding, capital outlays, and staff. How this is done among agencies varies, but the move toward using PMs to communicate this information is growing. PMs are widely used in business to gauge how well an organization is operating, what result its efforts have produced, and how efficient its processes are. Because of their wide use, PMs have been adopted by the public sector and adapted for use in transportation agencies and, as a result, to assess if their strategies are leading to the correct results (i.e., implementation of their goals and objectives). 2.3 Definitions for Use in This Guidebook PMs come in many forms and are called by many names. Since measuring the performance of an organization is not an exact science, many similar but slightly different definitions can often be found for the same terms, and the terms are often used interchangeably. This can be confusing, as stated in ACRP Report 19: Developing an Airport Performance-Measurement System: While public- and private-sector organizations have attempted to differentiate among the terms, this can confuse employees’ and stakeholders’ understanding of performance terminology and frameworks. The terms “metrics,” “measure,” and “indicator” are commonly used as synonyms. They all are defined as “a standard of measurement” or “an indicator for quantitative comparison.” (Infrastructure Manage- ment Group, Inc., et al., 2010, p. 2) There are many terms that can be applied to measurement of activity within an agency, but the specific term is not really the important issue—the issue is measurement of perfor- mance that is important to the specific agency. There are many terms used in the measure of performance for an organization, with the two most common being “PM” and “key performance indicator” (KPI). Different sources will attempt to differentiate between these terms and offer slightly different meanings, which can be confusing and thereby complicate the whole process. Ultimately, the underlying commonality is that all are a standard of measurement, so regardless of what they are called, it is fundamental that they are measuring something. So, despite all the different names and nuances used by different organizations and businesses, this guidebook will not differentiate between terms. Like ACRP Research Report 190: Common Performance Metrics for Airport Infrastructure and Operational Planning (Bottiger et al., 2018) this guidebook will consider “measure,” “metric,” and “indicator” synonymous. Each agency setting up PMs is unique and distinctive in characteristics, goals, and resources available. Some organizations may call a measure a metric, while others may call it an indicator, and they may define the difference. PMs are organization dependent, and they will vary, so when examples are given in the case studies in Appendix A, the terms used are those that the particular agency used. To simplify things, this report uses the following terminology and defi- nitions, which is not to say that other terminology is wrong. Rather, this terminology is implemented for consistency and ease of understanding. • PM – Any metric, indicator, or measure used to objectively evaluate a parameter of the airport system or agency overseeing the airport system. PMs are divided into the following two groups: – Reporting measure – A PM that the agency generally cannot control or influence, and – Influencing measure – A PM that the agency can control or influence. The specific term that is used is not really the important issue—measurement of performance that is important to the specific agency is.

8 Performance Measures for State Aviation Agencies The following example illustrates each of these terms: Assume that an agency is tasked with reporting the number of instrument approach procedures (IAPs) within the state. According to the previous defini- tions, this would be referred to as a “reporting measure.” The agency has no ability to change the number of IAPs in the state; it is simply reporting the number that exist. If the agency has the authority to commission and decommission navigational aids (NAVAIDs) that would be part of IAPs, then this metric may fall under the “influencing measure” definition, since the agency can increase or decrease the number of IAPs through its control of NAVAIDs. An additional point to keep in mind is that this guidebook reports what agencies conveyed that they were using as PMs. With the variation in terminology among agencies, a broad range of PMs are documented here. In some cases, agencies pointed out that the PMs they used were really just statistics. Agencies drew from various sources to develop a list of reported PMs, with many citing their state aviation system plans. 2.4 Fundamentals for Successful Performance Measures Since no two agencies are alike, their PMs will also differ, but at the most basic level, agencies ideally should consider the following: • Areas that fall within their mission, • Areas that move their agency toward their goals and objectives, and • Areas that help inform their decisions on budgeting and capital development projects. As agencies continue to compete with other more individually used and appreciated modes of transportation (e.g., highways, rail, non- motorized), being able to show the merit and impact of their spending can be invaluable in maintaining or increasing their capital outlays for maintaining and improving aviation infrastructure. If an agency cannot succinctly articulate what it does and how well it does it, it will likely struggle to meet its goals and objectives. While numerous agencies have implemented PM processes, many have not. The reasons most often cited for not implementing measures were limited time, staff, resources, data, and usefulness. Although the saying goes, “If you can’t measure it, you can’t control it,” many agencies do not feel as though they have much control at all in what they do or how they do things. While state aviation system plans identify critical components of the system and their associated activity levels, many of these aspects are totally outside the control of the agency. For example, airport enplanements are directly related to the airline service providers, so when an airline ceases operation and moves to another state, the agency has no control over the action regardless of how much money is spent on the infrastructure at its state airports. Yet, agencies often cite number of annual enplanements as a PM they are tracking even though they generally have no real role in affecting these figures. The challenge for agencies is defining how airport capital funding contributes to measurable goals. For example, if capital funding is spent on a project that is categorized as capacity like a runway extension, Performance Measure Reporting Measure Agency cannot control or influence this metric Influencing Measure Agency can control or influence this metric and may have set a target for it Avoid getting mired in nomenclature— while comprehensive, detailed definitions may assist in describing all the possible nuances of assessing performance, they are not needed for the basics. This guidebook uses “PM” throughout as an all-inclusive term to cover measures, metrics, and indicators, except for the case studies where the individual state has directly determined the usage. If an agency cannot succinctly articulate what it does and how well it does it, it will likely struggle to meet its goals and objectives.

Context and Fundamentals 9 how can that impact actually be measured in a cause-and-effect relationship? It cannot be measured by operations if there is no accurate way to track them (e.g., the airport does not have a control tower). It cannot be measured by fuel sales if the fixed-base operator controls the fuel prices and raises them. The same holds true for a wildlife fencing project. It cannot be measured by fewer deer on the airfield if a neighboring state park institutes an open hunt on deer for population management. On top of this challenge, most airport capital funding comes from the FAA, and the agency may have little control over how it is spent, with much of it being entitlements that airports can use for any eligible purpose. In the end, much of what agencies do is dictated by state legislation or limited by staffing and resources, so little time is spent on trying to determine measurable goals outside of basic processes. As aviation is often the least understood or appreciated mode of trans- portation in a larger transportation network, agencies commonly spend most of their time just getting done what the law requires and have little time to devote to developing measures and targets. However, PMs can be useful, at a minimum, to demonstrate compliance with regulations, educate legislators on agency roles and responsibilities, enhance communication within and outside of the agency, and bring to light process issues that may not have otherwise been as readily visible. 2.5 The Performance Measurement Process The research team heard numerous times throughout the preparation of this guidebook that a performance measurement process needed to be simple for agency officials to implement. Therefore, in the simplest of descriptions, four fundamental steps are necessary for an agency to succeed in developing its performance management system around its mission, goals, and objectives. These steps are shown in Figure 2.1. Figure 2.1. Steps to developing a performance management system.

10 Performance Measures for State Aviation Agencies As an example, assume that an agency wants to address its goal of improving airport system utility and, to do so, it establishes a specific objective of improving all-weather access to the air- port system through instrument approaches. Step 1 The first step of the process of measuring this performance is defin- ing the PM of all-weather access. There are many ways to do this just using data on the number of instrument approach procedures. For instance, the agency could measure the average number of instrument approaches per airport or per runway end. The agency could take a more refined approach and track instrument approaches by category (e.g., non-precision approaches, approaches with vertical guidance, and precision approaches) to provide more detail. To keep this example simple, a basic measure is used of the number of instrument approaches among all the system airports. Once the PM is defined, the agency needs to determine where to obtain the data, and how frequently. Also, the agency needs to assign responsibility for this PM, along with the format and reporting methodology. Obviously, factors such as the size of the airport system and the resources available to the agency will drive these determinations. In this case, assume that the PM is assigned to the agency’s airport planner, who is to gather the data from FAA databases and report it via memo to the agency director every 12 months. Step 2 Once the parameters of the PM are set, the agency can then establish the target for that PM. As described in Chapter 3, various factors affect the establishment of this target. For this example, assume the agency establishes a target of 100 instrument approaches for the airport system. Step 3 The third step involves executing what was planned in Step 1. The agency’s airport planner gathers FAA data on instrument approaches at the system airports and counts the total number at the designated time. The airport planner reports this number to the agency director by memo as directed by the decisions made in Step 1. For this example, the airport planner finds that there are 95 instrument approaches at system airports and reports this to the agency director. Step 4 The fourth and final step is comparing what was measured in Step 3 against the target established in Step 2 and assessing what changes, if any, are necessary. Using the assumed measure of 95 instrument approaches and comparing it to the target of 100 shows that the PM falls short of its target. The initial effort in addressing this is for the agency to investigate why the PM fell short. Are there runways lack- ing instrument approaches that could have an instrument approach? If historical data are available for this PM, the agency may look at the past number of instrument approaches to see if the number has fallen Step 1: Define PM Measure the number of instrument approaches across the entire system Step 2: Create Target 100 instrument approaches for the airport system Step 3: Take Measures Gather FAA data on instrument approaches and count the total number at the designated time Step 4: Manage PM Determine whether the measure is meeting the target and evaluate what, if any, changes need to be made to the defined PM and created target

Context and Fundamentals 11 and why. If it has dropped because some instrument approaches have been lost to obstructions, the agency can explore mitigating those obstructions. In contrast, if the number of instrument approaches has fallen because of the closure of navigation aids (such as the decommissioning of a nondirectional beacon), then the agency will need to consider a different approach. Part of this analysis also involves the agency understanding the target and why it was set at that level. If the target of 100 system instrument approaches was based on a new runway opening with multiple instrument approaches and its opening was delayed, then the agency needs to tailor its response to that situation. New information may indicate that the target of 100 instrument approaches is too aggressive and the agency needs to revise its target for this PM. Another scenario may be that anticipated instrument approaches may not have been imple- mented as planned due to FAA local delays. This may influence the inability to meet the target. Once this analysis is complete, the agency should evaluate options that will enable its PM to reach its target. The agency may consider actions such as increasing the number of instrument approaches available, changing the target, or a combination of both. During this step, the agency should also consider whether any changes are needed in the PM itself—reporting more frequently than every 12 months, refining the details of what is reported, or even adding additional PMs. In this example, the agency may decide that simply knowing the number of instrument approaches is not sufficient and that additional PMs are needed to adequately evaluate the airport system. This example illustrates the major elements undertaken for each of the four steps shown in Figure 2.1. Additional details on implementing successful PMs are provided in subsequent chapters of this guidebook. Specifically, more details on defining PMs can be found in Chapter 3: Considerations in Selecting and Defining a Performance Measure. As stated previously, it cannot be emphasized enough that PMs must be aligned to the agency’s strategic planning. Measuring things that are outside of the agency mission or ability to control has limited value. Examples of how measures can easily be related to strategic planning are shown in Table 2.1. The most important step in measuring performance is to start. Implementing anything new is always a challenge whether in a large or small organization. If the agency has no PMs in place, it should start small and work up to additional PMs. If the agency already has PMs, it should review Chapter 4 and Appendix A to find others that may be of value to add; however, this is not to suggest agencies should add PMs just for the sake of having more. Careful thought should be given to why PMs are added and what the benefit of including them in the greater performance measurement process will be. Careful thought should be given to why PMs are added and what the benefit of including them in the greater performance measurement process will be. Strategic Plan Element Measure Example Mission Plan, build, and maintain a superior transportation system Number of airports passing minimum safety standards inspection Goal Facilitate the timely development of airports Issuance of state matching grants within 30 days of federal grant issuance Objective Provide grants for airport improve-ment projects Number of airport improvement grants issued versus needs Table 2.1. Examples of how performance measures can relate to strategic planning.

12 Performance Measures for State Aviation Agencies When developing a performance management system, care should be taken not to include too many lagging measures. A lagging measure is one that tracks past performance, so the agency must react after the target has been missed, and the damage is done. The time it takes to close out a grant is an example of a lagging measure. Lagging mea- sures are measured after the fact and are easy to identify, so they are used more often. Leading measures predict future performance and are inherently harder to identify. However, tracking leading measures allows the agency to foresee potential problems before they occur. For instance, tracking employee recurrent training could prevent accidents. Many agencies perform airport inspec- tions and drive to and on the airport. Tracking training for safe driving on the airport surface could prevent an on-airport collision. With these four steps in mind to establish or enhance a performance measurement system, the following chapters will provide ideas on the areas that may be applicable to measure. Because each agency is unique and a product of its state government, how the guidance and examples apply to each agency is relative to its purpose for existing. When deciding what to measure, what is most important is that it be related to the agency’s mission, goals, and objectives. 2.6 Summary The topic of performance measurement is not new; in fact, the following represent recent studies related to performance measuring: • ACRP Report 19: Developing an Airport Performance-Measurement System (Infrastructure Management Group, Inc., et al., 2010) • ACRP Report 19A: Resource Guide to Airport Performance Indicators (Hazel et al., 2011) • ACRP Research Report 190: Common Performance Metrics for Airport Infrastructure and Opera- tional Planning (Bottiger et al., 2018) • NCHRP Synthesis 528: Analyzing Data for Measuring Transportation Performance by State DOTS and MPOs (Vandervalk, 2018) These reports are worthy of review for the concepts and processes they discuss that may be useful for agencies, even though the primary focus of the reports is not state aviation agencies. The goal of this guidebook is not to repeat the contents of those reports, but to supplement and simplify information based on the needs of agencies. Although leading measures are inherently harder to identify, they allow the agency to foresee potential problems before they occur.