National Academies Press: OpenBook

Performance Measures for State Aviation Agencies (2020)

Chapter: Chapter 3 - Considerations in Selecting and Defining a Performance Measure

« Previous: Chapter 2 - Context and Fundamentals
Page 13
Suggested Citation:"Chapter 3 - Considerations in Selecting and Defining a Performance Measure." National Academies of Sciences, Engineering, and Medicine. 2020. Performance Measures for State Aviation Agencies. Washington, DC: The National Academies Press. doi: 10.17226/25975.
×
Page 13
Page 14
Suggested Citation:"Chapter 3 - Considerations in Selecting and Defining a Performance Measure." National Academies of Sciences, Engineering, and Medicine. 2020. Performance Measures for State Aviation Agencies. Washington, DC: The National Academies Press. doi: 10.17226/25975.
×
Page 14
Page 15
Suggested Citation:"Chapter 3 - Considerations in Selecting and Defining a Performance Measure." National Academies of Sciences, Engineering, and Medicine. 2020. Performance Measures for State Aviation Agencies. Washington, DC: The National Academies Press. doi: 10.17226/25975.
×
Page 15
Page 16
Suggested Citation:"Chapter 3 - Considerations in Selecting and Defining a Performance Measure." National Academies of Sciences, Engineering, and Medicine. 2020. Performance Measures for State Aviation Agencies. Washington, DC: The National Academies Press. doi: 10.17226/25975.
×
Page 16
Page 17
Suggested Citation:"Chapter 3 - Considerations in Selecting and Defining a Performance Measure." National Academies of Sciences, Engineering, and Medicine. 2020. Performance Measures for State Aviation Agencies. Washington, DC: The National Academies Press. doi: 10.17226/25975.
×
Page 17
Page 18
Suggested Citation:"Chapter 3 - Considerations in Selecting and Defining a Performance Measure." National Academies of Sciences, Engineering, and Medicine. 2020. Performance Measures for State Aviation Agencies. Washington, DC: The National Academies Press. doi: 10.17226/25975.
×
Page 18

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

13 Considerations in Selecting and Defining a Performance Measure Building on the information in Chapter 2, this chapter focuses on providing guidance on how a state aviation agency may select the various aspects that will define the specifics of the PM to address its individual needs and situation. As previously discussed, no two agencies are necessarily organized in the same way, nor do they have the same needs, type of system, funding structure or options, and so forth. Consequently, each agency will need to evaluate how to select and refine each PM to fit its own parameters. The discussion in this chapter is meant to stimulate thought on a variety of ways an agency may fine-tune its selected PMs; it is not meant to contain an exhaustive list of PMs, nor is it meant to be prescriptive in the method of selection or level of detail. Each agency is encouraged to consider its own needs carefully before selecting individual PMs that align with its mission, vision, goals, or requirements. These are the consideration categories for use when identifying agency-specific PMs: • Connection to agency goals, • Data source, • Defining a specific measure, • Frequency of reporting or assessing a PM, • Reporting method, and • Responsibility to monitor. Each agency has its own statutory requirements, organizational structure, and operational procedure, as well as different airport systems and funding solutions. As such, specific step-by-step evaluations of these categories are not provided. Each agency must assess these considerations within the context of its own state and adjust them accordingly. 3.1 Connection to Agency Goals As discussed in Chapter 2, ideally, most PMs should be established to support the imple- mentation of an agency’s goals, which are derived to support the agency mission and vision. Most often, established agency goals center around broader concepts such as providing a safe system of airports; contributing to the economic vitality of the state; maintaining the existing infrastructure investment; or being good stewards of federal, state, and local funding. (This list is by no means exhaustive but meant to demonstrate that core concepts that come from the goals can then manifest themselves in a variety of specific PMs.) Within each of these four key concepts (safety, economics, preservation of existing infrastructure, and good use of funds), numerous PMs could be developed to measure the success or status of the support provided to accomplish each overall goal. C H A P T E R 3

14 Performance Measures for State Aviation Agencies For example, an agency may have a goal to support a vibrant aviation industry. A specific objective to support this goal could be for agency staff to attend aviation-related events and promote aviation education. The resulting PM may be to track the number of events that the agency attends annually where it promotes its aviation education activities. Maybe the agency sets a target of attending 20 events annually. The agency would then assess if it met its target of 20 events. Not only could it determine if it met the goal, but it could also evaluate trends related to when the events took place or the number of attendees to assess where the most engagement or exposure occurred. This information may influence future decisions about which events should be attended, or if more staffing is necessary to meet demands during a certain time of the year, or whether the target number needs to be changed if the agency attended more or fewer events. This is just one example on how a specific metric may expand to address other issues that may be valuable to measure and report—all to create an efficient and effective overall program. It should be acknowledged that, at times, a PM may not align with a goal. Often this is a result of a directive from outside the agency, such as a legislative directive to report a piece of data or metric that may be of interest. These are referred to as reporting measures (defined in Chapter 2). When this happens, portions of the discussion in this chapter may not be as relevant because the criteria used to define the PM may be beyond the agency’s control. In many instances, these reporting measures target issues that are more statistical in nature or are interesting pieces of information and that the agency itself may not have any ability to control. As an example, some agencies that were interviewed as part of the study indicated that they report the number of annual operations conducted in their state as a PM. In some cases, this is legislatively required. Few agencies have any ability to affect the annual number of aircraft operations that are conducted in their state; therefore, this is a reporting measure that may not tie to a specific goal. 3.2 Data Source An important factor for agencies to consider when developing PMs is where the data will come from to support the evaluation of the PM. Thorough consideration should be given to evaluate if the data are already available and in a format that can be readily used, or if they are to be collected but may need some refinement. If a data source is not readily available, the agency must also consider what other sources might be available for collecting the data. In some instances, a PM may be established, but it may take time to collect fresh or new data to support its evaluation. Specific sources for each PM may vary. Some may be internal to the agency with data it collects or generates from its day-to-day operations. Others may be information that can be obtained from sources such as other departments in the state government or federal sources such as the FAA. Additionally, data can be sourced from system airports via inspections or annual capital improvement program planning meetings. An example of information that can be gathered from an inspection is the number of airports with clear approaches. 3.3 Defining a Specific Measure Defining a specific PM should be done while keeping a few important issues in mind. First, as noted previously, how does this PM tie to the goals of the agency? Then, how will assessing this metric assist in managing the agency or the airport system? Finally, what will or can the agency do with the results?

Considerations in Selecting and Defining a Performance Measure 15 To address these issues, there may need to be additional drill down within the PM that considers different variables or sets different targets based on the individual elements of the PM. Following are several examples of different ways that a PM could be assessed to illustrate differ- ences or nuances in the system. Much like the selection of PMs themselves, this list is not meant to be exhaustive but instead to provide some examples for agencies to consider. Classifications or Categories As noted previously, some airports or some metrics may not be applicable in every PM. Consequently, an agency may wish to consider the use of classifications or categories to further refine the PM. In some cases, a PM may be applicable across the entire system. In others, the PM, while applicable to the entire system, may also benefit from the review of various segments or categories of the system. An agency may summarize the PM by looking at data for commercial service airports compared to general aviation (GA) airports. Some PMs may look at airports across different geographic areas or maybe congressional districts. Another classification might be by facilities with different runway lengths (those under 5,000 feet or those above 5,000 feet) or by the type of approaches they offer. When looking at PMs related to topics such as funding, there might be categories related to different types of federal funding (discretionary, state apportionment, non-primary entitlements, or primary entitlements). The point here is that by drilling down, more useful information can be obtained from the PM than just reporting out a raw total for the system. How focused or specific a PM is, again, a case-by-case decision that the agency should make when defining each PM. Figure 3.1 illustrates how Kansas has used the type of airport for classification when measuring the percentage of National Plan of Integrated Airport Systems (NPIAS) airports with an adopted snow removal plan. This sort of classification can be important, especially for PMs where there may be provisions made to exclude some element of the PM or where the agency does not expect to meet the PM. For example, a PM may be set indicating that the agency wants all airports with paved runways to have a pavement condition index of at least 60. The agency may wish to classify the airports by commercial service and various general aviation categories so that it can have additional detail on the degree of compliance by these various subcategories, which may provide more Source: Burns & McDonnell and CDM Smith, 2016. Graphic generated by Mead & Hunt. 77.3% 90.9% 100.0% 100.0% 22.7% 9.1% C O M M U N I T Y B U S I N E S S R E G I O N A L C O M M E R C I A L S E R V I C E Meets Benchmark Does Not Meet Benchmark Figure 3.1. NPIAS airports with adopted snow removal plan.

16 Performance Measures for State Aviation Agencies detail on where the deficiencies are across the system. Just reporting the overall total of how many airports are meeting the goal of a pavement condition index of 60 may not be helpful. Additionally, if airports without a paved runway are not excluded from the calculations, this may also skew the results. Targets Targets are the benchmarks that the agency is striving to meet for influencing measures. Targets are unsuitable for reporting measures because the agency does not have the ability to control a reporting measure. However, this report recognizes that this ideal is not reality, so it addresses the topic of targets as they are used relative to all PMs. PM targets should be established based on the goals of the agency. This can be as simple as translating desired compliance into a corresponding target. For example, a PM may be established that says 100 percent of system airports should have a current airport layout plan (ALP) on file. If 50 out of 60 airports have a current ALP on file, an agency could report having 83.3 percent compliance with this goal. This would be the report for the PM across the whole system. The target can be further refined to account for different airports in the system. If the 10 airports in the example in the previous paragraph do not have a current ALP because they are very small, privately owned, general aviation airports that have no obligation to maintain an ALP, then the target may be refined to state that of those system airports obligated to maintain an ALP, 100 percent are required to have a current ALP on file. This would then allow the PM to be shown as 100 percent since the 50 required airports comply. For other PMs, the choice of a target can be more complex. Using the example in Chapter 2 regarding instrument approach, an agency that is tracking IAPs has various options for estab- lishing one or more targets for this PM. The simplest option is to set a target for the total number of IAPs in the state. Setting a target for total number of IAPs can be appropriate for an agency that has a goal of right-sizing the facilities under its jurisdiction since the target may be below or above the existing number of IAPs. An agency with a goal of improving access to its airports can set a target for the average number of IAPs per airport that is higher than the current measure, knowing that raising this average will improve the ability to use airports during periods of poor weather. An agency aiming to improve safety may set a target of at least one IAP per system airport. Once that target is achieved, the target may be shifted to at least one IAP per runway end for each system airport. Achieving these targets reduces the reliance on circling approaches, which improves safety. An agency can further refine its target by making use of the system classification of its airports. For example, the agency can set a target of a specific type of IAP based on the airport’s system classification. This allows the agency to tailor the type of approach to the role that airport fulfills in the system. Continuing with this IAP example, other variations of a target for IAPs are possible based on the goals of the agency and the data available. There are numerous options for consideration by the agency when it comes to selecting a target for a PM. Variations of a Target for Instrument Approach Procedures • Total number of IAPs in state • Average number of IAPs per airport across the system • An average of one IAP per runway end • At least one IAP per system airport • At least one IAP per runway end • A specific type of IAP based on the airport’s system classification • Specific cloud ceiling and visibility minimums based on the airport’s system classification

Considerations in Selecting and Defining a Performance Measure 17 The target selected will also have an impact on the reporting of the PM, as explained in the next section. 3.4 Frequency of Reporting or Assessing a PM When selecting a PM, the agency should consider the frequency with which it reports or assesses the PM. Many PMs are best reviewed annually. For example, PMs related to items associated with funding, such as grant closeouts or the amount of money spent on different projects, likely lend themselves to annual assessments, once construction seasons are complete or full fiscal year data are available to review. However, there may be instances where more frequent reviewing and reporting are advantageous. For example, an agency may consider quarterly reporting for project funding drawdowns to assess how the cash flow on specific projects or specific funding streams is being used. This may result in an internal quarterly report that ultimately gets combined into a single, annual, external report. An example that may have a much longer reporting span is PMs related to economic impact. In many instances, agencies commission the development of statewide economic impact studies. Due to the complexity of these studies, they are often done on a rotating basis, with several years or even decades between their execution. In this instance, reporting of the economic impact annually would need to be qualified to say that no new data have been collected or assessed, even though it may still be reported annually. 3.5 Reporting Method This consideration is intended to consider the method in which the PM is reported, with regard to the audience that will use the PM results. For example, there may be some PMs that are strictly developed to guide internal staff and assist in their day-to-day management of the agency. Other PMs may be part of a required annual report to the agency, its governing body, or even the legislative body. Understanding who the PMs are for and how they will be used with regard to the methods of reporting can assist in shaping how the data may be collected, managed, and developed for dissemination. The example PM that focuses only on the internal staff may be reported verbally and weekly at a staff meeting. On the other hand, a PM that is for the legislative body is likely part of a larger report where data must be collected in advance, prepared in a report, printed, and then disseminated, likely on a specific schedule. This illustrates the variability of the reporting methods that might be used or required. 3.6 Responsibility to Monitor When establishing a PM, the agency should also consider who will be responsible for moni- toring the progress of the PM. As noted previously, some PMs may be internal and might be reported in a general manner on a frequent basis, while others may only be reported every few years. Some PMs may be part of a required annual reporting process, while others may be used for internal process improvement. All these factors should be considered when assigning responsibility for monitoring either some or all the PMs. In some agencies, there are entire departments focused on the review and assessment of the PMs, while in other agencies, the responsibility is spread across an array of staff members,

18 Performance Measures for State Aviation Agencies each with different focus areas that may address different PMs. Each agency should address what the best method is to review and monitor its PMs, according to the type and number of PMs and staffing available within the agency. 3.7 Summary The considerations discussed in this chapter should be kept in mind as agencies select, develop, and implement PMs to support their goals. As previously noted, there is no one-size-fits-all approach to the development of individual PMs. There is also no right or wrong way to establish them. The diversity of each agency must be considered when establishing individual PMs. Another consideration is the creation of an overall program to provide monitoring and continuous improvement of the process, refining the PMs as necessary. Keeping these consider- ations in mind while reviewing the summaries of various PMs in Chapter 4 is suggested.

Next: Chapter 4 - Detailed Performance Measure Summaries »
Performance Measures for State Aviation Agencies Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Across the country, state transportation agencies of various modes have been required to incorporate performance measures (PMs) into their core business functions.

With this trend developing nationwide, the TRB Airport Cooperative Research Program's ACRP Research Report 223: Performance Measures for State Aviation Agencies develops PMs for the National Association of State Aviation Officials (NASAO) membership and their respective state aviation agencies.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!