Performance Measures for State Aviation Agencies (2020)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

118 Annotated Review of TRB Resources This review of performance management metrics uses popular academic literature, along with internet searches of transportation agencies, both state and federal, combined with an in-depth search of the National Academies of Sciences database and library. Most of the infor- mation was garnered from the NCHRP guidebooks and syntheses that have been completed on the subject, in conjunction with AASHTO publications. Searches were completed by the research team along with the TRB librarians. The most pertinent literature is discussed here. This literature review examines the definitions of performance management, KPIs, a balanced scorecard (BSC), and ROI. These items are often interchanged with each other when an orga- nization is trying to assign data points and measurements to the inputs and outputs for its particular line of work. Each item is individual in nature, but the sum of the parts is essentially the heart of performance management. This appendix defines the key components and shows examples of federal, state, industry organizations and research efforts depicting the use of individual public performance manage- ment systems. Performance Management System Performance management has grown out of the business world and filtered to the public sector due to a need for accountability, effectiveness, and efficiency in relation to a limited budget. The use of performance indicators that allow decision makers to identify solutions to problems more quickly is significant in terms of financial, social, and environmental planning (Pintea et al., 2015). “If you cannot measure, you cannot control. If you cannot control, you cannot manage. If you cannot manage, you cannot improve and cannot be efficient” (Kuegen and Krahn, 1999 quoted by Albu and Albu, 2003, p. 41). Performance management is defined (by Marr, 2006) as a complex process consisting of: 1. Building a model of performance for each business separately, 2. Data collection, 3. Data analysis, and 4. Extracting and communicating information. Indicators are tools for performance management that should cover a spectrum of all economic entity performance. According to lead academicians (Kaplan and Norton, 1992), performance indicators are divided into two groups: past-oriented (lagging) indicators, which indicate performance as a consequence of past actions, such as the end of a reporting period, or are historical in nature, and future-oriented (leading), which signal future events deter- mined by the past activities (Pintea et al., 2015). A P P E N D I X C

Annotated Review of TRB Resources 119 Parmenter (2007) published a book on KPIs and divided the indicators three ways. 1. Key result indicators (KRIs) – measure results of different perspectives, 2. Performance indicators (PIs) – indicate the way to follow in a certain perspective, and 3. KPIs – indicate actions to improve the performance of the organization. KPIs are a set of measures focused on different aspects of performance that are critical to an organization for its success. Parmenter (2007) defined 7 characteristics of KPIs: 1. Take the form of non-financial measures; 2. Are calculated frequently (daily, weekly); 3. Are brought forward by the management team; 4. Are understood by all employees; 5. Link responsibility of each individual or work team; 6. Have significant impact; 7. Have positive impact (influence remaining performance measures). KPIs are the critical indicator of progress toward the intended result (see Figure C.1). They provide focus for strategic and operational improvement, create an analytical basis for decision making, and focus attention on what matters most (KPI.org, 2018). As Peter Drucker said, “What gets measured gets done” (Marr and Creelman, 2011). The BSC, first introduced by Kaplan and Norton in 1992, made a simple statement: “the financial model of business alone as the primary means of managing performance was no longer adequate. The financial model was useful for determining what happened yesterday but was little use in managing the future development of business” (Marr and Creelman, 2011). The idea for a scorecard was based on a corporation using a “corporate scorecard” to monitor its performance. Kaplan and Norton decided that there should be a balance between financial and non-financial metrics, thus a “balanced scorecard” (Marr and Creelman, 2011). The classic balanced scorecard (see Figure C.2) was developed to capture performance data from four perspectives (one financial and three non-financial): customer, internal process, learning, and innovation/growth. The balanced scorecard supports the organization’s central vision and strategy (Marr and Creelman, 2011). Source: WIlliams, 2011. Figure C.1. Key performance indicators.

120 Performance Measures for State Aviation Agencies The balanced scorecard technique has brought value to the organizations that use it, but it seems to work best when it is clearly tied to the vision and strategy of an organization. Therefore, a natural extension of the performance technique was to add a strategy map (see Figure C.3) and tie the balanced scorecard to a strategic implementation framework (Marr and Creelman, 2011). Norton and Kaplan describe the process as a “strategy-focused organization,” with five distinct focus areas: 1. Translate strategy into operational terms, 2. Align the organization to the strategy, 3. Make strategy everyone’s everyday job, 4. Make strategy a continual process, and 5. Mobilize change through executive leadership. The entire process of performance management encompasses KPIs, which are leading or lagging, internal and externally focused, and are used along with a BSC and an organizational strategy map. The process is linear, and each piece fits into the entire process. Source: Van Vliet, 2014. Figure C.2. Balanced scorecard by Kaplan and Norton.

Annotated Review of TRB Resources 121 The concept of ROI has been used extensively in business organizations for years. This concept continues to grow in the public sector as well. Phillips (2004) reported that many federal, state, and local agencies were beginning to use ROI (see Figure C.4). In a survey conducted of nearly 2,200 public-sector organizations, nearly 523 (24%) respondents said that interest in developing comprehensive ROI measures was well underway (Schell, 2011). Public-sector activities are often difficult to quantify, and the ROI technique can be used to determine if the expenditures and payoffs for programs and initiatives are in alignment. Source: Miyake, n.d. Figure C.3. Balanced scorecard strategy map. Source: Phillips, 2004. Figure C.4. Phillips return on investment model.

122 Performance Measures for State Aviation Agencies There is a continual and increasing call for accountability and transparency—or proof of program efficacy—from the public. Schell (2011) indicates several areas that seem ripe for the use of ROI: • Performance improvement programs, • Training and learning programs, • Change initiatives, • Technology implementation/improvements, • HR programs, and • Organizational development initiatives. With increasing taxpayer pressure to show the use and effectiveness of government funding, ROI programs can be a solid assessment technique for many public-sector activities. Individual Agency Approaches With the previous section providing a general understanding of the components of perfor- mance management, this section will shift to how these processes can be used at the federal and state levels. Examples and articles from transportation agencies and industry groups will also be discussed. Federal Aviation Administration In 2012, the FAA began an agency-wide performance management process that it called a “metrics harmonization effort” because the agency was reporting various metrics by different departments, and each called the process by a different name. In an effort to be uniform, its harmonization effort addressed terminology, methodology, and stewardship across the FAA lines of business. The main purpose was to bring order, consistency, and accuracy to metrics reporting. The FAA has devoted a website to show the agency’s harmonized operational metrics: https://faa.gov/data_research/aviation_data_statistics/operatonal_metrics/. The FAA has concentrated its efforts on metrics reporting for the areas of safety, efficiency, capacity, environment, and cost effectiveness, which also matches the requirements of the 2012 FAA Modernization and Reform Act. The FAA’s matrix depicts the (1) name of the metric; (2) definition of metric; (3) unit of measure; (4) reporting scope; (5) reported values; and (6) historical values of measure. Figure C.5 shows an example of the FAA’s William J. Hughes Technical Center’s balanced scorecard. State-Level DOTs Performance management techniques are used with variation at most state DOTs. The agencies use PMs to assess their strategic plans and to provide accountability. Most common are the use of PMs for state-level programming, which assists agencies with funding levels, strategic investment, and transparency to their stakeholders. The state of Washington DOT provides a state-by-state library to assist in cataloging what type of measurement and report each state DOT uses. This library is an excellent resource for those agencies for learning, reporting, and providing greater accountability to their constituents. The documents available for review include strategic plans, annual reports, fact books, 2012 Recovery Act information, key performance indicators, and state-specific dashboards. The library can be accessed at https://www.wsdot.wa.gov/Accountability/Publications/.

Annotated Review of TRB Resources 123 Industry Association – American Association of State Highway and Transportation Officials AASHTO has dedicated considerable resources and committee time in developing “transpor- tation performance management” programs (TPM) using its pooled fund program. The focus is to research and assess the training and education needs of its members and to “develop and deliver training, and to facilitate the sharing and retention of performance management best practices” (AASHTO, n.d.). AASHTO has developed a website to showcase best practices and collaboration and serve as a repository of performance measurement resources. The existence of the pooled fund dates back more than 20 years, and it is a means for DOT and federal highway offices to combine resources and achieve common goals (AASHTO, n.d.). Under the MAP-21 (Moving Ahead for Progress in the 21st Century), the FHWA established a performance-based approach to the federal highway program. Seven areas were to develop performance measures under this act; STAs, metropolitan planning organizations (MPOs), and public transit providers were required to develop strategies and targets for each performance measure established by the United States Department of Transportation. States were required Source: FAA, 1998. Figure C.5. FAA balanced scorecard, William J. Hughes Technical Center.

124 Performance Measures for State Aviation Agencies to set targets no later than 1 year thereafter (AASHTO, n.d.). The pooled fund supports the following objectives: • Identify gaps in TPM (transportation performance management) knowledge, skills and abilities, • Develop and deliver learning and capacity development resources, • Establish TPM information clearinghouse, and • Support knowledge transfer among pooled fund states. The website is https://www.transportationmanagement.us/. It provides various resources, examples, regulatory requirements, and a research roadmap designed to continue the research needed for its stakeholders. AASHTO – Statewide Planning Website, www.statewideplanning.org A joint endeavor between AASHTO and the FHWA, along with AASHTO’s Committee on Planning, comprised of 50 state members, created a website to share resources and the most current use of technology. The website is an effort to assist one another in deploying best practices across state DOTs. On this website are a “resources” tab and several topical areas of interest, including on performance management, as well as several related publications. Industry Association – Airports Council International – North America (ACI-NA) ACI-NA is one of the airport management industry’s membership groups, and it collects and reports 50 airport KPIs to provide insight into areas such as financial and employee perform- ance, fixed-asset productivity, and airport operations. The 50 KPIs each have detailed statistical tables that provide quantifiable barometers of industry activity for airport managers, analysts, investors, and other aviation stakeholders. The KPIs are derived from an annual survey; in the 2016 fiscal year, 919 airports responded to the survey. ACI-NA’s comprehensive report is available for purchase (https://store.aci.aero/). In review of the table of contents, the report establishes airports by country, the methodology used in the analysis, the actual questionnaire, and the participating airports. The 10 areas that are reported are: 1. Financial performance, 2. Distribution of revenue, 3. Distribution of costs, 4. Productivity and costs, 5. Aeronautical activity, 6. Non-aeronautical activity, 7. Airport capital expenditures (capex), 8. Liquidity and solvency, 9. Profitability, and 10. Employment. The Cooperative Research Programs of the Transportation Research Board Much of the research for the transportation industry takes place through the Transportation Research Board. Within TRB are several cooperative research programs that further knowledge and research for specific areas of the transportation industry. These programs are funded by their federal counterparts, and the research is determined by problem statement submissions

Annotated Review of TRB Resources 125 and awarded by panels of experts in the field. Using the library services at TRB, over sixty reports were found from 2010 through 2018 that were related to this research project. The research team reviewed the publications and has provided an annotated bibliography for the research most closely aligned with determining “state aviation agency metrics.” The research is listed in chronological order beginning in 2010 with NCHRP and followed by ACRP and TCRP. NCHRP Report 666: Target-Setting Methods and Data Management to Support Performance-Based Resource Allocation by Transportation Agencies: Volume I: Research Report, Volume II: Guide for Target-Setting and Data Management (Project 08-70) (Cambridge Systematics, Inc., et al., 2010) In 2010, NCHRP published NCHRP Report 666 in two volumes, in addition to NCHRP Web-Only Document 154. NCHRP Report 666 “describes methods that managers of state depart- ments of transportation and other agencies can use for setting performance targets to achieve multiple objectives and interact with multiple decision-makers and stakeholder groups, and how data management systems within a DOT can support performance-based decision-making” (Cambridge Systematics, Inc., et al., 2010). This report presents a framework and specific guidance for target setting and for ensuring that appropriate data are available to support performance management. Volume I reviews the current state of practice (2009–2010) and describes key elements of a performance-based resource allocation process and associated tools, along with data- management systems and the required institutional support needed. Volume II looks at the target-setting practice and methodology, including the role of forecasting, availability of data, precision, and reliability, combined with agency experience and benefit–cost analysis, as well as stakeholder expectations and perceptions. This research also generated guidance on how to set up an effective target-based performance management system (Cambridge Systematics, Inc., et al., 2010). Volume III is NCHRP Web-Only Document 154, which focuses on the data management systems and institutional relationships to support the system; it also presents case studies of organizations investigated in the research. NCHRP Report 708: A Guidebook for Sustainability Performance Measurement for Transportation Agencies (Zietsman et al., 2011) NCHRP Report 708: A Guidebook for Sustainability Performance Measurement for Trans- portation Agencies provides state DOTs and other transportation agencies with a practical and easy-to-use approach to identify and apply sustainability-related performance measures, some number of which may already be integrated into agency business practices, to produce a new lens through which decision makers can view their agency’s performance. It describes the underlying principles of sustainability as it relates to transportation, possible goals that can be used to address those principles, and performance measures that can be used to address those goals (Zietsman et al., 2011). Aspects of sustainability-related performance measures, including data sources and examples of use, are discussed. A reference compendium of performance measures has also been provided. This guidebook should be of immediate use to those who have mastered the basics of perfor- mance measurement and who are familiar with their own agency’s performance measurement program but who are challenged with providing useful information to agency leadership on how effectively their organization is meeting or can meet sustainability goals. Examples from DOTs, private industry, and Europe illustrate how sustainability can be successfully added to an agency’s extant performance measurement system (Zietsman et al., 2011).

126 Performance Measures for State Aviation Agencies NCHRP Research Results Digest 361: State DOT Public Transportation Performance Measures: State of the Practice and Future Needs [NCHRP Project 20-65 (Task 29)] (Grant et al., 2011) NCHRP Research Results Digest 361: State DOT Public Transportation Performance Measures: State of the Practice and Future Needs reported on the research effort associated with NCHRP Project 20-65 (Task 29). The digest states, “some state DOTs have developed sophisticated agency- wide strategic performance management initiatives that are credited with helping meet the chal- lenges such as managing scarce financial resources more effectively, focusing staff on leadership priorities, and providing the transparency the general public demands today” (Grant et al., 2011). The research team performed a national survey, and two-thirds of state DOTs indicated they used some sort of performance measure. The DOTs desired more accountability, and some of the state legislatures had imposed their own metrics and data collection and reporting requirements. The digest highlights case studies in Florida, Kansas, Minnesota, New Mexico, Virginia, and Washington. Challenges that were found among the survey and interviews were constraints to collecting raw data, the fractured nature of public-transportation decision making, difficulty in establishing appropriate performance measures, and DOT staff and funding shortages to develop and initiate performance measurement systems (Grant et al., 2011). The authors also noted characteristics of good performance measures: • Trackable over time, • Storytelling potential, • Meaningful for types of services measured, • Relation to statewide public transportation goals, and • Available data. In summary, the research found that many state DOTs were “tracking public transportation performance measures to increase accountability to stakeholders, improve management and decision making, and comply with state mandates and federal data requirements” (Grant et al., 2011). SHRP 2 Report S2-L05-RR-2: Guide to Incorporating Reliability Performance Measures into the Transportation Planning and Programming Processes (Cambridge Systematics, 2014) The guide is an easy-to-read explanation aimed at managers and others about how to incor- porate travel time reliability into planning and programming through a collaborative process. The guide introduces the concept of travel time reliability, identifies various reliability measures, explains how to incorporate reliability in policy statements, describes how to evaluate reliability needs and deficiencies, and, finally, offers suggestions on how to incorporate reliability measures into program and project investment decisions (Cambridge Systematics, 2014). NCHRP Report 809: Environmental Performance Measures for State Departments of Transportation (Crossett et al., 2015) This report identifies potential key environmental performance measures that can be inte- grated into a state DOT’s performance management program and provides a practical approach for adding environmental performance considerations to the suite of other performance topics that state DOTs routinely monitor (e.g., pavement conditions). The report establishes relation- ships between agency activities and environmental outcomes, thereby providing a means for

Annotated Review of TRB Resources 127 demonstrating progress and accountability for environmental performance. The report should be of immediate use to DOT executives and senior managers who set performance goals for their agencies, as well as for staff charged with designing and implementing an environmental performance program. A spreadsheet-based Measure Calculation Tool was developed to help state DOTs interested in implementing a project’s chosen performance measures. The tool can be used to record the component data needed to calculate the measures (Crossett et al., 2015). The tool is available for download from the TRB summary webpage at http://www.trb.org/ Main/Blurbs/173012.aspx. Transportation Research Board Conference Proceedings on the Web 18: Transportation Systems Performance Measurement and Data, Summary of the 5th International Conference, June 1–2, 2015 (Turnbull, 2016) The conference was organized by TRB and was sponsored by the FHWA, the FTA, and the Performance Measures Technical Transfer Pooled Fund project. The conference brought together personnel from public agencies, universities, and the private sector to address develop- ing, applying, and delivering performance measures to support transportation decisions. The conference was organized around four broad themes: (1) driving decisions—aligning performance measures to support decisions, (2) tracking the moves—intermodal performance measurement, (3) untangling the data web—using advances in data and technology to support performance measurement, and (4) the state of the practice and opportunities. A plenary session and four breakout sessions were associated with each theme. This document provides full summaries of the plenary session’s presentations and brief summaries of the breakout session presentations (Turnbull, 2016). NCHRP Report 866: Return on Investment in Transportation Asset Management Systems and Practices (Spy Pond Partners, LLC, et al., 2018) NCHRP Report 866 presents a critical review of generally beneficial results state DOTs and other agencies have achieved from implementing computer-based transportation asset management (TAM) systems. The objectives of the research were to (1) assess the experience of selected agencies that have adopted TAM systems in terms of the investment made and returns realized, and (2) develop guidance for estimating ROI for adopting or expanding TAM systems in an agency. Primary products of the report are a description of the research from selected agencies and a methodology to be used for estimating ROI (Spy Pond Partners, LLC, et al., 2018). NCHRP Synthesis 529: How Transportation Agencies Assess the Value of Added Capacity Highway Projects Versus Other Modal Projects and Strategies (Perrin et al., 2018) This synthesis investigates the extent to which state DOTs and MPOs compare different types of transportation improvement strategies and, in cases in which these agencies do undertake such evaluations, explores the reasons for conducting this analysis and the concomitant methods and policies. The synthesis identifies the current state of assessment of added high- way capacity projects versus how the value of different types of transportation projects are calculated to determine the best use of public funds to increase mode choices, reduce conges- tion, improve travel times, improve safety, and efficiently move freight (Perrin et al., 2018). This information will help to quantify the full spectrum of benefits, costs, and economic impacts of transportation improvement strategies.

128 Performance Measures for State Aviation Agencies NCHRP Synthesis 528: Analyzing Data for Measuring Transportation Performance by State DOTs and MPOs (NCHRP Project 20-05/Topic 48-14) (Vandervalk, 2018) This research was commissioned to outline what the current state of practice is with state DOTs and MPOs in response to the Moving Ahead for Progress in the 21st Century (MAP-21) legislation to include performance measurement into the Federal Highway Program. The objective of the federal program is to focus federal funds on the achievement of national goals, increase accountability and transparency, and improve investment decision making through performance-based planning and programming. The objective of the synthesis is “to better understand what data state DOTs and MPOs are using and how they are measuring transporta- tion performance as part of anticipated changes to the federal program or current performance- based programs an agency may have implemented” (Vandervalk, 2018). NCHRP Research Report 920: Management and Use of Data for Transportation Performance Management: Guide for Practitioners (NCHRP Project 08-108) (Harrison et al., 2018) This report provides guidance to improve data utilization in support of transportation perfor- mance management. The NCHRP RFP states, “effective data utilization is increasingly impor- tant to all public and private organizations and institutions. It is critical that all organizations carefully consider how they specify, define, obtain, store, manage, analyze, use, share, present, and communicate data. DOTs and MPOs face numerous demands, opportunities, and chal- lenges regarding data utilization.” ACRP Report 19: Developing an Airport Performance-Measurement System (Infrastructure Management Group, Inc., et al., 2010) and ACRP Report 19A: Resource Guide to Airport Performance Indicators (Hazel et al., 2011) ACRP Project 01-06 and ACRP Project 01-09 collected and presented the most current knowledge and practices from throughout the airport industry on performance measurement, published as ACRP Report 19 and ACRP Report 19A. These reports provide step-by-step guid- ance on how to develop and implement a performance management system for the airport management industry. The reports begin with the theoretical underpinnings to performance measure systems and key terms and definitions used therein. The guidebook chapters present a roadmap for measurement success in three parts, as shown in Figure C.6 from ACRP Report 19, Exhibit I-1-1. Chapter 2 of ACRP Report 19 outlines regional, state, and federal applications of performance measurement systems. ACRP Report 19A: Resource Guide to Airport Performance Indicators explores airport perfor- mance indicators (APIs) for use in benchmarking and performance measurement. These APIs are sorted by functional type and their criticality to the airport strategic plan. More than 800 performance indicators are presented in three main categories: core, key, and other APIs. Core or fundamental indicators are important for overall operation of the airport and of interest to the chief executive officer or governing board. Key or departmental indicators are important for the operations of key airport functions and departments. The remaining “other” indicators are considered useful as secondary departmental unit performance indicators but are not critical to the airport’s overall function (Hazel et al., 2011). The printed version of ACRP Report 19A includes a bound-in CD of the Resource Guide that is identical to the PDF file that is posted online.

Annotated Review of TRB Resources 129 TCRP Report 88: A Guidebook for Developing a Transit Performance- Measurement System (Kittelson & Associates, Inc., et al., 2003) In 2003, TCRP Report 88 was published outlining a rigorous process for determining the most appropriate PMs that can be used by a transit organization. The research developed a process that transit systems could use to initiate a performance measurement program that integrates customer-oriented and community issues. From TCRP Research Results Digest 56: A Summary of TCRP Report 88: This research needed to provide a context, or framework, in which to select and apply appropriate performance measures integral to transit-system decision making. It analyzed dimensions for agency performance, its definition, measurement and interpretation based on agency goals and objectives (Ryus, 2003). Source: Infrastructure Management Group, Inc., et al., 2010. Figure C.6. Exhibit I-1.1 from ACRP Report 19.

130 Performance Measures for State Aviation Agencies The guidebook presents an eight-step process for establishing a performance-measurement program or refining an already-established program: 1. Define organizational/agency goals and objectives; 2. Generate management support; 3. Identify users, stakeholders, and constraints; 4. Select performance measures and develop consensus; 5. Test and implement program; 6. Monitor and report performance; 7. Integrate results into agency decision making; and 8. Review and update the program. There is considerable overlap in each of these steps, which should not be done individually, as each step will influence others (Ryus, 2003). TCRP Research Report 197: Tools for a Sustainable Transit Agency (Gallivan and Hoffman, 2018) The first tool from TCRP Research Report 197 is the Sustainability Routemap, which is an interactive PDF, similar to a website, that guides the user to improve a transit agency’s sustain- ability program through application of change management principles, best practice examples, and references to online tools. A checklist of possible actions is included with the Routemap that helps users prioritize strategies and track progress. The second tool is the S+ROI Calculator, which is an Excel workbook that quantitatively evaluates potential sustainability projects in terms of financial, social, and environmental returns. The workbook is accompanied by two completed examples (Gallivan and Hoffman, 2018). Both tools are available for download from the TRB website at www.trb.org/Publications/ Blurbs/177296.aspx.