Program Evaluation Report Card Tool for Wildlife Hazard Management Plans: User Guide (2023)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

59   Research Summary, Findings, and Application A P P E N D I X B

60 Program Evaluation Report Card Tool for Wildlife Hazard Management Plans: User Guide 61 1.0 Introduction 61 2.0 Literature Review and Focus Groups 61 2.1 Literature Review 61 2.2 Focus Groups 64 2.3 Interviews with Representatives of Case Study Airports and Industry Experts 69 2.4 System Requirements 70 3.0 Summary and Conclusions 70 3.1 Opportunity to Reduce Site-Specific Risks and Wildlife Hazards 70 3.2 Support Existing Review Processes 71 3.3 Provide Support to GA Airports 71 4.0 Potential for Use Industry-Wide 71 4.1 Increased Opportunities for Engagement 72 4.2 Basis for Collaboration 72 4.3 Getting Beyond the Data 73 References 74 Attachment A Key Takeaways from Focus Group Discussion 77 Attachment B FAA AC 150/5200-38 Evaluation Forms T A B L E O F C O N T E N T S

Research Summary, Findings, and Application 61 1.0 Introduction Following contract initiation, the project team (Team) submitted an amplified work plan (AWP) for review and approval by the project panel (Panel). The AWP identified the research strategy that would be implemented in support of ACRP Project 10-30, “Evaluating the Effectiveness of an Airport Wildlife Hazard Management Program.” In accordance with the Panel-approved AWP, the Team undertook three research tasks that served as project milestones to gather and refine data for incorporation into the development of the PERC Tool: • Conduct Literature Review and Focus Groups (Task 2) • Conduct Case Study Interviews (Task 3) • Identify Tool User Needs and Functionality (Task 4) The Team prepared a white paper/summary report upon the completion of each research task to describe the research process and data/input received and to identify topics for a greater or more refined focus in subsequent projects. The summary report for each task was provided as an appendix to the ACRP Project 10-30 Interim Report (Mead & Hunt 2021b); therefore, the follow- ing discussion does not replicate the material in those reports pertaining to research methods. 2.0 Literature Review and Focus Groups 2.1 Literature Review The Team conducted a literature search that included a detailed review of 72 resource documents from the United States and seven other countries. The resource documents were published by government agencies, non-government aviation organizations, the National Academies of Sciences, Engineering, and Medicine, and peer-reviewed journals and websites. Based upon data obtained from the literature review, existing FAA regulations and guidance pertaining to wildlife hazard management programs and their evaluation, and professional experience, the project team identified six broad topics to address in the subsequent focus group discussions. • Wildlife Hazard Management Plans (WHMPs)/Programs. WHMP/ Program development and review procedures, goals, and definitions of effectiveness. • Regulatory Compliance vs. WHMP/Program Effectiveness. Perceived difference between regulatory compliance and WHMP/Program effective- ness in enhancing safety. • Evaluation Metrics and Key Performance Indicators (KPIs). Metrics currently considered in WHMPs/Program evaluations, KPIs, and partici- pant familiarity with KPIs. • Liability. Role of liability in WHMP development, review, and evaluation. • Outreach and Sustainability. Role of outreach to airport neighbors and agencies and the role of sustainability in evaluating WHMP/Program effectiveness. • New Tools for WHMP/Program Evaluation. Potential audiences, avail- able tools, and useful features that could be incorporated into new tools to evaluate effectiveness. 2.2 Focus Groups The Team conducted a series of six focus group meetings using Zoom. A standard presentation and discussion structure were provided to gain Focus Groups Composition Six focus groups were assembled as described in the AWP: • Group 1 - Regulatory, Resource, and Transportation Agencies • Group 2 - Large-Hub Commercial Service Airports • Group 3 - Medium-Hub, Small- Hub, and Other 139-Certificated Airports • Group 4 - General Aviation Airports and Joint Use Facilities • Group 5 - Wildlife Strike Database Users and Managers • Group 6 - Wildlife Management Practitioners and Stakeholders

62 Program Evaluation Report Card Tool for Wildlife Hazard Management Plans: User Guide input on each topic identified. Attachment A, which is excerpted from the Focus Group Summary included in the Interim Report, summarizes input from all six focus groups. Focus group input influenced subsequent research by identifying key takeaways and topics for greater exploration and subsequent research (see Sections 2.2.1 through 2.2.6). 2.2.1 WHMP/Program Compliance vs. WHMP/Program Effectiveness Compliance with Title 14 of the Code of Federal Regulations (CFR) Part 139.337, “Wildlife Hazard Management,” is required for certificated airports, and federal agency representatives clearly stated that regulatory compliance was associated with program effectiveness. An FAA representative concluded that a WHMP that complied with FAA regulations and was determined to be acceptable to the agency should be effective by definition. In other words, compliance with FAA guidance is the baseline for developing an effective WHMP/Program. Representatives of certificated airports agreed that compliance was a primary goal in WHMP/ Program development and considered compliance to be a measure of effectiveness. However, they agreed the plan must be implemented as described to be effective. Both agency and airport representatives emphasized that compliance and effectiveness are not always synonymous. Effect on Subsequent Research and Tool Development Noncertificated airports are not required to comply with 14 CFR Part 139, and the WHMPs/ Programs associated with these airports may not be based on a Wildlife Hazard Assessment or include all required components identified in 14 CFR Part 139.337. • A discussion of compliance and effectiveness was addressed during case study interviews. • Development of the proposed tool must consider compliance while remaining flexible enough to address the less structured WHMPs/Programs associated with noncertificated airports. • The forthcoming tool must evaluate compliance (for certificated airports) and an airport operator’s success in implementing the measures identified in the plan. • To consider effectiveness, the proposed evaluation tool must address the basic requirements of plan implementation, such as quality and extent of training provided and the availability of necessary resources, including time and people. 2.2.2 Wildlife Strike Data and Program Effectiveness Focus group participants identified strike data as the predominant criterion used to identify and evaluate plan effectiveness. While focus group participants agreed that the FAA’s National Wildlife Strike Database (NWSD) is the most comprehensive dataset available, several represen- tatives expressed caution that the data does not provide a complete picture to evaluate WHMP/ Program effectiveness. • Airport representatives indicated that while NWSD records can be used to identify changes in strike rates over time, they do not consider airport location, climate, unusual events, and other factors that influence wildlife behavior and abundance but cannot be addressed by the WHMP/Program. As a result, comparisons based on strike data alone may be incomplete. • The overall goal of a WHMP/Program, to enhance safety, would be achieved through a demonstrated reduction in strikes or the presence of high-risk species observed on and near the airport. Effect on Subsequent Research and Tool Development Existing WHMP/Program evaluation tools, such as the program evaluation forms in FAA Advisory Circular (AC) 150/5200-38, “Protocol for the Conduct and Review of Wildlife Hazard

Research Summary, Findings, and Application 63 Site Visits, Wildlife Hazard Assessments, and Wildlife Hazard Management Plans,” include an analysis of strike data that is readily available for airport operators who may not maintain strike records on site. • WHMP/Program evaluations must consider strike data from the NWSD or another source. • Strike data must be supplemented by other evaluation factors to provide a complete picture of program effectiveness. 2.2.3 Use of Quantitative and Qualitative Data Participants underscored the necessity to rely on quantitative data to the extent possible to conduct a meaningful evaluation. One representative from the U.S. Department of Agriculture (USDA) commented on the importance of quantitative data as an input to the proposed evalua- tion tool and as output to express WHMP/Program effectiveness. Airport representatives and practitioners identified numerous factors, such as airport location (coastal vs. inland), seasonality, and extreme or unusual weather conditions during the evaluation period (drought vs. extreme storms) that could significantly affect wildlife abundance and the probability of wildlife strikes. They voiced the need to consider such events in an evaluation, not just quantitative data. Effect on Subsequent Research and Tool Development Although strike data is central, the tool needed to provide an opportunity for users to document specific events, circumstances, or factors to be considered during WHMP/Program evaluation. 2.2.4 Risk Assessment/Analysis Most participants agreed that risk assessment was important to WHMP/Program development and evaluation, but were frustrated because most risk assessment discussions focus narrowly on two factors: • Likelihood. The likelihood of strikes based on strike records; and • Severity. The severity or damage potential associated with species/guilds based on published data and strike history. The focus group members identified that other tools or methods to conduct risk assessment have been developed, but they are unlikely to be implemented in the absence of an airport-wide safety management system (SMS) or based on the level of effort required to undertake a more detailed risk assessment/analysis. Effect on Subsequent Research and Tool Development Based on the extensive interest and discussion about risk assessment and analysis, the Team concluded that risk assessment required further industry research. The Team obtained Panel approval to amend its AWP to include discussions with industry experts with applicable experi- ence with wildlife hazard management and risk assessment. Three individuals were identified for expert interviews: • Richard Dolbeer, PhD, U.S. Department of Agriculture, Wildlife Services; • Isabel Metz, PhD, German Aerospace Center; and • Pierre Molina, Falcon Environmental. 2.2.5 Key Performance Indicators Not all participants were familiar with KPIs or the use of KPIs in relation to wildlife hazard management. Some focus group participants were familiar with KPIs as applied to other contexts or disciplines.

64 Program Evaluation Report Card Tool for Wildlife Hazard Management Plans: User Guide Effect on Subsequent Research and Tool Development Most participants agreed that KPIs could prove helpful in providing a useful tool that could be shared with diverse stakeholders. Specific suggestions included: • Providing user guidance that provide clear explanations of the KPIs used and the data/ evaluation result they represent; and • Incorporating dashboards. 2.2.6 Other Topics Two topics did not generate detailed discussion: • Liability. Most participants agreed that while liability is important and underscores the reason and need for WHMP/Program, liability does not have a role in WHMP/Program evaluation. The topic was carried forward for case study discussions. Regulators expressed concern that the community planners and decision makers frequently do not understand the connection between land use, wildlife hazards, and subsequent liability. • Sustainability. Most participants agreed that wildlife hazard management is part of the over- all umbrella of sustainable airport practice; presenting the WHMP/Program as a sustainable practice could make wildlife hazard management more acceptable to stakeholders and the community. 2.3 Interviews with Representatives of Case Study Airports and Industry Experts To support project research, the Team conducted interviews with representatives of 10 airports identified for more detailed case studies. The airports represented included large-, medium-, and small-hub commercial service airports, GA airports, and joint military-civilian use airports. The Team also interviewed five industry experts; three experts were interviewed to consider risk assessment specifically. Additional interviews were conducted with David Warfel, representative of Allianz Global, an aviation insurer, and with Troy Levanen, Maintenance Safety Manager of Alaska Air Group. 2.3.1 Purpose and Topics The Team conducted the case study interviews to collect information related to current views and practices about WHMPs/Programs and their evaluation and to further explore items identified in the focus group discussions. Each interview included a discussion of the following: • WHMP/Program. WHMP/Program description and goals, evaluation process, successes and challenges, and the difference between compliance and effectiveness. • Metrics and KPIs. Specific metrics and KPIs used in WHMP/Program evaluation, including use of the NWSD, the role of risk assessment/analyses, and recommended metrics and KPIs to be included in the PERC Tool. • Liability. The relationship between WHMP/Liability, including the opera- tional and economic impacts of wildlife hazard management efforts. The following sections summarize some of the key topics identified dur- ing the case study and expert interviews. Some topics, such as the role and use of NWSD and risk assessment/analyses are discussed in greater detail because they proved to be topics of heightened interest to airport representa- tives and required further discussion with industry experts. Topics that did Case Study Airports • Portland International Airport (PDX) • Seattle–Tacoma International Airport (SEA) • Charlotte Douglas International Airport (CLT) • Minneapolis–St. Paul International Airport (MSP) • Bob Hope Airport (BUR) • Dane County Regional Airport (MSN) • Crater Lake Regional–Klamath Falls Airport/Kingsley Field (LMT) • Cecil Airport (VQQ) • Teterboro Airport (TEB) • Arlington Municipal Airport (GKY)

Research Summary, Findings, and Application 65 not provide greater insights or build upon data obtained during the focus group summaries are not addressed. As previously noted, the Project 10-30 Interim Report provides a complete summary of case study interviews. 2.3.2 Wildlife Hazard Management Programs A primary goal of the case study interviews was to identify the differences in how diverse airports evaluate their WHMPs/Programs. To do so, participants were asked to identify key personnel involved in WHMP/Program evaluation, WHMP/Program goals, and their definitions of WHMP/Program success and/or effectiveness. Program Goals Airport representatives identified numerous goals for their WHMP/Programs. Their com- ments are summarized in Table 1, which is excerpted from the Interim Report. • To have a safe operating environment and limit attractiveness of birds at airport. • To provide a safe airfield environment for aircraft and the traveling public. Compliance • Meet FAA compliance requirements in 14 CFR Part 139. • Ensure compliance with environmental regulations. Wildlife Strikes • Eliminate strikes (zero strikes). • Reduce the number wildlife strikes. • Reduce the rate of wildlife strikes. • Decrease or stabilize damaging strikes over time. Attractants • Manage or limit wildlife attractants on the airport. • Have the ability to and properly address the hazards on the airport. Wildlife Observations • Have the ability to collect and review data accurately and in a timely fashion. • Reduce the number of wildlife observed, especially high-risk species. • Decrease sightings year over year. • Monitor wildlife activity on and near the airport. Wildlife Control • Prevent additional species from taking up residence at the airport. • Target activities on zero-tolerance species. Off-Airport/Land Use • Keep up with compatible land use planning. • Reduce hazardous wildlife near the airport. Program Operation • Make sure everyone is doing what they are supposed to be doing to support the program and in accordance with the expectations of the Wildlife Coordinator (documentation, tool understanding and use). Education/Outreach • Make sure that management understands the program and the airport's responsibilities for wildlife management. • Have the ability with training, staff, equipment to address hazards (response level). Innovation/Research • Conduct research and development to push forward and provide continuous improvement. Programmatic/ Management • Make sure everyone is doing what they are supposed to be doing to support the program and in accordance with the expectations of the Wildlife Coordinator (documentation, tool understanding and use). • Obtain full support of the wildlife management program and continuity of operations and trained staff. • Ensure adequate systems are in place to implement the program. KPI or Risk Analysis • Implement and track KPIs for future benchmarking in conjunction with the airport-wide SMS. • Identify a visible decrease in potential hazards/risk analysis. Source: Mead & Hunt, Inc. 2021b. Category/Focus Wildlife Hazard Management Program Goals Safety • To increase safety at our airport and make the airfield as safe as possible/minimize risk to aviation. Table 1. Wildlife hazard management goals identified by case study airports.

66 Program Evaluation Report Card Tool for Wildlife Hazard Management Plans: User Guide Evaluation and Metrics The majority of airport representatives used the form provided in FAA AC 150/5200-38, “Protocol for the Conduct and Review of Wildlife Hazard Site Visits, Wildlife Hazard Assessments, and Wildlife Hazard Management Plans,” Appendix F, as the basis of their WHMP/Program evaluation. In accordance with the AC, annual reviews were typically based on strike data summaries with some level of trend analysis, and formal risk assessments were not included in the annual reviews. Compliance and Effectiveness Airport representatives perceived a difference between evaluating a WHMP to determine compliance and evaluating the WHMP to determine program effectiveness. However, represen- tatives of certificated airports indicated that compliance and effectiveness could not be separated, as compliance was the minimum standard for airport management. However, all participants agreed that compliance represented a minimum threshold. Participants offered several examples to clarify the difference between compliance and WHMP/Program effectiveness including, but not limited to: • Management understanding of the WHMP/Program and the airport’s responsibility for wildlife management; • Understanding and implementation of WHMP/Program roles and responsibilities (training); • A decreased in the number of wildlife strike reports and wildlife observed at the airport; and • A reduced need for ongoing actions or reaction to address wildlife hazards. Challenges and Successes Many representatives identified resource limitations as their greatest challenge, such as financial constraints, staff limitations, and staff development/training. Other challenges were associated with addressing off-site hazards and attractants. Conversely, representatives identi- fied successes that addressed those challenges including: • Gaining management support for plans, program needs, and staff development; and • Establishing cooperative agreements with agencies and local jurisdictions to address wildlife management through program support or assistance with off-site attractants. Effect on Subsequent Research and Tool Development The description of WHMP/Programs had a profound effect on Tool development in several ways: • Overall Approach to Tool Development. The Team concluded that the proposed tool would supplement the review procedures provided in AC 150/5200-38 to specifically address and gauge effectiveness of the WHMP/Program. The PERC Tool was not developed to replace the annual WHMP audit form in AC 150/5200-38. • Strike Data. All participants said that they reviewed strike data when evaluating their programs. As such, an evaluation of the number of strikes and strike trends would be important. • Compliance vs. Success. All representatives perceived a difference between program compliance and success. An evaluation of program success must go beyond compliance to consider the ability of airport representatives to implement the items in the WHMP/Program. 2.3.3 Metrics and Key Performance Indicators A central goal of project research was to identify the metrics currently used by wildlife managers, the strengths and weaknesses of that data, and the KPIs that would be most useful for WHMP/Program evaluation. The discussion of metrics and KPIs considered: The PERC Tool was developed to support the WHMP Review procedures and audit form presented AC 150/5200-38, Appendix F.

Research Summary, Findings, and Application 67 • The role of the NWSD in program evaluation and how it is used in conjunction with other data sources; • Other metrics and KPIs used by wildlife coordinators to evaluate their WHMP/Program; • Metrics or KPIs that could be incorporated into the forthcoming tool to evaluate WHMP/ Program effectiveness; and • The role of risk assessment in evaluating WHMP/Program effectiveness. Role and Use of the National Wildlife Strike Database in WHMP/Program Evaluation Case study participants were asked to describe the role of the NWSD in their WHMP/Program evaluation and the relationship between the NWSD and other data sources. The sophistication of in-house reporting systems varied widely, and the degree to which airport representatives relied on the NWSD data varied based on the level of sophistication. Nine of the 10 participants reported that they maintain a separate database to track the strikes that occur at their airports, some of which are linked to the NWSD so that airport staff received notifications when a strike report for their airport is submitted to the NWSD. Some of the airports linked their reporting to location data using GIS-based tools and electronic reporting. One wildlife coordinator asso- ciated with a GA airport reported that the airport had few strikes (one or two per year), and he neither maintained a tracking system nor reviewed the NWSD. Eight of 10 representative said they regularly compared airport-generated strike data with the NWSD and expressed concern that the NWSD data did not accurately or completely reflect the strike records at their airports. Although their WHMP/Program evaluations considered strike data, they did not always defer to the NWSD when reviewing strike data during annual WHMP/Program evaluation. Specific concerns were associated with the following: • Incomplete dataset. While the NWSD is accurate, it is unlikely to provide a complete dataset for the airport. • Strikes on and near the airport. Participants raised concerns about whether strike reports in NWSD may be incorrectly attributed to the airport. For example, not all strikes include altitude data, and it is sometimes difficult to determine whether the strike occurred while en route or at a location that should not be attributed to the airport. Role of Risk Assessment in Program Evaluation Case study participants were asked about the role of risk analysis in their wildlife management programs and WHMP evaluations. Only three of the 10 case study airports currently incorporate formal risk analyses into their wildlife hazard management programs and consider the results of the analyses during annual WHMP/Program review. The seven remaining participants stated that they did not have sufficient resources to maintain the records necessary to support a risk analysis. The consensus among participants was that a risk assessment is not required to achieve compliance. When asked to consider the type of risk assessment or analysis that could be useful for evaluating WHMP/Program effectiveness, participants offered the following: • The relative risk associated with the wildlife species observed on and near the airport. • A risk analysis that would consider the prevalence and mass of species observed on the airfield. • An analysis that can look backward (trends and lagging indicators) and forward (predictive or leading indicators). Available Risk Assessment Method The Panel encouraged the Team to consider risk assessment as a component of the PERC Tool’s development, and to do so, the team interviewed three experts: Richard Dolbeer, PhD, Isabel

68 Program Evaluation Report Card Tool for Wildlife Hazard Management Plans: User Guide Metz, PhD, and Pier Molina, as all were familiar with risk assessment approaches that involved the use of latent and predictive measures. Following the interviews, the Team determined that the use of more sophisticated methodologies that address such variables as bird mass, aircraft fleet mix and engine type, or avian radar, would not be useful for incorporation into the PERC Tool based on the following: • Staff resources may not be available to conduct the risk assessment; • The data required to conduct the risk assessment might not be available to many airport operators, especially those of noncertificated airports; and • Several resources were available already to support risk assessment and analysis efforts. Metrics and KPIs Unlike focus group participants, the majority of airport representatives involved in the case study interviews were familiar with KPIs. One representative, the wildlife coordinator associated with SEA, stated that a KPI based on wildlife strike data had been developed and implemented as part of an airport-wide SMS. The Team asked case study participants to identify measures and metrics that they currently use to evaluate their WHMPs and that could prove valuable in the PERC Tool’s development. Table 2 summarizes the recommended metrics. Effect on Subsequent Research and Tool Development Interviews with representatives of case study airports enabled the Team to consider the following during Tool development: • NWSD Data. While participants agreed that strike data is important to the evaluation, the source of the data may include airport-specific records that may or may not deviate from the NWSD. The PERC Tool should provide an opportunity to include data from one or both sources. • Risk Assessment/Analysis. Although participants considered risk assessment to be optional for program evaluation, several thought it had the opportunity to prove useful. To incorporate Data Source/Type Metric Wildlife Strike Records • Total number of strikes and strike rate (number of strikes/number of operations) • Analysis of strike trends (monthly, seasonally, and annually) • Number and rate of strikes by species (more emphasis on high-risk species) • Year by year comparison of strikes • Annual average of strike rates to serve as a benchmark (e.g., three-year average) • Total number of triggering events /strikes Damaging Strike Records • Number and rate of damaging strikes • Severity of damage Observation Data • Comparison of wildlife strikes and wildlife observations Location Data • Comparison/correlation of wildlife strike data and location WHMP Goals • Identify whether WHMP goals are achieved (qualitative and quantitative) Mass • Weight of birds associated with individual strike events Risk Assessment • Incorporation of risk assessment metrics in the annual evaluation Source: Mead & Hunt, Inc. 2021b. Table 2. Metrics used by case study airports to evaluate Wildlife Hazard Management Plans and Programs.

Research Summary, Findings, and Application 69 risk assessment into WHMP/Program evaluation, the Team would need to provide a simple method that incorporates easily accessible data. • Strike Data. Case study participants identified wildlife strike records as a key metric for the proposed tool, including total strike rates and trends and damaging strike rates and tends. Data identified to supplement the strike record included the use of qualitative data, such as observation, and a correlation of strike and location data. As previously mentioned, strike data would be incorporated into the PERC Tool. 2.3.4 Liability Every case study participant stated that liability remains at the forefront of their daily activities as they implement their WHMPs/Programs; however, the relationship between wildlife hazard management and airport liability in the event of a strike is rarely discussed or raised by other departments or airport leadership. A wildlife coordinator from a commercial-service airport reported that airline Risk Team members inquire about the WHMP and its implementation following a strike, but liability itself does not come up directly. Airport insurers may ask to look at data or wildlife management measures, but their primary concern is to make sure there is a program in place (Mead & Hunt 2021b). Effect on Subsequent Research and Tool Development The Team concluded that although the subject of liability is unlikely to be addressed directly in the forthcoming tool, the inclusion of qualitative KPIs, such as staff training and resources, could be included to identify measures to consider liability. 2.4 System Requirements From the initiation of ACRP Project 10-30, the Team set out to establish a tool that would use standard, readily available software, could be downloaded from any personal computer or laptop, and would not require the purchase of additional software or training. Based on the project research, the team determined that the tool must provide the following features or enable airport managers to evaluate the effectiveness of their WHMPs/Programs in a mean- ingful way. • An option to enable an evaluation of compliance with 14 CFR Part 139.337 for certificated airports, and a separate option to evaluate programs for noncertificated airports; • Easy data entry using drop-down menus and easy input methods; • Opportunities for quantitative and qualitative assessments, to the extent possible, to help airport mangers to evaluate the effectiveness of their wildlife management measures and programs; • A method to evaluate specific WHMP/Program recommendations and measures over time; and • A summary that identifies areas requiring improvement. Using the data obtained during the focus group discussions, case study interviews, and consid- erations identified in Sections 2.1 and 2.2, the Team developed the PERC Tool as a self-reporting evaluation or Report Card that includes: • Evaluation results using an easy-to-understand dashboard with KPIs and graphs to facilitate understanding by a variety of stakeholders, not just wildlife managers; • Opportunities to view trends for multiple reporting periods (up to five previous reporting periods); and • Displays to indicate where improvement is most needed.

70 Program Evaluation Report Card Tool for Wildlife Hazard Management Plans: User Guide 3.0 Summary and Conclusions Using the data obtained during the focus group discussions, case study interviews, and concerns associated with Tool development and functionality, the Team prepared the PERC Tool to offer benefits identified in Sections 3.1 through 3.4. 3.1 Opportunity to Reduce Site-Specific Risks and Wildlife Hazards The PERC Tool provides airport operators and wildlife management staff with an opportu- nity to further evaluate their WHMP/Program and to identify opportunities for continuous improvement in reducing site-specific risks and wildlife hazards. Unlike other tools that only consider compliance, the PERC Tool considers both WHMP/Program compliance, if applicable, and WHMP/Program effectiveness in terms of implementation and changes in the strike record and wildlife presence over time. 3.2 Support Existing Review Processes To assist airport operators and staff, the PERC Tool provides analysis that can supplement or enhance the review processes prescribed by the FAA in AC 150/5200-38, Appendix F (see Attachment B). The following analyses provided by the PERC Tool can support the completion of the FAA forms for both annual WHMP review and WHMP review following a triggering event: • Part 2 – WHMP/Program Implementation Measures (Recommendations), which enables users to evaluate the success in carrying out individual recommendations in the plan, includ- ing such items as dispersal or strike response, and assign an implementation score for each recommendation using a percentage. • Part 3 – Strike Data, which enables an analysis of the number of total strikes/strike rates, damaging strikes/strike rates, and a comparison of each for the current evaluation period and up to five previous reporting periods. • Part 4 – High-Risk/Priority Species and Risk Assessment, which provides the user with an opportunity to evaluate the risk posed by the most hazardous species observed (up to five species). In addition to these specific analyses, the following evaluation topics may provide supporting information to address the “summary of progress and challenges in management of the most significant wildlife attractants and/or habitats on or near the airport” that is necessary for both annual review and review following a triggering event: • Part 5 – Equipment and Resources, which enables users to identify whether they have suf- ficient resources to implement the measures identified in the WHMP/Program. • Part 6 – Staff Training, which asks if the airport staff has received sufficient training to implement the WHMP/Program during the past 12 months or anytime during the past three years. • Part 7 – Managing Off-Site Wildlife Attractants, which identifies whether any off-site wildlife attractants have been identified to affect on-site attractants and the progress in addressing such hazards (the presence of off-site wildlife can affect the ability of airport staff to address on-site wildlife). • Part 8 – Outreach and Education, which addresses the efforts of airport staff to conduct outreach to on-site and off-site stakeholders about wildlife hazard management. • Part 9 – Observation, Events, and Conditions, which provides airport staff with an oppor- tunity to identify conditions or events that could affect the evaluation of plan effectiveness, such as unusual weather events, construction, changes in land use, etc. • Output Summary, which lists specific measures or areas in which improvement is needed.

Research Summary, Findings, and Application 71 3.3 Provide Support to GA Airports Certificated airports are required to comply with 14 CFR Part 139.337, “Wildlife Hazard Management,” but noncertificated airports have no such requirement. Moreover, GA airport operators may have fewer resources available to allocate to WHMP/Program development, implementation, and management. The PERC Tool can assist GA airport staff members in gaining support for wildlife hazard management activities by providing: • Nonregulatory review. The Tool offers a scalable structure for reviewing and maintaining WHMPs/Programs that do not focus primarily on regulatory compliance. • Step-by-step instructions and links to available information. During focus group discus- sions, case studies, and industry review of the PERC Tool, some GA airport staff members stated that they did not have the information available to review their plans as the airport did not maintain strike data, reliable operations data were not readily available, and they did not conduct ongoing monitoring. The PERC User Guide and Resource Guide includes explicit instructions for obtaining strike data from the NWSD, operations data from FAA’s OPSNET website, and opportunities to provide qualitative observation data. • An affordable option. As one GA airport operator mentioned during an interview, “In the GA world, it all comes down to the cost. GAs don’t have the funding or support to support wildlife management. We understand that we need to have better data, but we don’t have the resources to generate that data.” The PERC Tool will be free of charge. It will enable users to gather necessary data, such as strike data and operations data, from publicly available websites, and it will rely on quantitative data for several analyses. • Data for Program Support. The Report Card will generate KPIs pertaining to strike data, risk assessment, and resource needs that are easy to understand. The use of KPIs and graphical representations can enhance opportunities to engage with management and convey the relation- ship among strike history, program implementation, and resource needs. 4.0 Potential for Use Industry-Wide Implementation of the PERC Tool has the potential to enhance and support wildlife hazard management activities and understandings at individual airports and industry-wide. 4.1 Increased Opportunities for Engagement Focus group participants and representatives of case study airports were asked to identify potential stakeholders who would be interested in the forthcoming PERC Tool. The stakeholders identified them, and their interests include: • Executive staff and airport administration • Airport operations and risk managers • Safety/risk teams • Airlines • FAA certification inspectors • State safety inspectors • Tenants • Airport users • Regulatory agencies • Planning agencies • Elected officials and decision makers

72 Program Evaluation Report Card Tool for Wildlife Hazard Management Plans: User Guide 4.2 Basis for Collaboration The Team developed a tool that could be downloaded from the National Academies Press website (nap.nationalacademies.org) by searching for ACRP Research Report 250: Program Eval- uation Report Card Tool for Wildlife Hazard Management Plans: User Guide and saved locally. The Tool was designed so that data is entered into the Tool and the PERC Report Card is not shared with others unless the user chooses to do so. Nevertheless, the PERC Tool offers several opportunities to promote industry-wide collaboration through the following: • Common Approach to WHMP/Program Evaluation Among Diverse Airports. The Tool offers an opportunity for a systematic approach to WHMP/Program evaluation for certificated and noncertificated airports alike, thereby facilitating collaboration among a greater number of airports. • Opportunities to Share Data Among Airports. Although the Tool does not provide for comparisons with other airports, comparisons are helpful. As Richard Dolbeer of USDA (retired) pointed out, “To answer the question of effectives, we need to use the data and compare the results to what’s happening at other airports in the region.” Doing so can allow airports to share recent observational data such as species observed, current and anticipated migration, and changes over time. The PERC Tool allows users to print and distribute their evaluation results, but even if they choose not to do so, Tool distribution provides airport staff members with a Report Card that can serve as a point of departure for subsequent conversations regardless of airport type or size. 4.3 Getting Beyond the Data One of the greatest benefits that the Tool provides is an opportunity to expand program evaluation to consider factors beyond the strike data. FAA evaluation criteria identify an analysis of strike data as a component of program evaluation, and research participants iden- tified strike data as an important metric. As identified by Richard Dolbeer, during an expert interview, “One of the key issues is that airports use the database (i.e., number of strikes) as the only metric . . . rather than as a foundation. It has become the sole source of evaluation” (Mead & Hunt 2021b). The NWSD was developed to provide a scientific foundation for decision-making on the national level, to help guide the development of ACs and regulations pertaining to wildlife strikes and to provide data to aircraft manufactures about strike-related damage so that they could consider the data during aircraft design. At the individual airport level, the database provides a resource to develop, assess, and improve its WHMP/Program by considering the airport-specific strike data and the data associated with other airports in the region. The Team understands that the NWSD or airport-specific data is central to WHMP/Program review, but it sought to address concerns from airport staff members and practitioners that the data provided only a partial picture in describing WHMP/Program success. During the devel- opment of the PERC Tool, the Team sought to incorporate the NWSD data while considering other factors that influence staff ability to implement a WHMP/Program and its subsequent success, such as: • Airport/agency support through the allocation of resources, equipment, and training; • The ability of staff to implement specific measures and recommendations as described in the WHMP/Program; • The effects of off-site wildlife attractants on an airport and staff ’s ability to reach out to owners and regulatory agencies associated with those issues; and • Observations, events, and conditions affecting WHMP/Program efforts.

Research Summary, Findings, and Application 73 References Mead & Hunt, Inc. 2021a. ACRP Project 10-30, Amplified Work Plan. April 30, 2021. Sacramento, California. Mead & Hunt, Inc. 2021b. ACRP Project 10-30, Interim Report. November 23, 2021. Sacramento, California. U.S. Department of Transportation, Federal Aviation Administration. 2018. Advisory Circular 150.5200-38, Protocol for the Conduct and Review of Wildlife Hazard Site Visits, Wildlife Hazard Assessments, and Wildlife Hazard Management Plans. Washington, D.C. Available at: https://www.faa.gov/documentLibrary/media /Advisory_Circular/150-5200-38.pdf

74 Program Evaluation Report Card Tool for Wildlife Hazard Management Plans: User Guide Attachment A - Key Takeaways from Focus Group Discussion Key Takeaways from Focus Group Discussion WHMP Program Goals Participants offered many definitions of WHMP goals, most focused on the need to manage and reduce risk to enhance safety. Risk reduction was generally defined as reducing the presence of wildlife and decreasing the number of strikes. Some variation occurred (see Table 2-5). • Agency representatives, large-commercial service airports, and practitioners placed a greater emphasis on regulatory compliance as the primary goal. • The NWSD administrators and users identified the primary goal as data collection. WHMP/Program Effectiveness Participant perceptions of goals and effectiveness varied, but the characteristics were similar (see Table 2-6). • FAA representatives asserted that if a plan complied with regulations and was acceptable to the agency, it should be effective by definition. Large airport representatives agreed that compliance was a measure of effectiveness. • Airports regardless of size perceived a decreased number of strikes as a measure of effectiveness. • Small, medium, GA, and joint-use airports identified consistent reporting as a measure of success. Compliance vs. Effectiveness An overwhelming majority of participants who responded to the chat question (39 of 42) perceived a difference between regulatory compliance and program effectiveness. • The consensus was that the difference between compliance and effectiveness was associated with implementation. If a plan is not implemented well or consistently, it will not be effective. • From an agency perspective, compliance is considered the baseline for effectiveness; therefore, a scalable tool for determining effectiveness should begin with compliance as a minimum standard for determining effectiveness. • While a plan can be compliant, it is not effective if sightings or ongoing strikes continue following implementation. Metrics and Measures to Evaluate Effectiveness Participants were asked to identify specific metrics, measures and analyses necessary for evaluating WHMP effectiveness. Measures included both quantitative and qualitative data (see Table 2-7): • Strike data (total, damaging, by species, etc.) • Observation/survey data • Data from management actions (actions performed/frequency) • Staff training • Probability/risk analysis data • Data pertaining to wildlife mass • Land use data • Outreach/stakeholder interest Participants provided insight into the strengths and weaknesses associated with available/frequently used data to develop or evaluate WHMPs. The discussion focused specifically on observation data and strike data (see Table 2-8). • When asked what metrics airport managers were interested in, representatives from all airports identified strike data as the primary metric. • Representatives of large commercial-service airports identified strike- related damage and cost as being important to airport leadership. Familiarity with Key Performance Indicators (KPIs) All groups were asked whether they were familiar with or used KPIs as part of their WHMP/s programs. • All representatives of large commercial-service airports and most industry practitioners (7 of 8) were aware of KPIs, but only half of those associated with other groups were familiar with KPIs. • Only the U.S. Navy and half of the practitioners interviewed (4 of 8) included KPIs in their wildlife program evaluations.

Research Summary, Findings, and Application 75 Key Takeaways from Focus Group Discussion Role of Risk Assessment/Analysis Role of Risk Analysis Nearly all participants agreed that risk analysis is important for evaluating the effectiveness of a WHMP/Program. • NWSD identified cost as an important consideration/factor in obtaining the data necessary for risk assessment; however, trend data is necessary for evaluating effectiveness. Risk Assessment Metrics Participants identified several metrics that they considered important for identifying and assessing risks: • Quantitative data • Cost data • Strike data • Species data (highest-risk species, location, abundance, frequency, biomass) • Aircraft operations and type Liability Considerations All participants agreed that liability was important, but they did not believe that it had a role in WHMP/Program evaluation. In short, all believed that “the buck stops at the airport,” although litigation is atypical. Comments included the following: • Communities are not aware of liability and the connection between wildlife and land use decisions. • Liability concerns encourage due diligence in program implementation. Difficult-to-implement measures might not be included in the plan. • One commentor noted, “Liability is where compliance and effectiveness collide.” While having a WHMP can protect the airport operator, it can also make the airport operator vulnerable to questions about its program content and due diligence related to implementation. • Liability means documenting everything (training, education, observations, and actions). Off-Site Management and Community Outreach Participants were asked whether WHMP/Program management measures should extend off airport property. • Most agreed that off-site measures contributed to WHMP effectiveness, and it is necessary to communicate with landowners whose property included attractants. • Seventeen of 22 airports reported that their WHMPs included off-site measures. Participants were asked to consider whether public outreach had a role in developing and evaluating the effectiveness of a WHMP. • Sixteen of 22 airports agreed that community outreach has a role in developing a WHMP and evaluating its effectiveness. Participants identified outreach and education as important to: • Airport community: pilots, fixed-base operators, maintenance, tenants, operations staff, and leadership • Local land use planners and decision makers Role of Sustainability Participants were divided on the role of sustainability and its relationship to WHMP/Programs. • Most agreed that sustainability did not have a role in plan evaluation unless it was related to financial sustainability. • Most agreed that wildlife hazard management was a part of the overall umbrella of sustainable airport practice; presenting the WHMP/Program as a sustainable practice could make wildlife hazard management more acceptable to stakeholders and the community.

76 Program Evaluation Report Card Tool for Wildlife Hazard Management Plans: User Guide Key Takeaways from Focus Group Discussion New Tools and Output for WHMP Evaluation Each focus group was asked to identify specific characteristics or features that would make the proposed tool most useful. Key features included: • Simple and intuitive framework • Query-able • Customizable • Communicate risk • Compliance is a baseline • Punch lists • Provide education about relation between wildlife hazards and risk management for municipalities and others • Dashboards • Easy-to-interpret output • Heat map/spatial information • Enable data sharing among airports • Multiparametric approach to identify areas requiring attention • Multitiered approach (decision makers vs. field staff) • Provide education about the relationship between wildlife hazard management and aviation safety Potential users of the tool and its output that were envisioned by focus group participants included: • Executive staff • Airport management, operations, and senior staff • Safety/risk team • Airport administration • FAA certification inspectors • State safety inspectors • Tenants (including airlines) • Nearby landowners • Permitting agencies (to support permit applications) Source: Mead & Hunt Inc., 2021b.

Research Summary, Findings, and Application 77 Attachment B - FAA AC 150/5200-38 Evaluation Forms

78 Program Evaluation Report Card Tool for Wildlife Hazard Management Plans: User Guide

Abbreviations and acronyms used without denitions in TRB publications: A4A Airlines for America AAAE American Association of Airport Executives AASHO American Association of State Highway Officials AASHTO American Association of State Highway and Transportation Officials ACI–NA Airports Council International–North America ACRP Airport Cooperative Research Program ADA Americans with Disabilities Act APTA American Public Transportation Association ASCE American Society of Civil Engineers ASME American Society of Mechanical Engineers ASTM American Society for Testing and Materials ATA American Trucking Associations CTAA Community Transportation Association of America CTBSSP Commercial Truck and Bus Safety Synthesis Program DHS Department of Homeland Security DOE Department of Energy EPA Environmental Protection Agency FAA Federal Aviation Administration FAST Fixing America’s Surface Transportation Act (2015) FHWA Federal Highway Administration FMCSA Federal Motor Carrier Safety Administration FRA Federal Railroad Administration FTA Federal Transit Administration GHSA Governors Highway Safety Association HMCRP Hazardous Materials Cooperative Research Program IEEE Institute of Electrical and Electronics Engineers ISTEA Intermodal Surface Transportation Efficiency Act of 1991 ITE Institute of Transportation Engineers MAP-21 Moving Ahead for Progress in the 21st Century Act (2012) NASA National Aeronautics and Space Administration NASAO National Association of State Aviation Officials NCFRP National Cooperative Freight Research Program NCHRP National Cooperative Highway Research Program NHTSA National Highway Traffic Safety Administration NTSB National Transportation Safety Board PHMSA Pipeline and Hazardous Materials Safety Administration RITA Research and Innovative Technology Administration SAE Society of Automotive Engineers SAFETEA-LU Safe, Accountable, Flexible, Efficient Transportation Equity Act: A Legacy for Users (2005) TCRP Transit Cooperative Research Program TEA-21 Transportation Equity Act for the 21st Century (1998) TRB Transportation Research Board TSA Transportation Security Administration U.S. DOT United States Department of Transportation

Transportation Research Board 500 Fifth Street, NW Washington, DC 20001 ADDRESS SERVICE REQUESTED ISBN 978-0-309-69856-6 9 7 8 0 3 0 9 6 9 8 5 6 6 9 0 0 0 0