National Academies Press: OpenBook

Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes (2022)

Chapter: 4 Data Sources and Analysis Processes for Identifying Emerging Trends

« Previous: 3 Safety Culture and Its Assessment
Page 47
Suggested Citation:"4 Data Sources and Analysis Processes for Identifying Emerging Trends." National Academies of Sciences, Engineering, and Medicine. 2022. Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes. Washington, DC: The National Academies Press. doi: 10.17226/26673.
×
Page 47
Page 48
Suggested Citation:"4 Data Sources and Analysis Processes for Identifying Emerging Trends." National Academies of Sciences, Engineering, and Medicine. 2022. Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes. Washington, DC: The National Academies Press. doi: 10.17226/26673.
×
Page 48
Page 49
Suggested Citation:"4 Data Sources and Analysis Processes for Identifying Emerging Trends." National Academies of Sciences, Engineering, and Medicine. 2022. Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes. Washington, DC: The National Academies Press. doi: 10.17226/26673.
×
Page 49
Page 50
Suggested Citation:"4 Data Sources and Analysis Processes for Identifying Emerging Trends." National Academies of Sciences, Engineering, and Medicine. 2022. Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes. Washington, DC: The National Academies Press. doi: 10.17226/26673.
×
Page 50
Page 51
Suggested Citation:"4 Data Sources and Analysis Processes for Identifying Emerging Trends." National Academies of Sciences, Engineering, and Medicine. 2022. Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes. Washington, DC: The National Academies Press. doi: 10.17226/26673.
×
Page 51
Page 52
Suggested Citation:"4 Data Sources and Analysis Processes for Identifying Emerging Trends." National Academies of Sciences, Engineering, and Medicine. 2022. Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes. Washington, DC: The National Academies Press. doi: 10.17226/26673.
×
Page 52
Page 53
Suggested Citation:"4 Data Sources and Analysis Processes for Identifying Emerging Trends." National Academies of Sciences, Engineering, and Medicine. 2022. Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes. Washington, DC: The National Academies Press. doi: 10.17226/26673.
×
Page 53
Page 54
Suggested Citation:"4 Data Sources and Analysis Processes for Identifying Emerging Trends." National Academies of Sciences, Engineering, and Medicine. 2022. Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes. Washington, DC: The National Academies Press. doi: 10.17226/26673.
×
Page 54
Page 55
Suggested Citation:"4 Data Sources and Analysis Processes for Identifying Emerging Trends." National Academies of Sciences, Engineering, and Medicine. 2022. Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes. Washington, DC: The National Academies Press. doi: 10.17226/26673.
×
Page 55
Page 56
Suggested Citation:"4 Data Sources and Analysis Processes for Identifying Emerging Trends." National Academies of Sciences, Engineering, and Medicine. 2022. Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes. Washington, DC: The National Academies Press. doi: 10.17226/26673.
×
Page 56
Page 57
Suggested Citation:"4 Data Sources and Analysis Processes for Identifying Emerging Trends." National Academies of Sciences, Engineering, and Medicine. 2022. Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes. Washington, DC: The National Academies Press. doi: 10.17226/26673.
×
Page 57
Page 58
Suggested Citation:"4 Data Sources and Analysis Processes for Identifying Emerging Trends." National Academies of Sciences, Engineering, and Medicine. 2022. Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes. Washington, DC: The National Academies Press. doi: 10.17226/26673.
×
Page 58
Page 59
Suggested Citation:"4 Data Sources and Analysis Processes for Identifying Emerging Trends." National Academies of Sciences, Engineering, and Medicine. 2022. Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes. Washington, DC: The National Academies Press. doi: 10.17226/26673.
×
Page 59
Page 60
Suggested Citation:"4 Data Sources and Analysis Processes for Identifying Emerging Trends." National Academies of Sciences, Engineering, and Medicine. 2022. Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes. Washington, DC: The National Academies Press. doi: 10.17226/26673.
×
Page 60
Page 61
Suggested Citation:"4 Data Sources and Analysis Processes for Identifying Emerging Trends." National Academies of Sciences, Engineering, and Medicine. 2022. Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes. Washington, DC: The National Academies Press. doi: 10.17226/26673.
×
Page 61
Page 62
Suggested Citation:"4 Data Sources and Analysis Processes for Identifying Emerging Trends." National Academies of Sciences, Engineering, and Medicine. 2022. Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes. Washington, DC: The National Academies Press. doi: 10.17226/26673.
×
Page 62
Page 63
Suggested Citation:"4 Data Sources and Analysis Processes for Identifying Emerging Trends." National Academies of Sciences, Engineering, and Medicine. 2022. Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes. Washington, DC: The National Academies Press. doi: 10.17226/26673.
×
Page 63
Page 64
Suggested Citation:"4 Data Sources and Analysis Processes for Identifying Emerging Trends." National Academies of Sciences, Engineering, and Medicine. 2022. Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes. Washington, DC: The National Academies Press. doi: 10.17226/26673.
×
Page 64
Page 65
Suggested Citation:"4 Data Sources and Analysis Processes for Identifying Emerging Trends." National Academies of Sciences, Engineering, and Medicine. 2022. Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes. Washington, DC: The National Academies Press. doi: 10.17226/26673.
×
Page 65
Page 66
Suggested Citation:"4 Data Sources and Analysis Processes for Identifying Emerging Trends." National Academies of Sciences, Engineering, and Medicine. 2022. Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes. Washington, DC: The National Academies Press. doi: 10.17226/26673.
×
Page 66
Page 67
Suggested Citation:"4 Data Sources and Analysis Processes for Identifying Emerging Trends." National Academies of Sciences, Engineering, and Medicine. 2022. Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes. Washington, DC: The National Academies Press. doi: 10.17226/26673.
×
Page 67
Page 68
Suggested Citation:"4 Data Sources and Analysis Processes for Identifying Emerging Trends." National Academies of Sciences, Engineering, and Medicine. 2022. Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes. Washington, DC: The National Academies Press. doi: 10.17226/26673.
×
Page 68
Page 69
Suggested Citation:"4 Data Sources and Analysis Processes for Identifying Emerging Trends." National Academies of Sciences, Engineering, and Medicine. 2022. Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes. Washington, DC: The National Academies Press. doi: 10.17226/26673.
×
Page 69
Page 70
Suggested Citation:"4 Data Sources and Analysis Processes for Identifying Emerging Trends." National Academies of Sciences, Engineering, and Medicine. 2022. Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes. Washington, DC: The National Academies Press. doi: 10.17226/26673.
×
Page 70
Page 71
Suggested Citation:"4 Data Sources and Analysis Processes for Identifying Emerging Trends." National Academies of Sciences, Engineering, and Medicine. 2022. Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes. Washington, DC: The National Academies Press. doi: 10.17226/26673.
×
Page 71
Page 72
Suggested Citation:"4 Data Sources and Analysis Processes for Identifying Emerging Trends." National Academies of Sciences, Engineering, and Medicine. 2022. Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes. Washington, DC: The National Academies Press. doi: 10.17226/26673.
×
Page 72
Page 73
Suggested Citation:"4 Data Sources and Analysis Processes for Identifying Emerging Trends." National Academies of Sciences, Engineering, and Medicine. 2022. Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes. Washington, DC: The National Academies Press. doi: 10.17226/26673.
×
Page 73
Page 74
Suggested Citation:"4 Data Sources and Analysis Processes for Identifying Emerging Trends." National Academies of Sciences, Engineering, and Medicine. 2022. Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes. Washington, DC: The National Academies Press. doi: 10.17226/26673.
×
Page 74
Page 75
Suggested Citation:"4 Data Sources and Analysis Processes for Identifying Emerging Trends." National Academies of Sciences, Engineering, and Medicine. 2022. Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes. Washington, DC: The National Academies Press. doi: 10.17226/26673.
×
Page 75
Page 76
Suggested Citation:"4 Data Sources and Analysis Processes for Identifying Emerging Trends." National Academies of Sciences, Engineering, and Medicine. 2022. Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes. Washington, DC: The National Academies Press. doi: 10.17226/26673.
×
Page 76

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

47 Part of the committee’s Statement of Task calls on it to: Review data and analyses of all relevant sources of information, such as operational data being used by the Federal Aviation Administration (FAA) and the air transport industry to monitor for potential safety concerns; government and industry voluntary aviation safety reporting systems; FAA’s annual safety culture assessment; and other sources the commit- tee deems appropriate, including National Transportation Safety Board acci dent investigations; FAA investigations of accidents and incidents; air carrier incidents and safety indicators; and international investigations of accidents and incidents, including information from foreign authorities and the International Civil Aviation Organization (ICAO). The committee will assess whether these available sources of information are being ana- lyzed in ways that can help identify emerging safety risks as the aviation system evolves and whether other information should be collected and analyzed for this purpose, such as data on accident precursors. In this chapter we identify data that could serve as potential indica- tors of emerging trends and begin our review of how the data are being analyzed. The first section offers a review of the basic sources of potential precursor data in use by the commercial aviation system in the United States. The second section provides an overview of how such data are being analyzed by a range of entities in the United States, alone and col- laboratively, to identify emerging safety trends. The chapter concludes with observations that could strengthen these processes, especially for the principal public–private collaborations. Subsequent reports will evaluate 4 Data Sources and Analysis Processes for Identifying Emerging Trends

48 EMERGING HAZARDS IN COMMERCIAL AVIATION—REPORT 1 in greater depth how data can and should be further used to search for emerging safety trends and examine data sources and analysis processes in use in other countries and by international aviation safety organizations. DATA TYPES AND SOURCES In this section we describe some of the most important sources and types of data supporting analysis of emerging trends in aviation safety for the immediate, intermediate, and future time frames. This section comprises the sources the committee believes represent the baseline for current safety analyses and the data sources briefed to the committee. Voluntary Safety Reporting Programs The Voluntary Safety Reporting Programs (VSRPs) described in this section are fundamental to maintaining the high level of safety that commercial aviation has attained in the United States. These systems provide processes for employees, and sometimes others, to report near misses or safety con- cerns. To facilitate reporting, the identities of reporters are confidential and are non-punitive, as elaborated on below. Reports from the front line are essential as leading indicators for determining whether controls put in place are working as they should and that emerging hazards are recognized and addressed. These reports typically include a check-box section and a narra- tive about an incident that may involve a mistaken action or violation of a procedure or policy or other control failure. The check-box section provides a categorization of the type of incident and the narrative provides context for understanding what happened and allows inferences to be drawn about why. Thus, VSRP data are particularly important for insight into human factors issues such as fatigue, human–system interfaces with design and automation, crew interactions, reasons for non-compliance to procedures and policies, and communications among flight crews and with air traffic controllers. VSRP reports can also identify issues with runway marking or lighting, air traffic procedures, unusual weather events or turbulence, and the technologies that the system depends on. Fundamental to making VSRPs work is a guarantee to the people submitting the reports that (a) their confidentiality will be ensured and (b) the information cannot be used against them (unless their behavior was criminal). These guarantees have been ensured through legislation and regulation. The Aviation Safety Reporting System (ASRS), run by the National Aeronautics and Space Administration (NASA) with funding from FAA, is the model and progenitor of many other versions of VSRPs that have

IDENTIFYING EMERGING TRENDS 49 subsequently been developed in aviation.1 ASRS has been in place for more than 45 years and receives about 100,000 reports annually from pilots, air traffic controllers, cabin crew, dispatchers, maintenance technicians, unmanned aviation system operators, and others. ASRS has an additional degree of protection not available in other VSRPs: individuals who report concerning incidents and accidents cannot be punished for the event based on information gathered from other sources (unless their behavior was criminal). This extra layer of protection prompts some reporters to file with ASRS as well as other VSRPs described below. In contrast to the other VSRPs discussed in this section, ASRS reports are made public in a searchable database that is widely used for training and education, safety analyses, and research. In ASRS, in addition to the public release of the “check-box” data, the narrative statements of reports of particular interest are de-identified for public release such that they can- not be linked back to the individual reporter or company. ASRS has a team of independent professionals highly trained in grouping similar reports and making inferences from the narratives about why they happened. ASRS analysts routinely compile reports addressing recurring themes in recent reports and issue safety bulletins (Safety Alerts and For Your Information Notices) on topics about which aviators, operators, and regulators should be aware. ASRS also presents safety topics to FAA and the National Trans- portation Safety Board (NTSB) on monthly safety teleconference calls and issues a monthly electronic newsletter (CALLBACK) that shares ASRS excerpts in a lessons-learned format.2 ASRS covers all of aviation—about 19% of reports in the most re- cent year (2021) were from general aviation; 53% of reports came from air carrier pilots or crews, 10% from the cabin crew, 5% from air traffic control (ATC), 3% from ground staff, 3% from maintenance, and 2% from dispatch. For commercial aviation, almost all of the reports are about individual flight operational issues. However, because there is no way of knowing the base rate (whether similar incidents are being experienced by others without being reported) or whether a prominent accident stimulates a peaking in reporting of similar issues, it is unknown whether 100 reported incidents represent 95% of real incidents or 1%. Thus, the data extracted from the narratives are not appropriate for statistical analyses that infer 1 This material is drawn from a presentation to the committee by Becky Hooey, director of NASA ASRS, on October 20, 2021. Much of the same material is available on the ASRS website at http://asrs.arc.nasa.gov. 2 See https://asrs.arc.nasa.gov/publications/callback.html and https://asrs.arc.nasa.gov/ publications/operations.html.

50 EMERGING HAZARDS IN COMMERCIAL AVIATION—REPORT 1 how frequent or widespread the issues are.3 Furthermore, reports may be lacking in detail and presented from the reporter’s limited perspective. They can, however, be used as indicators of emerging trends and for the analysis of potential contributing factors. The success of ASRS has spawned similar VSRPs within entities, where the reports are analyzed internally and not necessarily distributed publicly, recognizing concerns with data protection discussed in more detail later in this chapter. The Aviation Safety Action Program (ASAP) generates em- ployee reports that are collected and analyzed within individual companies, particularly the air carriers. For ASAP reports, Part 121 carriers rely on analysis by qualified crew members familiar with the equipment, routes, and procedures of the involved company and by a “no-fault” interview of all applicable operating crew. Should a need for remediation be determined, it is administered in a no-fault manner. The reports are similar in kind to those in ASRS, and some reporters file reports in both systems (or request that their report in one system be shared with another), in case the report is not accepted under the ASAP program and because the protections offered by reporting to ASRS are stronger. The Air Traffic Safety Action Program (ATSAP) is used for FAA’s air traffic control system operations. (Since its implementation was completed in 2010, ATC incident reports in ASRS have declined.) This program reviews internally the reports received, and the distribution of the reports is limited. As described in Chapter 3, FAA’s Aviation Safety office (AVS) has recently established a VSRP for its employees in support of assessing safety culture. The Aircraft Certification, Safety, and Accountability Act of 2020 also requires rulemaking to begin requiring VSRPs for aviation manufacturers. Flight Operational Data Flight Operational Quality Assurance (FOQA) The digital flight data recorder systems on modern commercial aircraft generate a thousand or more data streams that enable very detailed analyses of numerous aspects of aircraft operation including how take-offs, en-route flights, runway approaches, and landings conform to company policy and 3 For certain types of events, such as hazards (e.g., malfunctioning airport lighting, similar sounding fixes, or charting errors), statistical analyses are not needed or appropriate. When a single report of these kinds of hazards is received, ASRS notifies the proper authority (such as the airport manager or chart publisher), who can address it right away. ASRS also disseminates this information so operators (pilots, airlines, etc.) can be aware of the hazard to prevent an incident. This hazard identification and alerting is a unique feature of ASRS.

IDENTIFYING EMERGING TRENDS 51 procedures. Air carriers establish parameters within which flights are ex- pected to remain for such items as the stability of approaches, take-off rotation rate, flap position in different flight phases, speed of approach at landing, and hardness of landings (Walker, 2017). Systems onboard also record automated warnings to pilots regarding potential stalls, ground proximity, and nearby aircraft, and pilot control activity including button presses for flight management and other aircraft systems. Air carriers can voluntarily participate in an FAA-approved FOQA pro- gram, in which case FOQA data are protected from enforcement actions. Part of participating in an approved FOQA program is an obligation to analyze the data and to follow up with corrective action plans (CAPs) to mitigate any safety issues identified. FOQA data have several important attributes: • FOQA provides a comprehensive set of data covering most of the main systems in the aircraft, pilot control activity, and aircraft trajectory; • Most data are recorded at frequent (up to 1-second) intervals; • Many data fields are commonly used across carrier/make/model of aircraft, making aggregation of these fields possible across airlines; and • FOQA data from flights of interest can be linked to VSRP reports by carriers under some circumstances. There are some limitations to the data provided by FOQA programs: • Those aspects of the vehicle structure and systems that are not instrumented or connected to the flight data recorder are not covered; • Internal calculations and state (or mode) within flight management system state may not always be completely captured; • The state of many software elements is not captured; • Some carriers do not fully instrument with FOQA; and • Crew state and fatigue are not captured. Line Operation Safety Audit (LOSA) LOSA is a front-line observational methodology with the objective of iden- tifying systemic and flight crew performance strengths and weaknesses. Originally developed as a method to understand the impact of crew re- source management performance in situ, the methodology was expanded in the 1990s to include the capture of Threat and Error Management

52 EMERGING HAZARDS IN COMMERCIAL AVIATION—REPORT 1 (TEM).4,5 LOSA was initially developed in a collaboration between an individual airline and university researchers in the mid-1990s and has since spread worldwide to many commercial carriers. In the early years the evaluations expanded to include the TEM methodology, by which crews are rated on how they respond to threats and errors arising during normal operations. Threats are events that originate beyond the influence or control of the pilot while errors are actions or inactions by the pilots; both must be recognized and handled by the pilots. Some examples of threats requir- ing flight crew management might include challenging ATC clearances, turbulence encounters, aircraft malfunctions, or operational tine pressures. Examples of error might include wrong automation entries, not turning on anti-ice systems in icing conditions, skipping a checklist item, or failing to make a required callout during an approach. As a rule, LOSA observers do not reveal to the pilots that they have observed a threat or error unless they are life threatening. Information gleaned from LOSA reports can serve as leading indicators of (a) what might have gone wrong, and inform carrier training programs accordingly, and (b) as a way organizations can learn from what goes right in normal practice. Aggregation of LOSA data over a series of flights allows carriers to iden- tify potential weaknesses in skills and procedures, which they can respond to with training and revisions to procedures and policies. Observations of crews struggling with instructing and monitoring automation provide useful information about design issues that can be fed back to designers and manu- facturers, and are useful to airlines for training purposes. However, different air carriers may emphasize collecting more data about specific phenomena, and may use different formats and types of measures, limiting its broad use across multiple airlines without careful coordination. LOSAs started as independent audits by trained observers, either on a continuous basis or at specific intervals, and have expanded to include training of carrier staff to conduct their own audits. The collection and evaluation of LOSA data is a voluntary program for carriers, including for the crews who are observed. The initial model of focusing on piloting has more recently been expanded to maintenance (M-LOSA) and ramp opera- tions (R-LOSA) for carriers (Crayton et al., 2017), and air traffic control in Europe, known as the Normal Observations Safety Survey.6 4 This section is based on a presentation to the committee by James Klinect on January 28, 2022. See also Klinect et al. (2003). 5 Crew Resource Management is an approach to managing flight in the cockpit that makes optimum use of all available resources (equipment, procedures, and people) to ensure a safe and efficient flight and to handle anomalies in a safe manner. 6 See https://skybrary.aero/articles/normal-operations-safety-survey-noss.

IDENTIFYING EMERGING TRENDS 53 Maintenance Maintenance errors contribute to 4–5% of accidents and incidents, but acci dents involving maintenance are 6.5 times more likely to involve a fatality and maintenance-involved fatal accidents result in 3.6 times more fatalities than other fatal accidents (Marias and Robichaud, 2012). The committee has not yet investigated in detail available maintenance-related data. The committee simply notes in this report the information about which it is aware. It should be noted that of the 126 CAST Safety Enhance- ments (SEs) described later, 29 were related to maintenance. The latter was initiated in 2016 for the purpose of reducing the “risk of taking off with misconfigured flaps or slats by directing airplane manufacturers and air carriers to ensure air carrier maintenance programs include appropriate ac- tions and procedures for the proper operation of the Takeoff Configuration Warning System (TCWS).”7 These details mirror several published recom- mendations of the report on a 2008 accident at Madrid-Barajas airport;8 thus, this example does not necessarily demonstrate the use of a separate maintenance data set. Airlines are required to have a Continuing Analysis and Surveillance System (CASS) for monitoring and analyzing the performance and effec- tiveness of their Continuous Airworthiness Maintenance Programs (FAA, 2019, which has subsequently been extended), which can serve as a data source for both the carrier and for FAA (USDOT, 2021). Noted above is the existence of M-LOSA, but the committee is as yet unaware of how widely it is being used. The committee is aware of use by some airlines of the Maintenance Error Decision Aid (MEDA) methodology to analyze potential maintenance errors following gate returns and air turnbacks to understand whether the problem resulted from improper maintenance and, if so, to identify proximate causes that contributed to it (Rankin, 2007). As noted, maintenance issues are reported to ASRS, but they represent 3% of ASRS reports, and are likely also reported in company ASAPs. Other data, such as the messages transmitted between aircraft and their air carrier bases via the Aircraft Communications, Addressing and Report- ing System (ACARS), contain messages about maintenance and aircraft malfunctions that are currently used within air carriers to actively manage their operations (e.g., schedule a maintenance check of an aircraft once it lands if it reports anomalies in flight and, as necessary, prepare to adjust the schedule and aircraft assignment for its subsequent flights). 7 See https://skybrary.aero/articles/se229-takeoff-misconfiguration-takeoff-configuration- warning-system-maintenance-and. 8 CIAIAC Report A-032/2008.

54 EMERGING HAZARDS IN COMMERCIAL AVIATION—REPORT 1 Incidents There are a variety of incident data reviewed in this section based on data sources investigated to date. NTSB Accident Reports NTSB prepares reports for all civil aviation accidents involving a U.S. entity and also may select incidents to investigate.9 This section refers to such reports on incidents. Preliminary reports provide basic factual information such as the date, operator, aircraft type, and location and a short narrative explaining what happened. A final report includes an analysis of the event and an assignment of probable cause(s). However, because they are selected for investigation for particular reasons, they may not be representative of the frequency of such events. As described next, FAA has a more complete database, but it lacks assignment of probable causes. FAA Accident and Incident Data System Reports FAA defines an incident as “an occurrence—other than an accident— associated with the operation of an aircraft that affects or could affect the safety of operations.”10 Aircraft operators are required to report certain incidents involving certain kinds of aircraft malfunctions, near-midair colli- sions, and reports of pilot loss of capacity. FAA incident data in the Aircraft Integrated Data System (AIDS) provide basic descriptive information about each incident and a short narrative. Because these events must be reported, AIDS reports may be more representative of the frequency of events, but lack insight into why they happened. The data are available to the public. ATC Mandatory Occurrence Reports (MORs) ATC staff are now required to report certain kinds of ATC-related incidents such as near-midair collisions (NMACs), pilot deviances from controller instructions or vehicle or pedestrian deviances (runway incursions), and those Traffic Collison Avoidance System resolution advisory occurrences known to them (FAA, 2020). These data are not made public. The Air Traffic Organization retains an Aviation Risk Identification and Assessment (ARIA) computer system that automatically analyzes radar and other sur- veillance data to identify air traffic operations that may represent potential 9 An incident is defined by NTSB as an occurrence, other than an accident, associated with the operation of an aircraft, which affects or could affect the safety of operations. 49 CFR § 830.2. 10 49 CFR § 830.2 definitions. See https://www.law.cornell.edu/cfr/text/49/830.2.

IDENTIFYING EMERGING TRENDS 55 risks. These Preliminary ARIA Reports are reviewed by designated quality assurance (QA) staff. Service Difficulty Reports FAA certificate holders and certificated maintenance, repair, and overhaul stations must report defects, malfunctions, or serious failures of aircraft components within 96 hours of their discovery. These reports are used by FAA and NTSB, and data are publicly available.11 There is apparently some concern about how thoroughly certificate holders are reporting such inci- dents.12 One gap in the reporting of service and other performance issues is that FAA does not always receive information from regulatory authorities in other nations about service difficulty and performance issues with U.S.- certified and -manufactured aircraft operated abroad by non-U.S. carriers.13 Near-Midair Collision Database In addition to the required reporting of NMACs by air traffic controllers, flight crews are strongly encouraged to report aircraft being within 500 feet of each other by phone or radio to the nearest FAA ATC facility. Details of these events are not public, but the annual number of events is reported by the Bureau of Transportation Statistics. Measures of Fatigue Fatigue is a well-documented contributing factor of degraded performance and safety (FAA, 2013) and is also an inherent part of a transportation system that operates 24/7 and often includes trips across time zones that adversely affect the circadian rhythms of flight crews. Flight crews (both pilots and cabin crew) can be asked to complete long flights or long duty cycles comprising several flights and may have extensive commutes prior to their flight origin; ground personnel at both carriers (e.g., maintenance, dispatchers, ground handlers) and at FAA (e.g., air traffic controllers) may likewise struggle with long shifts when staffing concerns lead to compressed schedules and overtime. Fatigue Risk Management Systems (FRMSs) are “data driven means of continuously monitoring fatigue related safety risks.”14 They are used in aviation as an alternative method of compliance 11 See https://av-info.faa.gov/dd_sublevel.asp?Folder=%5CSDRS. 12 See https://www.faa.gov/other_visit/aviation_industry/airline_operators/airline_safety/ info/all_infos/media/2016/InFO16009.pdf. 13 Presentation to the committee by David Hempe, FAA, January 28, 2022, The Future of Data Analytics and Safety Certification. 14 See https://skybrary.aero/articles/fatigue-risk-management-system-frms.

56 EMERGING HAZARDS IN COMMERCIAL AVIATION—REPORT 1 with prescriptive FAA regulations regarding work hours and rest periods (FAA, 2013). Regulations regarding work hours and rest periods do not ensure that a person rests adequately when they are not on duty, whereas FRMS is an effort to help people manage fatigue issues even while they are not on duty. According to FAA, “The FRMS approach is to apply risk management techniques to identify and reduce the risk of fatigue relevant to specific operational circumstances. An FRMS aims to ensure high levels of alertness in personnel to maintain acceptable levels of performance and safety” (FAA, 2013, p. 4). FRMSs are data driven rather than prescriptive requirements of a certain number of hours of rest between flights or shifts. They generally reflect scheduling and periods for recovery and sleep and do not include biometric measures. The various forms of fatigue that impair human performance are difficult to measure, and fatigued indi viduals can be poor judges of their own state of awareness, which can inhibit volun- tary reporting about fatigue on this subject. However, there are accepted markers of fatigue (not enough sleep or opportunities for sleep, travel across time zones) that can serve as indicators. Some of the ongoing moni- toring by carriers includes analysis of self-reports of fatigue in VSRPs or self-reports as part of the FRMS. Other sources available with an FRMS are statistical analyses examining outliers and trends in • “Crew flight and duty periods, and rest breaks to reduce fatigue; • Additional duties assigned to flight crews that further reduce sleep opportunities; • Schedule changes that extend duties beyond the published schedule; • The duration and timing of layovers between successive flight segments; • Recovery days, following a trip, that permit sufficient sleep to eliminate any accumulated sleep debt prior to scheduling or per- forming additional flight duties; and • Optimal utilization of available rest opportunities” (FAA, 2013, p. 7). Results of statistical analyses on these topics can serve as risk indicators. FRMS data can be considered leading indicators. They are only available within individual companies. Safety Management Systems Since 2015, Part 121 air carriers have been required to implement safety management systems (SMSs) (FAA, 2015). (SMSs for certified aviation com- panies other than Part 121 are voluntary at this time.) The hazard analysis and safety assurance (SA) processes required by SMSs would obligate air

IDENTIFYING EMERGING TRENDS 57 carriers to analyze and monitor many of the same data described in the sec- tion above. SMSs include well-defined safety goals and metrics and means of measuring performance. SMSs themselves, however, should also generate information about how well the process is working and whether the organi- zation is making progress toward its safety goals. Such information would include internal program evaluations, self-audits, third-party or FAA audits, root-cause analyses of lapses in safety controls and control management, and CAPs to address audit and other findings. FAA is still in the process of developing and implementing procedures that will allow it to draw data from its inspections of carrier safety pro- grams that would provide insight into the effectiveness of carrier SMS programs. Recent U.S. Department of Transportation (USDOT) Office of Inspector General audits of FAA oversight of individual carrier metrics for their maintenance and SA programs found gaps in FAA training of inspec- tors to evaluate SMSs at two carriers and weaknesses in FAA’s oversight of SA activities in another (USDOT, 2019, 2020, 2021). For example, FAA inspectors accepted inadequate root-cause analyses of carrier maintenance errors at two airlines and approved CAPs before they had been closed out. Inspectors were also untrained in assessing indicators of the safety culture of a third air carrier. Identifying the root causes of errors and implementing corrective actions are critical indicators of corporate vigilance regarding safety. Safety Culture Assessments The collaborative approach that FAA has adopted to work with industry to maintain and improve safety depends heavily on the safety cultures of the organizations involved. Success requires creating (a) a trusting report- ing culture by industry to ensure reporting of incidents to VSRPs (and commitment to identify and resolve weaknesses in safety management), and (b) a trusting oversight culture that will be learning oriented rather than punitive. As discussed in Chapter 3, an ongoing process of employee surveys, focus groups, and other assessment methods can provide senior management with qualitative and quantitative information and trends for monitoring the strength of an organization’s safety culture. Well-designed and -implemented assessments can provide timely leading indicator data by highlighting such things as loss of organizational motivation to attend to safety management, failures to identify hazards and execute safety con- trols, and normalizing weak signals that can be precursors to catastrophic accidents (Weick and Sutcliffe, 2015). Culture assessment data are only available within individual companies.

58 EMERGING HAZARDS IN COMMERCIAL AVIATION—REPORT 1 Other Potential Data Sources Topics and data sources that the committee has not yet reviewed that were suggested by congressional staff include • The rigor of the certifications done through the organizational desig nation authority process, whereby employees of the applicant are delegated by FAA with key roles in decision making and report- ing safety concerns to FAA; • Indicators of airport safety monitoring and management; • Manufacturing quality control/quality assurance processes; • Indicators of training competence; and • Tracking of airworthiness directives and other updates to initial certification decisions, to analyze for trends in where certification decisions have later been found to have gaps. Other data sources are also available, such as the flight track of the aircraft and neighboring aircraft and weather. While many of them do not directly contain safety measures, they may provide contextual information about operations to better inform analysis of the data sets just described. ANALYSIS PROCESSES In principle, reflecting the scope and multifaceted aspects of managing avia- tion safety as discussed earlier in Chapter 2, many simultaneous analyses are needed to analyze the just-discussed data sources to identify and char- acterize emerging hazards. These span many different dimensions. • The time frame of the analysis: o As noted in Chapter 2, immediate-time-frame analyses monitor phenomena currently observed in aviation operations, includ- ing both now-apparent hazards needing characterization and ongoing monitoring of hazard mitigations as they are fielded. o Intermediate-time-frame analyses tend to examine hazards that may only now be coming observable in operational data. o Future-time-frame analyses may need predictive modeling of effects potentially resulting from the upcoming introduction of new designs and procedures. • The scope of the analysis: o Some analyses focus exclusively on particular hazards. For example, the committee was briefed by FAA officials who each focused on specific issues. These included pilot interaction with

IDENTIFYING EMERGING TRENDS 59 automated flight deck systems,15 training pilots to protect against loss-of-control (LOC) events,16 and tracking design assumptions through the certification process and then following up once a system is certificated to confirm the certification rationale reflects actual behavior in the field.17 o Some analyses focus on broad issues within one organization. For example, LOSA analyses are typically aggregated within each air carrier, even as they span many potential hazards that a carrier’s operations may be seeking to identify, characterize, and handle. o The broadest analyses are intended to look at system-wide haz- ards that transcend the attributes of one air carrier and location. Of note, identifying and addressing system-wide hazards are defin ing attributes of both the Commercial Aviation Safety Team (CAST) and the Aviation Safety Information Analysis and Sharing (ASIAS) initiative. • Who is involved in the analysis, and who has access to the data: o Some analyses and monitoring processes serve a valuable func- tion within a specific organization as a mechanism for self- monitoring and learning, and are conducted internally. These span many of the data sources just noted, including the FOQA, ASAP, and LOSA data and their associated analyses conducted within air carriers. This type of analysis is particularly suited to addressing concerns specific to that organization (e.g., an air carrier’s particular equipage and training programs), building on data and expertise available within the organization. It is worth noting that many of these analyses still involve some level of regulatory oversight, and may also involve high-level collabora- tions in terms of sharing findings that might be relevant to others and sharing best practices in gathering and analyzing data (e.g., the LOSA Collaborative). o In many cases, such as the studies noted above focused on spe- cific hazards led by FAA officials, a broad collaborative team may be formed that represents many organizational perspectives. For example, working groups or Aviation Rulemaking Commit- tees may be formed including relevant personnel from FAA, air carriers, original equipment manufacturers (OEMs), labor (e.g., 15 Presentation to the committee by Kathy Abbott, January 8, 2022, Using Multiple Data for Safety and Operational Analyses: Some Lessons from Aviation. 16 Presentation to the committee by Jeffrey Schroeder and Barbara Adams, January 28, 2022, Emerging Safety Trends in Aviation: Pilot Training Aspects. 17 Presentation to the committee by David Hempe, January 28, 2022, The Future of Data Analytics and Safety Certification.

60 EMERGING HAZARDS IN COMMERCIAL AVIATION—REPORT 1 pilot or controller associations), and researchers from other government agencies (e.g., NASA) and from academia. o In the broadest sense, CAST and ASIAS seek to represent the many relevant aspects of the entire aviation industry in the United States. Similar collaborations can also be found internationally, and the International Civil Aviation Organization (ICAO) pro- vides a global integrative function. In summary, a wide range of analysis processes are possible and, even in the preliminary briefings received by the committee, appear to be ongoing. Some may be focused on specific hazards or conducted within specific organizations—at the other extreme, the broadest “system-wide” charter is associated with CAST and ASIAS. Data Sharing and Integration and Concerns with Data Protection Many aviation safety data sources noted earlier are considered highly sensitive. Data collected by air carriers can include elements considered proprietary by themselves or by the manufacturers of equipment that is being sensed and recorded. Requests for sharing measures of human perfor- mance, behavior, or physiology, including audio and video recordings, raise significant concerns with individual privacy and impact labor relations. Taken out of context, raw data are easily misrepresented or misinterpreted. Thus, an important consideration in the analysis of emerging trends in avia- tion is understanding who can have access to data, and understanding the processes by which data can be shared with others or, ultimately, be made publicly available. These processes can be extensive and require experience and resources, such as the review of ASRS narrative reports by professional analysts who carefully remove any information that make it identifiable in terms of reporter, location, or the people and business entities involved where it is not necessary to characterize the safety concern and compare it to other reports. Likewise, different organizations may collect data in different formats, units, and frequencies (e.g., different digital flight data recorders collecting FOQA data), requiring sometimes-significant effort to translate and integrate. One major contribution of ASIAS is its function of gathering together major data sources and controlling their dissemination to provide public access where possible, and to provide strong data protections otherwise. The development of this data-sharing collaborative is now 15 years in the making, reflecting the long time frame required to develop the trust, policies, and formats to the most sensitive data. Their strong data protec- tions, including a clear understanding of who will have access to the data,

IDENTIFYING EMERGING TRENDS 61 the analyses they will be used for, and the type of results and outputs of the analyses (and their dissemination), are generally considered important fac- tors in decisions by air carriers, labor associations, and other parties to voluntarily contribute data that they otherwise might not share—indeed, that they otherwise might not even collect. The availability of the data sets reviewed earlier in this chapter is shown in Table 4-1, highlighting which data sets remain private to entities such as air carriers and which are provided by ASIAS both publicly and to the private and public stakeholders participating in this initiative. Other data sharing beyond ASIAS was also part of some of the studies briefed to the committee. Of note, LOSA data, although a rich source of information about flight crew performance, are not included in ASIAS. There are difficul- ties in aggregating LOSA data from individual carriers given variations in carrier measures and practices. However, the committee is aware of at least one instance where data collected consistently by the LOSA Collaborative have been applied to research with permission from the carriers involved (Performance-based Operations Rulemaking Committee and Commercial Aviation Safety Team Flight Deck Automation Working Group, 2013). These protections on data can potentially obstruct data-driven re- search into aviation safety by independent parties. These can include other government agencies interested in aviation safety or in comparing safety methods across safety-critical disciplines, as well as academic research and workforce development in these areas. Where this is the case, the safety analyses will need to have their in-house expertise sufficient to create or adapt appropriate measures, and may benefit from purposefully creating opportunities for cross-talk and independent analysis. Although programs such as ASIAS require the physical sharing of data, advanced analytical techniques exist to analyze disparate heterogeneous data sets without centralizing the data in a single system. These techniques can simultaneously maintain appropriate privacy and security controls while enabling sophisticated safety analyses. Where the right data sets are available to the analyst, their integration can provide significant value to certain types of analysis. For example, FOQA can reveal what happened during a flight, but not why. Linking numeric FOQA data with textual data from sources such as ASAP, Advanced Qualification Program, and ATSAP narratives, when available, provides in- sight into causality, and subsequent interviews with flight crews or other reporters can help understand how an accident was avoided. Similarly, the data collected from different sources—such as FOQA data gathered from many airlines—provide the large amount of data amenable to analysis pro- cesses looking for relatively weak signals or rare events. Thus, the selection of analysis processes to identify, characterize, and handle emerging trends in aviation safety has both an increasing availability

62 EMERGING HAZARDS IN COMMERCIAL AVIATION—REPORT 1 TABLE 4-1 Examples of Precursor Data Sources Data Type Included in ASIAS Not in ASIAS Public Not Public Voluntary Safety Reports ASRS ASAP, ATSAP Not all companies include ASAP details that others share with ASIAS on event types and contributing factors. Flight Operational Data Advisory Directives FAA-Approved FOQA Programs Not all companies share all FOQA data they collect (e.g., gatekeeper comments). LOSA Controller–pilot datalink communications. OEM recommendations on performance limits. Maintenance SDRS M-LOSA MEDA CASS Reports included in ACARS messages. OEM data on maintenance issues/risks. Mandatory Irregularity Reports on maintenance. Incidents NTSB incidents FAA AIDS ATC MORs (pilot deviances, NMACs, runway incursions) Flight crew reports of NMACs Ground Aviation Safety Action Reports (off runways and taxiways). Fatigue FRMS data Organizational Safety Management May receive comment in ASRS reports May receive comments in ASAP and ATSAP reports SMS data (CAPs, audits, self-inspections) FAA inspections Safety Culture May receive comment in ASRS reports May receive comments in ASAP and ATSAP reports Company self-assessments FAA inspections SOURCES: See previous section of this chapter for the descriptions of data types. Comments in italic based on Gap Analysis of ASIAS Data Collection, provided by The MITRE Corporation.

IDENTIFYING EMERGING TRENDS 63 of data analysis tools at hand, and fairly specific constraints with pragmatic and policy concerns with which data sets are available and with difficulties in their aggregation and integration. The next sections briefly review the categories of analysis processes suitable for different types of analysis. Discovery and Characterization of Newly Emerging Trends The most ambitious interpretation of the task of identifying emerging safety trends focuses on identifying new, trending phenomena—or those that will emerge in the future with proposed changes in aviation. In this case, analysis processes are needed to discover, identify, and characterize as-yet-unknown hazards. Because the recent safety record in aviation has not provided a large data set of obvious accidents and incidents, analysis must be directed at “weak signals” (i.e., patterns in the data that, until rec- ognized as reflecting a distinct phenomenon, are difficult to separate from the normal variance in day-to-day operations and the noise added by the measurement process). Broadly speaking, two classes of analysis processes can be applied here, which we will term “hypothesis free” and “hypothesis driven.” Hypothesis- free processes, adapted from analysis methods used in other industries experi enced in big data methods, search large data sets for recurring pat- terns that are not already identified; as patterns are identified their effects can be factored in so that the analysis can iterate looking for progressively smaller patterns. As a corollary, similar methods might choose instead to look for statistically anomalous instances (e.g., flights) that do not fit into normal recognized patterns. Either way, new patterns potentially reflecting emergence of new behaviors—or instances not reflecting the patterns asso- ciated with normal flights—can identify individual flights or trends across flights reflecting unusual behavior (Das et al., 2010; Gariel et al., 2011). While such hypothesis-free processes start with automated, algorithmic analysis of large data sets, once patterns or statistical anomalies are identi- fied there is a strong role for subject-matter experts (SMEs) to make sense of these patterns and anomalies. As a thought experiment, the introduction of a new aircraft designed for maximum efficiency and using new propul- sion systems (e.g., many small engines distributed along the leading edge of the wing) would be identified by the automatic algorithms as “different” in the trajectory, speed, climb, descent, and propulsion data streams within a FOQA data set and other related data sets such as air traffic surveil- lance data, with potential ripple-out effects into other measures such as the dynamics of neighboring traffic flow. These differences may not reflect safety concerns: expert analysis is required to further investigate their safety impact, recognizing that (a) some differences may be benign—or even positive—in their safety impact, and (b) experts may not have the

64 EMERGING HAZARDS IN COMMERCIAL AVIATION—REPORT 1 perspective to look for significantly different concerns in the future beyond those they are knowledgeable about. Methods to discover previously un- known patterns from massive data sets have been developed and applied to aviation data for the purposes of revealing atypical flights (Gariel et al., 2011). These techniques can simultaneously combine numeric FOQA data streams with textual data from ASRS reports to dramatically increase the ability to detect operationally significant anomalies (Das et al., 2010). Hypothesis-driven processes, in contrast, generally start with expert analysis of proposed changes to, or stressors on, aviation operations. As noted in Chapter 2, one method guiding detailed hypotheses is to examine the assumptions created during design to constrain the design space; a hy- pothesis may consider “what if” one or more of these assumptions are vio- lated by, for example, an aircraft finding itself in flight conditions beyond assumed operating conditions. This hypothesis-driven analysis also extends to projecting hazards potentially to emerge in the future when examined through methods such as horizon scanning, as described in Chapter 5. Applying the thought experiment given in the previous paragraph, the safety impact of introduction of a radically new aircraft with different maneuvering capability can be examined by poking at the assumptions of the airspace system, which assumes aircraft can (and generally will) climb and descend at least 500 fpm, turn at a rate of 3 degrees per second, and follow a glideslope elevated upward at 3 degrees; when juxtaposed against the maneuvering capability of new aircraft, cases may be identified where key assumptions in the airspace are no longer valid. This analysis leads to well-posed hypotheses whose confirmation (or negation) can then be verified by investigation of the data for the specific patterns or signals they would reflect. Where data do not yet exist, more- predictive models may be applied to simulate or extrapolate, from first- principles and physics-based models, the range of possible evolutions of a hazardous state. At this time, the wide range of methods seeking to discover new trends remain largely topics of research. Of the briefings provided by FAA and other entities to the committee, while the ability to detect “weak signals” and identify new hazards is recognized as an ideal, none of the briefings or materials described current studies employing the methods described in this section in a substantive manner. Analyzing for Precursor Measures of Identified Hazards Where hazards have been identified, increasingly sophisticated methods can be brought to bear to better characterize them, and to use this char- acterization to make better use of data. For example, the committee re- ceived a briefing authored by MITRE, the Federally Funded Research and

IDENTIFYING EMERGING TRENDS 65 Development Center that also performs much of the ASIAS data analysis, describing a structured approach for examining the general phenomenon of LOC accidents, which is attributed to the most commercial aviation acci- dents worldwide in the past 20 years (The MITRE Corporation, 2021). The study defined multivariate criteria that, alone or together, signal an LOC event; highlighted the range of triggers or hazards that might precipitate these events; and quantified the criteria in terms of data streams found in the digital flight data recorder data used in FOQA analysis by air carriers and provided by them to ASIAS. The study then trained several “deep learning” algorithms to not only detect LOC events within the FOQA data taken from presumably “normal” flights, but also identify time-series patterns in the data immediately preceding the events, which themselves could be viewed as “precursor measures” capable of both more definitively identifying LOC events when they do occur, and identifying the precursor progression of states that could lead to LOC events within 30 seconds. The resulting trained models (each created by different algorithms) were then used to examine 2,000 flights to see how well they predicted LOC events. The study concluded with a proposal for extending these methods to two other significant accident categories (controlled flight into terrain [CFIT] and runway excursions); the briefing did not clarify the extent to which this study’s methods are being applied broadly as part of ASIAS. The committee was also briefed by FAA officials on studies each focused on specific issues, including pilot interaction with automated flight deck systems, training pilots to protect against LOC events, and tracking design assumptions through the certification process and then following up once a system is deployed to confirm the certification rationale reflects actual behavior in the field. In these cases, it appears that the collaborative teams examining these issues sought to characterize the hazard sufficient to identify precursor measures. For example, studies of pilot interaction with automated flight deck systems included analysis of accident and incident reports, ASRS reports, and LOSA data from 10,000 flights to characterize which conditions or events appear to be inherent to everyday flight opera- tions, and which appear to correlate with increasingly severe outcomes (Flight Deck Automation Working Group, 2013). These studies—and there may be many others that the committee is not yet aware of—reflect the growing potential for in-depth characterization of hazards sufficient to both target monitoring of current operations toward precursor measures that capture known hazards and to support develop- ment of methods to eliminate or mitigate the hazards. However, these types of methods do reflect several pragmatic concerns, particularly around the difficulties of dealing with large amounts of data of uncertain quality, and integrating data sets that reflect fundamentally different perspectives on a phenomenon. Thus, the committee recognizes the challenges in deep, rich

66 EMERGING HAZARDS IN COMMERCIAL AVIATION—REPORT 1 characterization of precursor measures of hazards, and the need for im- proved analysis processes to assess them. Monitoring Exceedances and Lagging Indicators An immediate method for analyzing aviation safety data is to monitor operational data for “exceedances” (i.e., instances where data fall outside specific limits specified for each data stream [e.g., airspeed higher or lower than some criteria]) and for lagging indicators such as incident reports. Based on the briefings and materials that were provided, the committee is under the impression that many aspects of FOQA data analysis within air carriers, and of the data analysis by ASIAS and CAST, are best categorized as fitting into this category. Monitoring exceedances and lagging indicators serves a vital role in managing safety within the immediate time frame. These methods can serve as an efficient first screening of the large volume of data, digital, and text that is now being created. Thus, they tend to be the first to pick up on trends in specific issues such as pilot confusion when runway mark- ings or airport lighting is changed at a specific airport. Similarly, they can be purposefully chosen to monitor known issues of concern, and to track whether mitigations (such as CAST SEs) appear to be effective in reducing their rate of occurrence. When the data set is sufficiently comprehensive, such as spanning an entire air carrier’s operation or most of the operations in the nation, they can provide some reasonable prediction of frequency of occurrence of the specific conditions or events they measure. Relative to the committee’s charter to look at “emerging trends” in aviation safety, beyond their ability to note trends in frequency of specific immediate issues, these methods have limited impact, for several reasons. First, their correlation and causal relationship with the concept of hazards can be unclear or limited; for example, LOC events can occur at an airspeed that is considered “high,” “low,” or “normal” depending on many other factors, and by the time its value triggers an exceedance a hazardous state may be well developed. Thus, deep characterization of hazards requires the more in-depth analysis noted in the previous section on “precursor measures,” both to better identify when the system is vulnerable to the hazards and the patterns in conditions leading up to them. Second, while exceedances and lagging indicators are vital to managing safety in the im- mediate time frame, they lack the understanding of hazards to provide the extrapolations and predictions vital to the intermediate and future time frames in which trends in safety will emerge.

IDENTIFYING EMERGING TRENDS 67 CAST and ASIAS CAST and ASIAS have unique roles with their charter to examine system- wide safety. While many of the efforts described earlier involve significant collaboration in purpose of specific analyses, CAST and ASIAS seek to extend the collaboration across the entire commercial aviation industry. CAST is a voluntary collaboration of the aviation industry with government agencies that identifies, ranks, and analyzes hazards and develops and im- plements safety enhancements to manage these hazards. CAST membership consists of several government organizations, most air carriers (especially the largest air carriers), aircraft and avionics manufacturers, several labor groups, and numerous aviation associations. ASIAS is also an industry– government collaboration with industry-wide participation of many of the same members that participate in CAST, which is overseen by an execu- tive board of industry and government officials. As of May 2022, ASIAS stakeholders include 47 commercial carriers, many general aviation and on-demand Part 135 carriers, as well as numerous aviation associations, training organizations, and government entities.18 ASIAS has a repository of public and proprietary private data, which is monitored and analyzed to determine trends and the effects of implemented SEs. ASIAS also carries out special studies at the request of CAST. CAST is funded by in-kind support of staff time by industry and gov- ernment. ASIAS is funded by FAA, at about $20 million annually (roughly 10% of the FAA research and development [R&D] budget in 2021), and also depends heavily on in-kind contributions of private data shared by industry. CAST and ASIAS are interdependent, as CAST influences what ASIAS undertakes, and because the data that industry voluntarily provides to ASIAS depend on the CAST partnership and the trust that has developed between the private and public sectors. Obtaining and protecting private data for this purpose has developed over the past 15 years, largely because of the time it has taken to develop (a) the trust and willingness to share sensitive data; (b) the technical, legal, and administrative methods for pro- tecting the shared data; and (c) FAA policy to refrain from using the shared data for enforcement actions. Since its inception, ASIAS has been managed by The MITRE Corpora- tion, which is a federally funded research and development corporation that supports FAA. At the time of the committee’s information gathering about ASIAS (January through May 2022), the further development of the ASIAS platform was being competitively bid, which may have affected the ability of FAA to help the committee to understand and assess its processes and methods. 18 See https://portal.asias.aero/home.

68 EMERGING HAZARDS IN COMMERCIAL AVIATION—REPORT 1 More specific background on CAST and ASIAS is provided in the next two sections. CAST CAST is an outgrowth of two major commission reports from 1997 that addressed concerns that the commercial aviation fatality rate had leveled off after decades of steady decline.19,20 With demand for air travel accel- erating, the implication of this leveling off was that the frequency of fatal accidents would result in unacceptable increases in fatalities as demand for air travel increased. In response, the industry and the federal government established this unique, voluntary collaboration to identify SEs to mitigate known risks combined with a commitment by industry and the government to implement the SEs on a voluntary basis. CAST has evolved and matured over the past two decades. Similar teams have been formed for rotorcraft and general aviation, but this re- port focuses on the commercial aircraft, Part 121 sector, which is the most advanced and most squarely within the committee’s remit. On the private side, CAST has representatives of all of the major commercial aviation asso ciations (carriers, airports, and manufacturers), avionics manufac- turers, airline pilots union and association, airframe and engine manu- facturers, and the Flight Safety Foundation. On the public side, CAST has representatives from FAA, NASA, the Department of Defense, the Canadian civil authority counterpart to FAA, the two ATC associations, and observers from international aviation safety organizations and NTSB. Roughly 60 organizations are involved with CAST and subteams involv- ing 30–35 individuals who meet monthly to keep aware of operations and monitor the CAST safety portfolio. CAST started in the late 1990s with a known list of major accident types, such as runway excursions, CFIT, unstable approach and landing, and LOC. It identified a specific set of SEs to reduce these types of acci- dents, which have been reduced over time. It subsequently developed pro- cesses for CAST members for the purposes of identifying other potential risks, ranking their severity, analyzing the probability of accidents, and, for those with a likelihood of occurrence in a 20-year period, identifying cost-beneficial SEs.21 19 The two major reports cited are Final Report to President Clinton (White House Commis- sion on Aviation Safety and Security, 1997) and Avoiding Aviation Gridlock and Reducing the Accident Rate: A Consensus for Change (National Civil Aviation Review Commission, 1997). 20 This text is largely based on presentations to the committee by Vivek Sood, FAA, Oc- tober 20, 2021, and January 28, 2022, and subsequent correspondence. 21 See https://skybrary.aero/enhancing-safety/cast-safety-enhancements/cast-safety- enhancements- plan.

IDENTIFYING EMERGING TRENDS 69 Once a hazard has been determined by CAST as requiring mitigation, it develops SEs. A considerable additional benefit of the CAST collaboration is the buy-in generated by the extensive consultative decision process for developing the recommended SEs. Recognizing that commercial aviation’s system of controls is complex, CAST engages ASIAS to monitor whether mitigations introduced cause any undesirable side effects or are effective at mitigating identified risk. Between 1997 and May 2022, CAST has developed 126 SEs, many of which were implemented, a few of which were withdrawn, and others of which remain active. Since CAST’s creation, the fatal accident rate of commercial aviation in the United States has fallen by more than 80%, which is impressive compared with the expectation that the rate had leveled off in the 1990s. Whether this improvement is solely attributable to CAST cannot be determined, but CAST has surely been influential. The committee was briefed that many of the CAST SEs resulted from input received from CAST members and from analysis of the ASIAS data- base, including analysis of FOQA and other data sources. The committee finds that a subset of CAST SEs address several of the probable causes of fatal commercial airline accidents with 50 or more fatalities around the world in the past 15 years (see Appendix A). CAST has also proven to be flexible in responding to unanticipated major events. For example, as COVID-19 cases and deaths fell after mid- 2021, and the industry began expanding, CAST reworked its processes to meet weekly instead of monthly to develop safety guidance to carriers as workers were recalled from extended leave and parked aircraft were re- placed to service. Thus, CAST has a clear role in managing aviation safety in the immediate time frame. ASIAS The ASIAS system is both a collection of data sets, some of which are quite voluminous, and multiple processes and techniques for data analysis and numerous analysts who evaluate the data and produce reports.22 On the process side, the ASIAS Executive Board has established principles for the aggregation, monitoring, and analysis of voluntarily supplied sensi- tive safety data solely for the purposes of enhancing safety and protecting it from disclosure. The ASIAS Executive Board approves analyses to be undertaken by ASIAS. CAST often draws on data in ASIAS for risk assess- ments of identified issues brought to CAST by its members. In turn, once SEs are voluntarily agreed upon, ASIAS analysts will monitor data to assess 22 This text is largely based on presentations to the committee by Vivek Sood, FAA, Oc- tober 20, 2021, and January 28, 2022, and subsequent correspondence.

70 EMERGING HAZARDS IN COMMERCIAL AVIATION—REPORT 1 intended and unintended consequences. There is overlap between the vari- ous teams that each creates for their analyses, with many personnel serving on both CAST and ASIAS teams. CAST and ASIAS also undertake joint studies on special topics, such as approach and landing misalignment, take- off misconfiguration, and go-around during approach to land. As mentioned, ASIAS is also a collection of numerous databases, the most important of which are voluntarily provided and protected measures of operations, such as FOQA and employee VSRP reports. Also included are government data on National Airspace System (NAS) traffic flows and conditions, weather and weather alerts, NTSB and FAA accident and incident data, reports of near-midair collisions, hazards/terrain around airports, and others. As extensive and valuable as these data are for CAST and ASIAS purposes for system-wide analysis of (primarily) flight opera- tions in the immediate time frame, the ASIAS data set does not contain all data; data still held privately by air carriers (and other private entities such as OEMs) were noted earlier in Table 4-1. In response to questions from the committee, FAA’s Mr. Sood indicated an interest in extending ASIAS’s capability to examine human factors issues such as fatigue (through ac- cess to FRMS data) and human–machine interfaces (human in the loop). Access to FRMS data could also expand the CAST/ASIAS domain aware- ness into organizational management of fatigue. He also expressed interest in gaining access to airline LOSA data. Gaining access to these data sets will require developing trust levels further and building on the data shar- ing and data protection principles that have made the CAST/ASIAS efforts successful to date. The integration of many of these data sets would also require a significant level of effort, as noted earlier. ASIAS is undergoing further expansion of its analytic and data-storage capabilities. Originally labeled ASIAS 2.0, this expansion in capability is now being referred to as “Next Generation” ASIAS. This will require estab lishing a data platform for analyzing its massive data sets, which may benefit from commercial platforms that are accustomed to the volume of data involved and the data protections that many of these data sets require. Based on information the committee has been able to gather, the typical ASIAS analytic approach taken over the past decade or so appears to have been to (1) identify the general issues with a common accident cause or other observed safety concerns based on expert opinion, (2) isolate relevant data sets with permission of and visibility to CAST participants, (3) formulate queries assessing exceedances across a directed set of parameters, (4) assess whether those queries indicate a potential issue, and (5) discuss results with the safety teams to make the determination for mitigations. The hypotheses investigated are dependent on the expert judgment of SMEs; hence, the perspectives of the engaged SMEs will determine the types of issues deemed worthy of investigation.

IDENTIFYING EMERGING TRENDS 71 Furthermore, earlier discussion noted MITRE studies using artificial intelligence (AI) and machine-learning technologies that support analysis processes capable of identifying precursor measures that better correlate with, and are able to predict, hazardous states. These methods can add con- siderable power to the analysis. They still appear to be targeted at known hazards that have manifested as the leading contributors to accidents in the most recent 20 years, as appropriate to managing safety in the immediate and, to some degree, intermediate time frames. SUMMARY Leading Indicators and Precursor Measures In general, all of the briefings the committee received highlighted the value in collaborating by sharing best practices and, where possible, sharing data, as well as collaboration in making sense of the data analysis and in handling safety concerns as they appear. Thus, safety assessments are not performed solely by FAA, and many different organizations can contribute data and contribute perspectives to the leading indicators and precursor measures that should be identified and monitored. Within this large, collaborative community there currently exists a con- siderable and varied set of data sources. The committee has begun to inven- tory this array of data. It has only had the opportunity, to date, to delve into the main ones in use, and is therefore not yet able to identify specific gaps as called for in its Statement of Task, with two exceptions. First, it has not found examples of measures of issues associated with the software that modern aircraft and automation systems are heavily dependent on, includ- ing (a) pragmatic issues such as organizations being fully up to date with current versions, and (b) more advanced records of behavior within safety- critical software, including flight control and flight management software, such as records of the internal states and when the software perceives the vehicle hitting key limits, which may both highlight bugs within the soft- ware and facilitate reviews of assumption- and hypothesis-based analyses. Second, it is unclear overall how comprehensively data about maintenance operations (including maintenance errors) are included in studies by ASIAS, CAST, and others who briefed the committee, further complicated by the fact that FAA does not receive complete information about aircraft service difficulties and performance issues of U.S.-manufactured aircraft that are operated abroad by non-U.S. carriers. SMS data and safety culture assessments can provide insights to support immediate-time-frame analyses of safety, in principle, but the com mittee has not yet taken stock of current industry practice. It does appear from USDOT Inspector General reports cited in the text above that Part 121 carrier

72 EMERGING HAZARDS IN COMMERCIAL AVIATION—REPORT 1 adoption of SMSs and FAA’s oversight of them have not yet fully matured across carriers and FAA field offices since the mandated introduction of SMSs in 2015. Intermediate- and future-time-frame analyses of safety would likewise benefit from understanding the practices and safety culture in the design, certification, and operational approval functions within the industry. It appears that data gathering and data analysis focus on the immedi- ate front line of air carrier operations. FOQA data can serve for an initial identification of “exceedances,” and VSRP reports can flag issues noted by front-line personnel, which then can be analyzed in depth, where war- ranted, by the integration of information about the event from multiple data sources. CAST/ASIAS analyses of these types of events serve to look for system-wide issues, while individual carriers may be in a position to draw inferences about the influences of crew fatigue, effectiveness of their hazard controls, or influence of their safety cultures. The committee also recognizes that some studies have examined specific issues or hazards in more depth than counting exceedances. Through a combination of SME review, modeling and characterization of hazards, and a variety of data analysis methods (including those building on “big data” methods applying AI and machine learning), these studies have developed measures of precursors with demonstrated correlation and, in some cases, inferences about causation. These deeper analyses can inform more trans- formative improvements to aviation that may require a longer time frame, especially those involving the long time scales of design, certification, and production of new technologies. At least based on the information received so far, the committee is not aware of future-time-frame efforts seeking to discover new hazards that are not already identified beyond direct research studies attempting to develop methods for them. Such studies could include discovery efforts seeking to identify hitherto-unidentified anomalous patterns of behavior in the avia- tion system visible only as “weak signals,” as well as hypothesis-driven investigation of the impact of likely stressors on the aviation industry or proposed changes to technology, traffic density and distribution, and pro- cedures and policies that may warrant a structured exploration by SMEs and predictive modeling. Initial Assessment of CAST/ASIAS The CAST/ASIAS collaboration is a truly remarkable one that is unrivaled in any other high-hazard industry of which the committee is aware, with a unique and valuable role in identifying and addressing safety concerns in the immediate time frame. The committee recognizes that the CAST/ASIAS process depends on the existing trust developed between the government and private sectors over the past two decades and the sensitivity, value,

IDENTIFYING EMERGING TRENDS 73 and importance of the proprietary information shared by private industry. These public–private collaborations have undoubtedly contributed to the considerable improvements in commercial aviation safety since their incep- tion. Having tackled the most significant causes of accidents, CAST and ASIAS continually review those causes of accidents which still have not been sufficiently eliminated or mitigated, which is a challenging task. As an example of its ongoing central role in managing system-wide safety in the immediate time frame, CAST proved itself to be highly flexible in adapting to manage safety concerns that could arise during the COVID-19 pandemic with stressors such as retraining of personnel after furloughs and layoffs, and aircraft needing to be readied for flight after extended storage. The CAST/ASIAS process is heavily oriented around available flight operational issues and proximate causes of errors. In some cases, data about important factors, such as fatigue, are not directly available to CAST/ ASIAS and could benefit ASIAS’s central analysis, tempered by the fact that the results of CAST/ASIAS studies are available to the air carriers that col- laborate with CAST and who can manage their organizational structures to address these factors. The committee has two conclusions that could strengthen CAST and ASIAS. First, for understandable reasons, participants in CAST delibera- tions and ASIAS analyses are carefully limited and controlled. While CAST and ASIAS appear to be attentive to include a wide and deep range of subject-matter expertise in these processes, the restrictions on participants do put limits on the extent that they can draw on the insights of others. Thus, they may benefit both from routinely rotating specific SME partici- pants such that they are consistently bringing in fresh perspectives from within their member organizations, and from coordinating with outside experts for cross-talk and information sharing (even if only at the levels of coordinating on modeling and analysis techniques and methods for assess- ing and managing system safety). Second, in addition to FAA’s proposed updates to the ASIAS data plat- form, it appears much of its current focus is on monitoring exceedances and lagging indicators. The committee recognizes the value of current directed efforts to develop methods to identify and monitor precursor measures that provide both deeper insight into the underlying hazard and provide more predictive measures of patterns of behaviors that suggest the development of a hazardous state. These efforts could benefit from bringing in outside experts who are knowledgeable about both the multiple dimensions of avia- tion safety controls and the most advanced data mining applications and tools. In addition, R&D agencies such as FAA, NASA, and the National Science Foundation (NSF) have an opportunity to enhance the state of the art through development of advanced techniques for detecting weak signals such as those that arise in commercial aviation.

74 EMERGING HAZARDS IN COMMERCIAL AVIATION—REPORT 1 Whereas the CAST/ASIAS collaboration has a valuable role in immediate safety management, particularly in collaboratively developing mitigations for known hazards, it may not be the best forum for application of tools and methods for future-time-frame analyses of hazards that are not yet known (and indeed may not manifest until proposed changes are implemented). There may be better venues, such as advanced simulation laboratories used in development of new technologies and R&D that draws on the design and operational assumptions incorporated in the certification process (and their potential violation in practice). Development and testing of methods for further developing precursor measures reflecting more elaborate models of the evolution of hazardous states, and discovery and characterization of newly emerging hazards through both hypothesis-free mining for anomalies and hypothesis-driven analysis and modeling, appear to be important areas for future R&D funded by agencies such as FAA, NASA, and NSF, and by industry. REFERENCES Crayton, L., C. Hackworth, C. A. Roberts, and S. J. King. 2017. Line Operations Safety Assessments (LOSA) in maintenance and operations: Final report. DOT/FAA/AM-17/7. Office of Aerospace Medicine, Federal Aviation Administration. https://libraryonline. erau.edu/online-full-text/faa-aviation-medicine-reports/AM17-07.pdf. Das, S., B. Matthews, A. N. Srivastava, and N. C. Oza. 2010. Multiple kernel learning for heterogeneous anomaly detection: Algorithm and aviation safety case study. Proceedings of the ACM SIGKDD Knowledge Discovery and Data Mining Conference (pp. 47–56). FAA (Federal Aviation Administration). 2013. Fatigue risk management systems for avia- tion safety. Advisory Circular 120-103A. https://www.faa.gov/documentLibrary/media/ Advisory_Circular/AC_120-103A.pdf. FAA. 2015. Safety management systems for aviation service providers. Advisory Circular 120-92B. https://www.faa.gov/regulations_policies/advisory_circulars/index.cfm/go/ document.information/documentid/1026670. FAA. 2019. Continuous airworthiness maintenance program guidance and policy. Notice No. 8900.516. https://www.faa.gov/documentLibrary/media/Notice/N_8900.516.pdf. FAA. 2020. Air traffic organization occurrence reporting. Order 7210.632A. https://www.faa. gov/documentLibrary/media/Order/JO_7210.632A.pdf. Flight Deck Automation Working Group. 2013. Operational use of flight path management systems: Final report of the Performance-based Aviation Rulemaking Committee (PARC) and the Commercial Aviation Safety Team (CAST). https://nbaa.org/wp-content/uploads/ aircraft-operations/safety/Final_Report_Recommendations.pdf. Gariel, M., A. N. Srivastava, and E. Feron. 2011. Trajectory clustering and an applica- tion to airspace monitoring. IEEE Transactions on Intelligent Transportation Systems 12(4):1511–1524. Klinect, J. R., P. Murray, A. Merritt, and R. Helmreich. 2003. Line Operations Safety Audit (LOSA): Definition and operating characteristics. In Proceedings of the 12th Interna- tional Symposium on Aviation Psychology (pp. 663–668). The Ohio State University. Marais, K., and M. Robichaud. 2012. Analysis of trends in aviation maintenance risk: An empirical approach. Reliability Engineering and System Safety 106:104–118.

IDENTIFYING EMERGING TRENDS 75 The MITRE Corporation. 2021. Predictive Analytics for Aviation Safety: Project Status Update. National Civil Aviation Review Commission. 1997. Avoiding Aviation Gridlock and Reducing the Accident Rate: A Consensus for Change. Performance-based Operations Rulemaking Committee and Commercial Aviation Safety Team Flight Deck Automation Working Group. 2013. Operational Use of Flight Path Manage- ment Systems. Federal Aviation Administration, U.S. Department of Transportation. Rankin, W. 2007. MEDA investigation process. Aero Quarterly, Q2. https://www.boeing.com/ commercial/aeromagazine/articles/qtr_2_07/article_03_1.html. USDOT (U.S. Department of Transportation). 2019. FAA Needs to Improve its Oversight to Address Maintenance Issues Impacting Safety Issues at Allegiant Air. Report AV 2020013, December. Office of Inspector General. USDOT. 2020. FAA Has Not Effectively Overseen Southwest Airlines’ Systems for Managing Safety Risk. Report AV 2020019, February. Office of Inspector General. USDOT. 2021. FAA Lacks Effective Oversight Controls to Determine Whether American Airlines Appropriately Identifies, Assesses, and Mitigates Aircraft Maintenance Risks. Report AV 2022004, October. Office of Inspector General. Walker, G. 2017. Redefining the incidents to learn from: Safety science insights acquired on the journey from black boxes to digital data monitoring. Safety Science 99. Weick, K., and K. Sutcliffe. 2015. Managing the Unexpected, 3rd ed. Wiley. White House Commission on Aviation Safety and Security. 1997. Final Report to President Clinton. https://irp.fas.org/threat/212fin~1.html.

Next: 5 Potential Emerging Aviation Safety Hazards and Future Identification »
Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes Get This Book
×
 Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Commercial aviation safety in the United States has improved more than 40-fold over the last several decades, according to industry statistics. The biggest risks include managing safety in the face of climate change, increasingly complex systems, changing workforce needs, and new players, business models, and technologies.

TRB Special Report 344: Emerging Hazards in Commercial Aviation—Report 1: Initial Assessment of Safety Data and Analysis Processes is the first of a series of six reports that will be issued from TRB and the National Academies of Sciences, Engineering, and Medicine over the next 10 years on commercial aviation safety trends in the U.S.

READ FREE ONLINE

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!