3
Role of TARAM Within the FAA’s Overall Safety Oversight System
As decribed in Chapter 2, the Transport Airplane Risk Assessment Methodology (TARAM) process is a subset of the Federal Aviation Administration’s (FAA’s) Monitor Safety/Analyze Data (MSAD) process, defined in FAA Order 8110.107A.1 The MSAD process is designed to promote an improved continued operational safety (COS) methodology by incorporating a data-driven, risk-informed approach for safety assurance and safety risk management.
TARAM AND MSAD IN THE CONTEXT OF SAFETY OVERSIGHT
FAA Order 8110.107A, describes how MSAD and TARAM align with policy and guidance by citing specific FAA regulations, policy, and guidance to provide context to its relationship. However, the publications referenced by the Order have had notable revisions following this Order’s last revision in 2012. For example, FAA Order 8110.107A, section 6-1 bullet c., directly quotes FAA Order 8040.4, Safety Risk Management Policy, original revision, as a basis to substantiate the use of MSAD, quoting “The FAA shall use a formal, disciplined, and documented decision-making process to address safety risks in relation to high-consequence decisions affecting the complete life cycle.”
However, FAA Order 8040.4, revision B, does not include the quote found in FAA Order 8110.107A, which was removed in FAA Order 8040.4, revision A, in 2012. Seemingly contrary to the quote from FAA Order 8110.107A, FAA Order 8040.4B explicitly defines steps of the Safety Risk Management (SRM) process and requires the use of a Hazard Identification, Risk Management, and Tracking (HIRMT) tool. There are conceptual overlaps of the FAA Order 8040.4B process and HIRMT, to MSAD and TARAM. In a related fashion, FAA Order 8000.369C, Safety Management System (SMS), refers to FAA Order 8040.4B and the HIRMT tool several times as the source for SRM guidance. Depending on the reading, one might conclude that FAA Order 8040.4B and HIRMT supersedes FAA Order 8110.107A and, consequently, MSAD and TARAM. Therefore, within policy and guidance there exists a disconnect as to the role of MSAD and TARAM relative to HIRMT, SRM, and SMS.
Similarly, the Seattle Aircraft Certification Office (ACO) has created the Transport Airplane Safety Manual, released September 1, 2021. This manual was approved by the Seattle ACO Branch Manager and provides further details on that ACO’s practices regarding the application of MSAD and TARAM. It references FAA Order
___________________
1 Federal Aviation Administration, 2012, “Monitor Safety/Analyze Data (MSAD),” Order 8110.107A, Washington, DC: Aircraft Certification Service (AIR).
8110.107A and the TARAM Handbook but, also, does not disambiguate the seeming conflict of FAA Order 8040.4B. The manual’s preface states that it “provides guidance on how to perform risk assessment for transport airplanes, develop a recommendation to the Corrective Action Review Board (CARB) whether a condition is unsafe, and select appropriate timing for the control program for transport category airplanes.” This scope is reiterated in its introduction as well. However, the purview of the Seattle ACO does not include all transport airplanes. “Transport Category Airplanes” include a variety of commercial and business airplanes that are certified to 14 CFR Part 25 regulations and that are monitored by other ACOs, such as Wichita and Atlanta. Therefore, it is unclear how the Seattle ACO can produce and approve guidance with a scope as broad as “transport airplanes” without concurrent approval from other ACOs or higher-level FAA authority.
Finding: It is unclear how MSAD and TARAM align with current safety policy and guidance as the regulations and publications referenced by FAA Order 8110.107A have been revised in ways that are relevant to TARAM.
Finding: The scope of ACO guidance needs to clearly define the limits of its application, otherwise, disjointed application of TARAM can cause confusion within the FAA.
Recommendation 1: Within 18 months of receipt of this report, the Federal Aviation Administration (FAA) should update its policy and guidance regarding the application of Transport Airplane Risk Assessment Methodology and Monitor Safety/Analyze Data processes so that they align with other FAA orders that describe the agency’s overarching safety policies and processes for Safety Management Systems and Safety Risk Management.
The existence of this disconnect is also contrary to the TARAM Handbook’s own processes description, which states on section 6.3, page 34, that it “will monitor the results of the analyses and associated safety decisions to ensure that the methodology and guidance reflect the risk-management policy” of the FAA Aircraft Certification Service. It also states that the handbook “will change based on changing agency goals and expectations” (page 35) and that it “will also make changes based on lessons learned during application” (page 35). However, the TARAM Handbook has not been updated in over a decade. The FAA is aware of this, pointing out that efforts to update the order were begun, but never concluded, and any suggestions for improvement have not yet been implemented. The committee also learned in its engagement with the FAA that the agency now has only one recognized subject-matter expert for TARAM, following the recent retirement of the aviation safety engineer (ASE) who first developed TARAM. This lack of robust expertise for this process within the FAA may have contributed to the inability to keep the handbook up to date and perpetuated the disconnects.
Finding: A single recognized subject-matter expert for TARAM is not sufficient to maintain, train, facilitate, and advocate for an institutional practice that is vital to the practice of ensuring aviation safety.
Recommendation 2: Within 6 months of receipt of this report, the Federal Aviation Administration should formally designate multiple employees within its organization as experts for the Transport Airplane Risk Assessment Methodology (TARAM) process. These experts should be responsible for the advocacy, maintenance, and training of TARAM guidance and processes, including updating the TARAM Handbook to reflect, among other things, current National Transportation Safety Board accident rates.
TARAM IN THE CONTEXT OF RULEMAKING
Notwithstanding the above-noted lack of clarity of the role of MSAD and TARAM within the FAA safety policy, the committee addressed the statement of task given to it, taking the MSAD and TARAM guidance documents at face value—the results of the committee’s analysis are presented in the remainder of this section and subsequent chapters of this report.
TARAM as a method for monitoring and analyzing the performance of the in-service fleet (both constant failure rate and wear-out failures) for transport airplanes is used by ACOs, independent of the Type Certificate holder or operator, although it analyzes data that come from them. There are no explicit agreements for the TARAM process to require the support of these external organizations with necessary and timely data. Therefore, the TARAM analyst is not guaranteed consistent or timely data, and the effort to acquire this information adds time to the analysis. To attain safety of flight, it takes the entire industry: original equipment manufacturers (OEMs), suppliers, operators, pilots and flight crews, and maintenance personnel with FAA-designated personnel oversight.
In producing the TARAM process, only the FAA was involved, despite its dependence on outside data. Typically, when rulemaking is planned, or processes are developed that require a relationship between organizations, the FAA draws on those involved to not only support the effort but also provide valuable input to the process. Examples of this successful partnership are the Aircraft Systems Harmonization Working Group and the Avionics Systems Harmonization Working Group.
Another example of this is highlighted in AC 39-8, Continued Airworthiness Assessments of Powerplant and Auxiliary Power Unit Installations of Transport Category Airplanes. The Advisory Circular, issued September 3, 2003, describes Continued Airworthiness Assessment Methodologies that the FAA Engine and Propeller Directorate (EPD) and the Aircraft Certification Service may use to identify unsafe conditions and determine when an “unsafe condition is likely to exist or develop in other products of the same type design” before prescribing corrective action in accordance with 14 CFR Part 39.
This originated from a 1991 Aerospace Industries Association chartered working group to develop methods to identify, prioritize, and resolve safety-related problems occurring on aircraft engines. The group included OEMs, airframe, and FAA personnel who looked at 10 years of events. The results of the efforts have been used by the EPD since 1994. In 2001 the committee was reformed to update the database from 1992–2000. The assessment methodologies are being used across the industry.
Finding: TARAM Aviation Rulemaking Committee dated June 22, 2015, was never established.
TARAM IN THE CONTEXT OF TYPE CERTIFICATION
Of particular interest to the committee has been the degree of decoupling in practice between the safety assessment process performed during Type Certification of an aircraft and subsequent COS considerations for that aircraft. It is generally understood that the long-established design safety processes have served the industry well and have resulted in significant and sustained advances in safety. The FAA regulation that provides the basis for aircraft systems design safety analysis is documented in 14 CFR 25.1309 along with the complementary guidance material found in Advisory Circular 25.1309-1A and Draft Advisory Circular 25.1309-Arsenal found in Task 2—System Design and Analysis Harmonization and Technology Update, TAEsdaT2-5241996. To supplement this FAA regulatory material, the industry developed additional guidance material found in SAE ARP4761. Airplanes certified to 14 CFR Part 25 apply this policy, guidance, and industry standards in the form of a Safety Assessment Process to their systems to demonstrate compliance to 14 CFR 25.1309. While the application of this process is limited in scope to an evaluation of systems, its resulting data could prove beneficial as a component of evaluation within TARAM. It is recognized that the TARAM Handbook discusses the relationship of 14 CFR 25.1309 to post-certification risk assessment but does not discuss how to apply it.
However, as discussed in detail in later chapters, potential interfaces between processes may aid TARAM. As, the severity definitions provided in Order 8040.4B, Appendix C, bring the post-certification severity definitions more closely in line with 14 CFR Part 25 definitions, significant benefit could be attained by developing the alignment between TARAM, Type Certification, SRM, and last, SMS. This aspect of the TARAM process will be discussed further in the subsequent chapters.
Finding: The lack of explicit agreements to provide necessary data for TARAM analysis can drive inconsistent evaluations and increase the time to complete the analysis.
Recommendation 3: Within 6 months of receipt of this report, the Federal Aviation Administration (FAA) should convene an industry harmonization rulemaking advisory committee to develop regulatory guidance material within 18 months for establishing detailed continued operational safety (COS) agreements. These agreements should address the monitoring and analysis of operational safety performance of transport category airplanes to support the required input for constant failure rate and wear-out analyses in the Transport Airplane Risk Assessment Methodology (TARAM). These agreements should be established between the FAA and airplane type certificate holders, manufacturers, their suppliers, and aircraft operators. The agreements should explicitly define the monitoring and analysis process, including the type of data collected and the collection process necessary, to improve the completeness, accessibility, quality, and maintenance of TARAM input data for supporting the COS process.
Chapter 4 provides details on the TARAM input data mentioned in this recommendation, and Chapter 5 provides details on the COS analysis process—specifically the TARAM analysis process. Chapter 6 provides details on the COS decision-making process utilizing the TARAM outputs.
