Summary
The Transport Airplane Risk Assessment Methodology (TARAM) is a process for calculating risk associated with continued operational safety (COS) issues in the U.S. transport airplane fleet. TARAM is important because its risk-analysis calculations are used when making determinations of unsafe conditions in transport airplanes, and when selecting and implementing corrective actions. TARAM is used by the Federal Aviation Administration (FAA) for risk analysis and risk management decisions regarding COS. Aircraft certification offices use it to resolve COS issues for transport category airplanes. This methodology is also used by design approval holders, in whole or in part, by agreement with the applicable aircraft certification office. The goal of this study is to assess the TARAM process used by the FAA.
Including a systematic risk assessment methodology in the continued operational safety analysis is important for a comprehensive approach to the overall safety of the transport airplane fleet. TARAM is an initial attempt by the FAA to fill such a role. A healthy safety culture requires commitment to continuous improvement. This report provides recommendations to the FAA to address the gaps and strengthen the TARAM.
ROLE OF TARAM WITHIN THE FAA’S OVERALL SAFETY OVERSIGHT SYSTEM
The TARAM process is a subset of the FAA’s Monitor Safety/Analyze Data (MSAD) process, defined in FAA Order 8110.107A.1 The MSAD process is designed to promote an improved COS methodology by incorporating a data-driven, risk-based approach for safety assurance and safety risk management. The TARAM process has evolved significantly from lessons learned over the past decade to support the MSAD process, which is described in the TARAM Handbook2 and in the FAA Policy Statement PS-ANM-25-05, Risk Assessment Methodology for Transport Category Airplanes dated November 4, 2011.
___________________
1 Federal Aviation Administration, 2012, “Monitor Safety/Analyze Data (MSAD),” Order 8110.107A, Washington, DC: Aircraft Certification Service (AIR).
2 Federal Aviation Administration, 2011, Transport Airplane Risk Assessment Methodology (TARAM) Handbook, PS-ANM-25-05, Washington, DC: Transport Airplane Directorate ANM-100.
TARAM and MSAD in the Context of Safety Oversight
FAA Order 8110.107A describes how MSAD and TARAM align with policy by citing specific FAA regulations and guidance. However, the publications referenced by the order have had notable revisions following its last revision in 2012. For example, FAA Order 8000.369C,3Safety Management System, refers to FAA Order 8040.4B and the Hazard Identification, Risk Management and Tracking (HIRMT) tool several times as the source for Safety Risk Management (SRM) guidance. Depending on the reading, one might conclude that FAA Order 8040.4B and HIRMT supersedes FAA Order 8110.107A and, consequently, MSAD and TARAM. Therefore, within policy and guidance there exists a disconnect as to the role of MSAD and TARAM.
Similarly, the Seattle Aircraft Certification Office (ACO) has created the Transport Airplane Safety Manual in 2021. This manual provides details on ACO’s practices on the application of MSAD and TARAM. It references FAA Order 8110.107A and the TARAM Handbook but, also, does not disambiguate the seeming conflict of FAA Order 8040.4B.
Recommendation 1: Within 18 months of receipt of this report, the Federal Aviation Administration (FAA) should update its policy and guidance regarding the application of Transport Airplane Risk Assessment Methodology and Monitor Safety/Analyze Data processes so that they align with other FAA orders that describe the agency’s overarching safety policies and processes for Safety Management Systems and Safety Risk Management.
The committee also learned in its engagement with the FAA that the agency now has only one recognized subject-matter expert for TARAM.
Recommendation 2: Within 6 months of receipt of this report, the Federal Aviation Administration should formally designate multiple employees within its organization as experts for the Transport Airplane Risk Assessment Methodology (TARAM) process. These experts should be responsible for the advocacy, maintenance, and training of TARAM guidance and processes, including updating the TARAM Handbook to reflect, among other things, current National Transportation Safety Board accident rates.
TARAM in the Context of Rulemaking
TARAM as a method for analyzing the performance of the in-service fleet (both constant failure rate and wear-out) for transport airplanes is used by ACOs, independent of the Type Certificate holder or operator, although it analyzes data that comes from them. There are no explicit agreements for the TARAM process to require the support of these external organizations with necessary and timely data. Therefore, the TARAM analyst is not guaranteed consistent or timely data, and the effort to acquire this information adds time to the analysis.
TARAM in the Context of Type Certification
Of particular interest to the committee has been the degree of decoupling in practice between the safety assessment process performed during Type Certification of an aircraft and subsequent COS considerations for that aircraft. Airplanes certified to 14 CFR Part 25 apply FAA policy, guidance, and industry standards in the form of a Safety Assessment Process to their systems to demonstrate compliance to 14 CFR 25.1309.4 To supplement this FAA regulatory material, the industry developed additional guidance material found in SAE ARP4761.5 While the
___________________
3 See Federal Aviation Administration, 2020, “Safety Management System,” Order 8000.369C, Washington, DC, https://www.faa.gov/documentLibrary/media/Order/Order_8000.369C.pdf.
4 See Cornell Law School, “14 CFR § 25.1309 – Equipment, Systems, and Installations,” https://www.law.cornell.edu/cfr/text/14/25.1309, accessed February 19, 2022.
5 SAE International, 1996, SAE ARP4761: Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne Systems and Equipment, Warrendale, PA.
application of this process is limited in scope to an evaluation of systems, its resulting data could prove beneficial as a component of evaluation within TARAM.
Recommendation 3: Within 6 months of receipt of this report, the Federal Aviation Administration (FAA) should convene an industry harmonization rulemaking advisory committee to develop regulatory guidance material within 18 months for establishing detailed continued operational safety (COS) agreements. These agreements should address the monitoring and analysis of operational safety performance of transport category airplanes to support the required input for constant failure rate and wear-out analyses in the Transport Airplane Risk Assessment Methodology (TARAM). These agreements should be established between the FAA and airplane type certificate holders, manufacturers, their suppliers, and aircraft operators. The agreements should explicitly define the monitoring and analysis process, including the type of data collected and the collection process necessary, to improve the completeness, accessibility, quality, and maintenance of TARAM input data for supporting the COS process.
IMPROVEMENTS FOR THE INPUT DATA TO TARAM
Input data for TARAM is collected from various sources. The quality of the estimated risk depends on the trustworthiness and quality of the input data. While the current TARAM Handbook requires specific input data to assess the risk of failure, the data sources to be used are not always specified. Without clear guidance on what data to use, the TARAM analysis may produce results that lack the consistency and reproducibility required for regulatory purposes. Additionally, some of the required data sources are either outdated or not always available.
To improve the completeness, accessibility, and quality of TARAM input data, the following considerations are needed:
- In line with Recommendation 3, the FAA needs to reach COS agreements with airplane type certificate holders, manufacturers, their suppliers, and aircraft operators to develop an agreed-upon framework by which Aviation Safety Engineers (ASEs) can access, in a timely manner, relevant data in support of TARAM inputs.
- A team of data specialists could oversee all input data to TARAM to allow for a more homogeneous and consistent use of data across ACOs.
- The FAA could implement a periodic independent review process for assessing the quality of TARAM data sources, data access processes, and data mining techniques.
Most data inputs to TARAM lack uncertainty characterization. Uncertainties in the TARAM input data can be characterized by leveraging approaches used in probabilistic risk assessment (PRA) of other complex technological systems such as those for the nuclear power plants6 and space exploration.7 In the current TARAM practice, the dependencies among aleatory uncertainty sources and various system components are modeled by constructing a probabilistic causal chain, considering the randomness associated with the events and conditions included in the causal chain. Based on the FAA briefing regarding the Seattle ACO Transport Airplane Safety Manual, in the current practice of COS decision-making for transport airplanes, risk sensitivity is sometimes studied by examining the impact of varying each input or modeling assumption on the risk outputs. Because some of the TARAM inputs are estimated based on limited empirical data or engineering judgments, the epistemic uncertainty of the estimated TARAM risk outputs induced by the TARAM input uncertainty can be quite large, possibly creating a significant impact on the FAA’s COS decisions. The lack of quantitative treatment of epistemic uncertainty may mislead the COS decision-making.
___________________
6 American Society of Mechanical Engineers and American Nuclear Society, 2008, “ASME/ANS RA-S−2008: Standard for Level 1/Large Early Release Frequency Probabilistic Risk Assessment for Nuclear Power Plant Applications.”
7 National Aeronautics and Space Administration, 2011, Probabilistic Risk Assessment Procedures Guide for NASA Managers and Practitioners, 2nd ed., NASA/SP-2011-3421, Washington, DC: NASA Center for AeroSpace Information.
IMPROVEMENTS TO THE TARAM PROCESS
Various ACOs are using “TARAM worksheets” to conduct the risk analysis for the constant rate and the wear-out failure rate cases as recommneded by the TARAM Handbook. Meanwhile, other ACOs are using another methodolgy referred to as TARA (Transport Airplane Risk Analysis). The Seattle ACO has developed three spreadsheets that cover analysis prescribed in TARA, the constant and wear-out failure rate analysis, and analysis to determine the risk related to maintenance and operational personnel. The FAA could codify these analyses to be consistent across all ACOs.
Improving Systematic Risk Modeling in the TARAM Process
TARAM would benefit from having a model-based approach to treat both common cause failures (CCFs) and functional dependencies. Regarding the treatment of functional dependencies, for instance, the integration of 14 CFR 25.1309 FTs with the TARAM causal chain (converted to Event Trees [ETs]) would help. As part of the Safety Assessment Process in support of the Design Certificate analysis, CCF analysis is “qualitatively” conducted. Based on the Seattle ACO Transport Airplane Safety Manual, treatment of CCF in the current COS decision-making is also qualitative and relies on engineering judgment by the Corrective Action Review Board (CARB). Quantitative CCF analysis, performed under PRA, could be leveraged, evaluated, adjusted (if needed), and when practical be implemented in TARAM.
Recommendation 4: Within 6 months of receipt of this report, the Federal Aviation Administration should evaluate and document its approach to the use of quantitative common cause failure analysis, performed under probabilistic risk assessment, to determine its applicability for the continued operational safety process.
Incorporating Human Reliability Analysis in the TARAM Process
On the human side, recognition needs to be given to the fact that flight, cabin, and maintenance crew all play an important and interconnected role in maintaining safe operations. To ensure operational safety, specific actions undertaken by these crews are relied on; yet, there is no apparent mechanism inside TARAM for properly assessing the reliability of these crews in their appropriate contexts. Recommendation 5 below addresses the human reliability aspects while the software reliability aspects are discussed in the next section.
Recommendation 5: Within 18 months of receipt of this report, the Federal Aviation Administration should initiate and report on an effort to quantify the human performance of flight, maintenance, and cabin crews under the wide range of contexts experienced in civil aviation. This should be a broad-based effort including regulatory agencies, manufacturers, operators, and industry associations. The resultant data set of baseline human capabilities should be regularly maintained and be appropriate for a modern Human Reliability Analysis and used for continued operational safety analyses.
Incorporating Software Reliability Analysis in the TARAM Process
In TARAM, the risk outputs are calculated and presented in spreadsheets. When the scope of TARAM is expanded based on the recommendations in this report, the spreadsheet format may not be practical for timely analysis and decision-making. The computational tools that fit the practical needs in the COS analysis would need to be evaluated and, if any of the existing ones are relevant, they can be adopted for TARAM; otherwise, a new computational tool may need to be developed for TARAM leveraging the existing tools.
Recommendation 6: Within 18 months of receipt of this report, the Federal Aviation Administration should identify or develop and implement methods and computational tools that leverage 14 CFR
25.1309 (SAE ARP4761) compliance for use in conducting the in-service safety process. These methods and tools should take advantage of Development Assurance Level assessments of software/airborne electronic hardware, Fault Tree analysis, and other probabilistic risk assessment methodologies that support software reliability analyses.
Incorporating Uncertainty Analysis in the TARAM Process
The TARAM Handbook and MSAD Order (FAA Order 8110.107A) provide no guidance on uncertainty analysis. The COS decision-making practice, documented in the Seattle ACO Transport Airplane Safety Manual provides a limited-scope sensitivity analysis to study the risk output change when any of the TARAM inputs are varied. This is based on the analyst’s judgment to study how the TARAM outputs could be influenced when each TARAM input (or modeling assumption) is varied individually to a certain value or condition. The uncertainty analysis in the TARAM process has two limitations. First, sensitivity analysis is only executed at the analysts’ discretion, and the procedure has no guidelines for how to determine the range of input values and modeling assumptions or which sensitivity analysis methods to use. Second, conducting sensitivity analysis only is not a substitute for uncertainty quantification.
Recommendation 7: Within 12 months of receipt of this report, the Federal Aviation Administration should establish and document guidance to account for the uncertainties associated with inputs and models used in the Transport Airplane Risk Assessment Methodology process. To the extent practical, quantitative uncertainty analysis should be adopted.
IMPROVEMENTS FOR THE USE OF TARAM OUTPUTS
While TARAM is only one facet of the safety decision-making process, it is an important one. Because of this, it is critical that the Administrator and senior staff are made aware of, and understand, the TARAM results from the analysis of a potential unsafe condition that has significant consequences for transport category airplanes. By maximizing the confidence of the TARAM results, the Administrator will be in a better position to make well-informed determinations to improve commercial airplane safety.
Improving Uncertainty Consideration in TARAM Decision-Making Guidance
Neither the TARAM Handbook nor FAA Order 8110.107A provides guidance on how uncertainty associated with the risk outputs should be considered in COS decision-making. The current practice of uncertainty consideration in MSAD, using the TARAM results, is limited to qualitative considerations. Based on the Seattle ACO Transport Airplane Safety Manual, sensitivity analyses are reported to CARB as part of the TARAM results and considered in COS decisions. The sensitivity analyses are limited to checking the impact of a bounding input value or assumption on the TARAM risk outputs in the one-at-a-time method, rather than checking the aggregated impact of all the dominant uncertainty sources on the risk outputs.
Recommendation 8: Within 18 months of receipt of this report, the Federal Aviation Administration should create a documented protocol addressing how uncertainties associated with Transport Airplane Risk Assessment Methodology outputs should be accounted for in continued operational safety decision-making.
Incorporating Risk Importance Ranking in TARAM Decision-Making Guidance
Research needs to be conducted on the risk importance measure methodology for TARAM. The results of risk importance measure analysis can provide a quantitative way for the identification and prioritization of corrective action alternatives in the COS decision-making.
Recommendation 9: Within 12 months of receipt of this report, the Federal Aviation Administration should enhance the Transport Airplane Risk Assessment Methodology decision-making guidance by incorporating risk importance ranking methods to generate quantitative ranking measures for the prioritization of alternative corrective actions and risk-informed inspections.
Improving Quality of COS Decision-Making Process When Using the TARAM Results
The Seattle ACO Transport Airplane Safety Manual provides guidance as to how the quantitative risk results from TARAM should be combined with other safety considerations in the COS decision-making and that the CARB decision as to whether a condition is unsafe should account for other criteria, including high-visibility events, lessons learned from the past accidents, risk to maintenance and operations personnel, fail-safe design, and qualitative safety criteria.
Recommendation 10: Within 6 months of receipt of this report, the Federal Aviation Administration should document as national guidance how Transport Airplane Risk Assessment Methodology results are to be integrated with other safety principles throughout the continued operational safety decision-making process.
To support the quality of COS decision-making, a review process is required to continuously evaluate (1) the adequacy of the TARAM analysis to generate risk results and (2) the adequacy of the use of the TARAM results in the COS decision-making process. The review process could consist of multiple layers of reviews at different phases of the COS decisions involving various stakeholders to provide evaluations from diverse perspectives.
Recommendation 11: Within 12 months of receipt of this report, the Federal Aviation Administration (FAA) should conduct and document a study to determine the requirements and viability of an independent peer review and quality assurance process for (1) the results from the Transport Airplane Risk Assessment Methodology (TARAM) analysis of significant in-service safety issues and (2) the continued operational safety (COS) decisions resulting from TARAM outputs. Details of the independent peer review and quality assurance process should be documented in the COS agreements between the manufacturers and the FAA.
National guidance for TARAM is contained in the TARAM Handbook and in its associated set of presentation slides that are intended to train ASEs who perform or oversee risk analysis for transport airplanes as part of FAA Order 8110.107 MSAD process. The Seattle ACO also utilizes its own document for further guidance, but this guidance is not national policy or performed uniformly across other ACOs involved in transport airplane COS. The FAA currently has no formal training curriculum or recurrent training schedule for TARAM. The FAA would also benefit from establishing a research group to keep the risk methodologies up-to-date.
Recommendation 12: Within 18 months of receipt of this report, the Federal Aviation Administration should develop and maintain a technical training program for aviation safety engineers and their management who conduct and review Transport Airplane Risk Assessment Methodology analysis. The training should include the concepts of probabilistic risk analysis and the use of risk assessment results in the continued operational safety (COS) decision-making, similar in scope to those used in other federal agencies, to ensure the assumptions and limitations of the probabilistic risk analysis techniques are applied to the COS of commercial airplane operations.
Recommendation 13: Within 6 months of receipt of this report, the Federal Aviation Administration should initiate research and continuous improvement programs in probabilistic risk analysis, including the use of risk assessment results in continued operational safety decision-making.
__________________
