Transformation in the Air: A Review of the FAA's Certification Research Plan (2015)

1

A Review of the FAA Research Plan

In response to the FAA Modernization and Reform Act of 2012, the FAA Office of NextGen prepared a 10-page research plan¹ for the Next Generation Air Transportation System (NextGen) that was completed in February 2014 and approved by FAA management in April 2014. (Because the FAA has other research plans covering other activities, this report will refer to the “February 2014 Research Plan” throughout this report.) In response to the request to review the research plan, the National Research Council created a committee that gathered data from the FAA, congressional staff, industry, and other sources to assist in its review. The committee received briefings by the individual that generated the February 2014 Research Plan and was able to have dialogue with the relevant FAA officials and discuss the management guidance that went into generating it.

The FAA Modernization and Reform Act refers to methods to improve confidence in the certification of new technologies. True confidence requires ownership by the various stakeholders in the application and usage of NextGen capabilities. (Chapter 3 of this report addresses examples of successful projects that have helped stakeholders gain confidence in NextGen implementation.)

The transition of technologies into the National Airspace System and the generation of the associated procedures, regulations, and certification processes is a major challenge for the FAA. One of the problems is that improvements in avionics systems are occurring at a far more rapid pace than the procedures, regulations, and certification processes. Another issue is that avionics systems are becoming relatively cheaper whereas the certification costs are not. When the results of research are handed over for certification, a whole new process begins where the resulting new equipment must be designed, built, and then certified. Different functions of the FAA are required to be engaged in those processes—from airworthiness, to operational specification approval, to training, to certifying new air traffic procedures—and provide the interface with the industries producing the hardware and the operators that use the system.

Certification of the new technology is not as important as the approval of the operational capability of that technology and its ultimate implementation in the National Airspace System. The many stakeholders play a major role, from the airlines and other users buying, installing, training, and using the new capabilities, to the operators of the National Airspace System having sufficient training, procedures, regulations, and policies to take advantage of the technology.

____________________

¹ FAA, Research Plan: Methods and Procedures to Improve Confidence in and Timeliness of Certification of New Technologies Into the National Airspace System, Final, Office of NextGen, Washington, D.C., February, 2014; reprinted in Appendix A.

The users and supporters of the National Airspace System are a very broad mix of stakeholders from airlines to the military to general aviation, the manufacturers of the air and ground equipment, and multiple labor organizations, all with different and sometimes conflicting interests and expectations. The operation of the National Airspace System affects the lives of people around the world in terms of travel, commerce, and national security. This in turn presents the FAA with a complex and dynamic set of challenges. All of these users and stakeholders have a significant impact on the scope and type of research that the FAA must conduct.

An effective plan for research on methods and procedures to improve both confidence in and the timeliness of certification of new technologies for their introduction into the National Airspace System should capture the strategic and vision-oriented expectations of the entire FAA organization and its stakeholders. The details of the translation of the plan from vision, to objectives, to tasks, to implementation of outcomes and operations in the real world are fundamental to the success of the program. The plan would be a high-level description of the FAA research planning process that includes the following elements:

• A description of the strategic and prescriptive value of a research plan;
• A description of the value of the expected content of such a plan; and
• An explanation of the expected outcomes from executing the plan.

The following goals would be addressed in a comprehensive research plan:

• Enhancing timeliness,
• Improving confidence,
• Adopting a total system perspective,
• Acknowledging user adoption/operational transition,
• Addressing overall approval as well as certification,
• Increasing integrated accountability by researching critical dependencies and defining a clear and achievable outcome, and
• Integrating emerging technologies upfront.

Instead of a high-level description of the FAA research planning process, the committee concludes that the February 2014 Research Plan is more of a high-level task plan for incrementally developing a detailed research plan over the next 5 years. In other words, it is a plan for developing a plan. It fails to address the full scope of research necessary to meet the direction from Congress.

The research plan states that it represents “the FAA strategy for conducting research on methods and procedures to improve the confidence and timeliness of certification of new technologies.” While this objective is well summarized in introductory material in the plan, the plan includes only limited discussion of processes, programs, and procedures to achieve this objective. Also missing is the presentation of an integrated approach or an end-to-end process that would ultimately result in a cohesive, comprehensive, and integrated plan for transitioning certified technologies into actual end-user capabilities in the National Airspace System.

The committee is concerned that the FAA February 2014 Research Plan cannot improve timeliness or be effective for the following reasons:

• It assumes a traditional definition of “research.” Research in the traditional sense can take years to produce results.
• The schedule presented fails the timeliness test—witness the time taken just to write this 10-page plan that presents an approach that will deliver a “report” in the last quarter of 2018 that gets the FAA ready to develop an implementation plan. The plan implies that the FAA will be starting to “plan” how to be more timely when the bulk of the NextGen technology has already been delivered. The committee can only conclude that this timeframe is not what Congress had in mind when tasking the FAA with developing a research plan.

• The plan includes neither a flow diagram of tasks and milestones, nor does it have any mapping of a holistic plan that includes both ground and air.
• The plan focuses on system design without addressing facility and service approval processes (Figure 1.1), missing the opportunity to research process improvements that might deliver near- and mid-term capabilities in a timely manner. Any improvements to timely delivery will accrue to capabilities well outside the NextGen mid-term (i.e., 5-7 years).
• The plan does not discuss research on critical dependencies that could increase integrated accountability, such as an illustration of linkages and interdependencies of the approval process elements that are listed in Figure 1.1.
• The plan does not explain that this plan is one part of a larger set of documents that describe the FAA’s implementation of NextGen. The plan misses the opportunity to methodically outline the current processes so they can be reviewed for improvement. Such an outline could have been provided by reference, and the holistic view and mapping to the basic objectives of the outline might have identified high-benefit, quick-return improvement topics. Further, it could have addressed the integration of lessons learned into ongoing and future programs.

FIGURE 1.1 National Airspace System ground-system approval process overview. SOURCE: Federal Aviation Administration, Research Plan: Methods and Procedures to Improve Confidence in and Timeliness of Certification of New Technologies Into the National Airspace System, Final, Office of NextGen, Washington, D.C., February 2014.

• The plan shows no recognition of the implementation of the evolutionary improvements that have been and are currently being made in the National Airspace System and how any proposed new technology may be introduced. The plan does not make clear how the research in this plan is being used to accelerate the use of technology found in existing certified hardware and procedures that might benefit the National Airspace System.
• The plan does not reference other agency reports, plans, and resources that inform and frame research to improve confidence in and timeliness of certification of new technologies. These include the NextGen Implementation Plan,² which sets the context and goals for delivery of NextGen capabilities; the National Aviation Research Plan,³ which details investment of FAA’s research budget, some current elements of which are relevant to timeliness and confidence of certification; the National Airspace System Enterprise Architecture,⁴ the comprehensive blueprint being used to build NextGen; the NextGen Priorities Joint Implementation Plan,⁵ which lays out priorities for implementation through 2018, developed jointly with the NextGen Advisory Committee; and the Navigation Procedures Initial Implementation Plan (NAV Lean) Report,⁶ which identifies ways that the FAA will streamline the processes necessary to implement new procedures.
• The plan also does not discuss how its research relates to the responses prepared by the FAA to Sections 215 (Certification Standards and Resources) and 312 (Aircraft Certification Process Review and Reform) of the FAA Modernization and Reform Act of 2012—which address other aspects of certification (aircraft, equipage, procedures). The plan does not indicate how these efforts together will improve timeliness and confidence.
• The plan could enhance the FAA’s ability to gain confidence in software and hardware systems through testing and statistical methods, but the relative proof of confidence is proving to the user (and the public) that implementation will produce economic, safety, security, or other benefits. However, the plan does not include these elements:
  • — Details for how validation and verification would occur.
  • —Performance-based metrics (for example, fuel burn, delays, and carbon emissions) common to many other FAA documents that allow stakeholders to determine what the goals are and how they are being achieved.
• Even though the plan itself states, “In preparing this report, we have considered certification process for two basic categories of technologies . . . . (1) those associated with aircraft; and (2) those associated with ground based systems and air traffic control (ATC),” the plan only addresses the second item, ground-based systems, and not the first, despite the fact that aircraft systems are a vital, contentious, and complex issue, and procedures certification remains a major bottleneck for implementation. Committee discussions with FAA representatives revealed that aircraft systems were omitted from the February 2014 Research Plan at the direction of FAA management. No explanation was provided for this decision. Not including aircraft systems or procedures makes it unlikely that the plan by itself would address all the elements necessary to improve the timeliness and effectiveness of the certification and implementation of technologies into the National Airspace System. The focus of the plan on only one segment of the National Airspace System is a major deficiency. The tightly integrated nature of the system—whereby the ultimate success of implementing the program depends on the integration of air and ground capabilities with operational procedures—requires that all three segments be addressed by a research plan. Omitting aircraft systems and procedures from the research plan makes the plan unresponsive to the request from Congress.
• The plan’s focus on certification of technologies without sufficient integration with and approval of the

____________________

² FAA, NextGen Implementation Plan, August 2014, https://www.faa.gov/nextgen/library/media/NextGen_Implementation_Plan_2014.pdf.

³ FAA, National Aviation Research Plan, September 2013, https://www.faa.gov/about/office_org/headquarters_offices/ang/offices/tc/about/campus/faa_host/rdm/media/pdf/2013%20NARP.pdf.

⁴ National Airspace System Enterprise Architecture.

⁵ FAA, NextGen Priorities Joint Implementation Plan, October 2014, http://www.faa.gov/nextgen/media/ng_priorities.pdf.

⁶ FAA, Navigation Procedures Initial Implementation Plan (NAV Lean), June 1, 2011, http://www.faa.gov/nextgen/media/SIGNED%20Initial%20NavLean%20Implementation%20Plan%201%20June%202011.pdf.

• operations they will enable, does not provide any mechanisms for examining systems and operations that require a close integration of air and ground technologies.

• A major missing element in the plan is a holistic analysis of the FAA’s many stakeholders who play a variety of roles in the processes leading to certification and ultimate use of NextGen improvements. The plan does not discuss in depth the ability to organize the stakeholders or direct them with a view to creating timely implementation.
• The plan does not mention efforts at global harmonization, one of the most important reasons for a systemic approach to NextGen. The FAA may be taking actions to assure a systemic approach globally, but the plan does not mention this in the proposed research or in present ongoing FAA efforts.
• The plan also fails to reference Acquisition Management Guidance, as required by the FAA’s Acquisition Management System (AMS) policy, as well as an outline of how it will be applied. The AMS provides mandatory requirements for procurement, deployment of products and services, and in-service management of fielded capabilities and, therefore, is directly applicable to the transition of the NextGen technology to operational application.
• The plan does not follow a normal technical reporting format and could have included more information that would have enhanced the public’s understanding of and confidence in the FAA’s work. In areas where information was provided, more detail would have established greater insight and confidence that the FAA can deliver NextGen in a timely fashion.
• The plan also does not discuss any possible organizational changes to improve efficiency or effectiveness. Although reorganization can create additional problems while sometimes failing to solve others, it may be useful and necessary in limited cases to help clarify lines of authority.

COMPONENTS OF AN EFFECTIVE FAA RESEARCH PLAN

The goals discussed below would normally be addressed as part of a more comprehensive research plan.

Enhance Timeliness

A valid research plan would characterize the steps involved in developing, certifying, and transitioning technology into operation in the National Airspace System. The characterized process would then serve as the basis for defining areas of improvement.

Improve Confidence

Process improvements that lead to immediate realization of benefits will increase confidence in the FAA’s ability to fully implement technology modernization. A successful design, even though tested and operationally evaluated, will not deliver benefits until it has transitioned into the operation and users have adopted it. A successful plan would characterize the technical modernization activities from an end-user perspective, for example, citing these programs as enhancements or a gap filler or new capabilities. Including an integrated perspective and presenting the targeted impact or improvements to the user would instill confidence that many well-executed disparate program components are manageable as an integrated program and truly improve the baseline.

Adopt a Total System Perspective

A plan focusing on research aimed at all aspects of approval, including accelerating operational transition, cross-organizational collaboration, and user adoption, would give a total system view that includes integrated testing, validation activities, and cybersecurity in operationally representative environments. These key technologies and others, such as human factors, need to be rigorously addressed. A viable plan would present how technologies will be integrated and addressed throughout the implementation and adoption process. While critical and chal-

lenging, integration needs to occur as an integral part of programs and integrated system development without disrupting the overall safety and quality posture.

Global harmonization—the need to harmonize aircraft systems and ground systems as much as possible across the entire world—is one of the most important reasons for a systemic approach to NextGen. Airlines and other operators of intercontinental aircraft cannot be expected to meet multiple and disparate levels of mandates for equipping their airplanes. The European Union’s Single European Skies Air Traffic Management Research program (SESAR) and NextGen have similar goals and mandates, and although the FAA may be taking actions to assure a systemic approach globally, a complete research plan would mention these efforts either in proposed research or in present, ongoing FAA efforts.

FAA initiatives aimed at aircraft and aircraft equipage certification would also be noted in a comprehensive research plan. The FAA’s initiative focusing on streamlining the implementation of procedures in the National Airspace System with its NAV Lean project, for example, followed the AMS process for implementation as described in the report Navigation Procedures Initial Implementation Plan (NAV Lean).⁷

Acknowledge user Adoption/Operational Transition

Operational transition and user adoption were identified as challenges that drive gaps between the FAA’s documented descriptions of NextGen and what is being accomplished in a recent comprehensive and independent assessment of NextGen by MITRE.⁸ These gaps, which contribute to the different perceptions within the community about the amount of progress the FAA has made on NextGen, need to be addressed.

Address Overall Approval as well as Certification

System certification alone does not guarantee approval for use in all applications, nor does it guarantee user adoption. The committee concludes that a complete plan would go beyond traditional technology-based certification.

Increase Integrated Accountability

An effective research plan would include these two components to increase integrated accountability:

• Research critical dependencies. Focusing on understanding critical approval dependencies, intersecting organizational responsibilities, linkages between decision points and decision makers, and barriers to successful operational transition and user adoption can produce an illustration of linkages and interdependencies of the approval process elements that are listed in Figure 1.1.
• Define a clear and achievable outcome. Clear articulation of a common understanding of the path to a successful implementation of NextGen capabilities across the stakeholder spectrum could lead to a better ability to streamline processes and to an improved confidence in the FAA’s ability to deliver NextGen.

upfront Integration of Emerging Technologies

A better plan would address methods for integrating emerging technologies early and throughout the system development, certification, and implementation process. An approach might include addressing the integration of the topics cited in the FAA plan, including software assurance, cybersecurity, and human factors, while maintaining comprehensive safety and compliance standards.

____________________

⁷ FAA, Navigation Procedures Initial Implementation Plan (NAV Lean), 2011.

⁸ “NextGen Independent Assessment and Recommendations,” MITRE Project No. 0214DL01-IF, Center for Advanced Aviation System Development, October 2014.