Response to Congress
In keeping with the Statement of Task for this study, the panel reviewed the FAA's deployment of technologies to improve aviation security, including explosives-detection equipment and HULDs. Given the complexity of the terrorist threat and the short response time mandated by Congress, the deployment has been successful. Considering that hardware had to be deployed immediately, the FAA did not have sufficient time to follow a systems analysis and design protocol to achieve an optimum systems architecture. Therefore, at this stage, the lack of comprehensive data for evaluating the effectiveness of the deployed equipment is not surprising. However, the FAA must now address the issues of collecting performance data and systems integration.
Explosives-detection equipment and HULDs are part of a TAAS (total architecture for aviation security), and the panel strongly believes that they should be evaluated in that context in this and in future NRC studies. Although both the FAA and its contractors have adopted some aspects of this systems approach, they have not characterized the optimal deployment scenario. With the systems approach, improvements in security can be characterized by an SEF.
Based on available data and current understanding of the various elements of the system, the panel was able to represent a notional analytical summary of security enhancement as a function of current and projected system performance and concluded that substantial technical improvements in security have already been made. As more data are collected and analyzed, Figure 10-1 can be refined and improved.
The SEF also includes the resolution of detected alarms, which requires that operators make decisions based on the full capability of the hardware. For this reason, operator training could have a significant effect on the SEF. Based on the statistically significant probability that an alarmed bag containing an explosive device might be "cleared" by an operator, the panel determined that an order of magnitude increase in the SEF could result if all alarmed bags cleared by an operator (during the alarm-resolution process) were placed in a HULD (see Chapter 10).
The following overarching recommendations address the four issues raised by Congress.
1. Assess the weapons and explosives-detection technologies available at the time of the study that are capable of being effectively deployed in commercial aviation.
This study focused on explosives-detection technologies. While it is conceivable that some of these technologies could also be used for weapons detection, this topic was not addressed in this report.
Bulk Explosives-Detection Equipment
The vast majority of bulk explosives-detection equipment deployed is the FAA-certified In Vision CTX-series EDS. Most of the performance data on this equipment was generated during laboratory testinglargely certification testingat the FAA Technical Center. Certification tests, however, only reflect the ability of the equipment to detect a bag that contains an explosive, and the detection rates are based on bag-alarm rates. That is, an explosive is considered to be detected if the alarm is set off for the bag containing the explosive, even if the alarm is triggered by a nonexplosive object in the bag. Certification testing does not measure alarm resolution and does not include testing in the operational environment of an airport, making it difficult to evaluate the operational performance of explosives-detection technologies considered for deployment. In the panel's opinion, some of the unanticipated problems encountered with the CTX-5000 SP in the field can be reasonably related to the limitations of certification testing. Under current certification guidelines, equipment certified in the future may encounter similar problems.
During certification testing, the FAA should, whenever possible, measure both true detection rates (i.e., correctly identifying where the explosive is when an
alarm occurs) and false-detection rates (i.e., an alarm triggered by something other than the explosive in a bag that contains an explosive). The FAA should also include the ability of explosives-detection equipment to assist operators in resolving alarms (including in an airport) as part of certification testing. Alarm resolution should be included in the measurement of throughput rate, detection rate, and false-alarm rate.
Trace Explosives-Detection Devices
TEDDs are widely used in airports, but no comprehensive methodology has been developed to evaluate their effectiveness, such as standard test articles or instrument and operator requirements. Because no standard test articles for TEDDs have been demonstratedand because of the resultant inability to separate instrument and operator performanceit is not possible to measure the performance of TEDDs.
The FAA should develop and implement a program to evaluate the effectiveness of deployed trace explosives-detection devices. This evaluation should include measurements of instrument and operator performance, including measurements in the deployed (i.e., airport) environment.
Computer-Assisted Passenger Screening and Positive Passenger-Bag Matching
CAPS appears to be an effective method of screening passengers to identify selectees for further security measures, such as bag matching or bag screening. The panel believes that CAPS is an effective method of focusing resources (personnel and equipment) on high-risk passengers that does not impede the operation of the air carriers. No quantitative measure of the effectiveness of CAPS has been demonstrated, however. The panel believes that testing the performance of CAPS will be necessary for its long-term viability.
The 1997 White House Commission on Aviation Safety and Security recommended that the FAA begin implementation of full passenger-bag matching. The FAA interpreted this recommendation to mean that passengers who were either randomly selected or who were identified through a profiling system (e.g., CAPS) should be subject to bag matching. The panel anticipates that PPBM combined with CAPS will be an effective tool for improving aviation security. However, under the current deployment scenario, when a selectee changes planes (or simply deplanes during a stopover) at a connecting airport, PPBM is not repeated.
Computer-assisted passenger screening (CAPS) should continue to be used as a means of identifying selectee passengers whose bags will be subject to positive passenger-bag matching (PPBM), screening by explosives-detection equipment, or both. PPBM combined with CAPS should be part of the five-year plan recommended below. Passengers designated as selectees at the origination of their flights should remain selectees on all connecting legs of their flights. Within six months, the FAA should develop and implement a method of testing the effectiveness of CAPS.
2. Determine how the technologies referred to in paragraph (1) could be used more effectively to promote and improve security at airport and aviation facilities and other secured areas.
Progress in the Deployment of Aviation Security Equipment
As directed by Congress in Section 305 of the FAA Reauthorization Act of 1996 (PL 104-264) and recommended by the White House Commission on Aviation Safety and Security, the FAA has initiated the deployment of bulk and trace explosives-detection equipment, as well as the implementation of CAPS and PPBM. These measures have significantly improved aviation security. Nevertheless, the FAA/SEIPT is behind schedule in its deployment of aviation security equipment. Congress provided $144.2 million in the Omnibus Consolidated Appropriations Act of 1997 for the purchase of commercially available security screening equipment, and the FAA/SEIPT planned to deploy 54 certified explosives-detection systems and 489 trace-detection devices by December 1997. The FAA also planned to fully implement CAPS by December 1997. Once it became apparent that these goals could not be met, the FAA set a new goal of deploying 54 certified EDSs, 22 noncertified bulk explosives-detection devices, and 489 trace-detection devices by December 31, 1998. The FAA also planned to implement CAPS fully by December 31, 1998. As of January 1, 1999, more than 70 certified EDSs, six noncertified bulk explosives-detection devices, and 366 TEDDs had been installed in airports. In addition, 10 HULDs have been deployed to three airlines for operational testing. Thus, the FAA has made significant progress towards achieving its updated goals. Nevertheless, there is not a sufficient amount of deployed equipment to screen every passenger bag or, in some cases, even every selectee bag without interrupting passenger flow.
The panel concluded that the FAA/SEIPT, the airlines, airports, and associated contractors have gained significant experience from the initial deployment of security equipment and procedures, and the current implementation of security equipment does not appear to have interfered unreasonably with airline operations. Most importantly, in the collective opinion of the panel, the deployment of security equipment has improved aviation security. The panel believes that continued emphasis on, funding of, and deployment of security equipment will further enhance aviation
security. Future deployments should be more efficient if they are based on the experience from the initial deployment.
The U.S. Congress should continue to fund and mandate the deployment of commercially available explosives-detection equipment through the FAA/SEIPT. Continued deployments will increase the coverage of domestic airports and eventually provide state-of-the-art security equipment systemwide. Further deployments can improve aviation security in the short term and provide the infrastructure for mitigating potential threats in the long term.
Human operators are integral to the performance of all deployed explosives-detection equipment. Because fully automated explosives-detection equipment will not be developed in the foreseeable future, particularly with respect to alarm resolution, human operators will continue to be immensely important to realizing the full potential of deployed security hardware. The TAAS analysis presented in this report quantifies the impact of the operator on the SEF. Certification testing of explosives-detection equipment, however, does not include testing of human operators. Current testing only defines the operational capability (or performance) of the equipment.
It has been demonstrated that the introduction of an operator (who is needed to resolve alarms) decreases the effectiveness of deployed explosives-detection systems. FAA test results indicate that, even if explosives-detection equipment correctly identifies a potential threat, the operator can make an incorrect decision and ''clear" a bag with a bomb in it. The Pd of the equipment/operator combination is, therefore, lower than the Pd of the equipment alone.
The FAA should institute a program to qualify security-equipment operators to ensure that the human operator/explosives-detection system (EDS) combination meets the performance requirements of a certified EDS. This program should include the definition of operator performance standards and a means of monitoring operator performance. The FAA should implement this program within six months of receipt of this report.
Measuring Operational Performance
Because of the paucity of operational data for deployed explosives-detection equipment, the panel found it impracticable to characterize the deployment status of security equipment and processes quantitatively. The data are insufficient both for the equipment and for operator performance, and no quantitative measures of the effectiveness of the total security system (e.g., TAAS) were provided to the panel. The majority of data focused on subsystems, such as bulk explosives-detection systems. A thorough assessment of equipment and system performance requires well defined performance metrics and the collection of data. The panel concluded that the FAA has not defined adequate performance metrics for security subsystems (e.g., TEDDs) or for the TAAS.
The FAA should make a concerted effort to define operational performance metrics for security subsystems and for the total architecture for aviation security (TAAS). The FAA should also create an action team in the next six months to systematically collect operational data, which should be used to optimize the TAAS, as well as to identify and correct substandard performance of equipment and operators. The data collected would also provide insights into the deployment and use of equipment in the future.
Measuring Security Enhancement
Besides the lack of operational data and the apparent lack of performance metrics for the total system, no overall measure of security enhancement has been defined. Because the primary performance measure for the TAAS is, of course, protection against explosive threats, the critical factor for assessing the performance of the TAAS is the leakage of false negatives (i.e., bags that contain bombs). The panel defined improvements in performance, or the SEF, as the number of simulated bombs that defeat a baseline security system divided by the number of simulated bombs that defeat the newly deployed system. The panel believes that the SEF is essential to assessing the impact of security equipment and procedures.
The FAA should formulate a security enhancement factor (SEF) for the integrated total architecture for aviation security systems. The SEF should be calculated from data collected during operational testing. Nonclassified SEF measures should be published and used as a project-control and management-control tool. The SEF would provide the FAA with a quantitative measure of the impact of security equipment and procedures.
Five-Year Deployment Plan
TAAS-based decisions involve airport and airline-specific management and costs. Airline and airport buy-in are, therefore, necessary for effective deployment. The FAA has not demonstrated to the panel a long-range (five-year) TAAS deployment plan developed jointly and agreed to by the stakeholders. Thus, to the knowledge of this panel, the airlines have not agreed to a long-term deployment plan that addresses all of the relevant issues, such as operator training, the optimal location of detection equipment, and the
deployment of HULDs. The panel believes that the development of a long-term deployment plan is critical for continued improvements in aviation security.
Within one year, in cooperation with the other stakeholders, the FAA should develop a five-year joint-deployment plan that includes cost, stakeholder responsibilities, quality measures, and other important factors. This plan should be a living document that is formally updated annually. Buy-in from all stakeholders will be necessary for the plan to be effective.
3. Assess the cost and advisability of requiring hardened cargo containers to enhance aviation security and reduce the required sensitivity of bomb-detection equipment.
Two HULDs (both LD-3 size) have passed the FAA blast and shockholing tests and conform to the NAS-3610-2K2C airworthiness criterion. However, U.S. airlines have resisted the deployment of HULDs on the basis of operational concerns. The effects of operational wear and tear on the blast-containment characteristics of HULDs must be tested operationally.
The HULDs that passed the FAA blast test are the size of an LD-3, a container size used on wide-body aircraft. No HULD concepts for narrow-body aircraft have been developed. More than 70 percent of bombing attempts have been against narrow-body aircraft, and 75 percent of the aircraft in service (as of 1994) are narrow-body aircraft. Therefore, before the FAA recommends, mandates, or regulates the use of hardened containers for airline operations, the feasibility of HULDs designed for narrow-body aircraft or, perhaps, other protection concepts will have to be further investigated.
The FAA should continue to support research and development on hardened unit-loading devices (HULDs), including ongoing operational testing. If the FAA recommends, mandates, or regulates the use of HULDs, explosion-containment strategies for narrow-body aircraft, including the development of narrow-body HULDs and cargo-hold hardening concepts, should be investigated. However, the FAA should not deploy HULDs unless they are part of the TAAS joint five-year deployment plan.
4. On the basis of the assessments and determinations made under paragraphs (1), (2), and (3), identify the most promising technologies for improving the efficiency and cost effectiveness of weapons and explosives detection.
The data were not sufficient for a comprehensive assessment of available technologies for improving aviation security. Therefore, at this time, the panel is not able to identify or recommend the most promising technologies for improving the efficiency and cost effectiveness of weapons and explosives detection. If the recommendations in this report are followed, these data will become available for subsequent assessments.