Assessment of the Practicality of Pulsed Fast Neutron Analysis for Aviation Security: Summary (2002)

The federal government has invested considerable funds in recent years to develop new technologies to detect explosives and drugs. One such technology is the pulsed fast neutron analysis (PFNA) inspection system. PFNA uses a collimated, nanosecond-pulse-width beam of monoenergetic fast neutrons to excite the nuclei of common elements in bulk materials to produce gamma-ray emissions (Brown, 1994; Gozani, 1995). The primary excitations of interest for explosives and drugs detection are gamma-ray emissions from carbon, nitrogen, chlorine, and oxygen. While PFNA can detect explosives and drugs hidden in vehicles and in large cargo containers, it has major technical limitations in its depth of penetration into large containers and in its ability to characterize certain explosives. PFNA also has a number of practical limitations, including large size and weight, the need for radiation shielding, and regulatory and safety issues associated with nuclear-based technologies¹

In 1997 the Federal Aviation Administration (FAA) funded Science Applications International Corporation (SAIC) with approximately $935,000 to demonstrate the feasibility of using PFNA to search for "small" explosives in air cargo containers. The objective of the program was to learn if the PFNA system for cargo inspection that had been developed by SAIC for the Advanced Research Projects Agency (now Defense Advanced Research Projects Agency, or DARPA) could be exploited for FAA's air cargo application. The FAA then requested the National Research Council (NRC) to form a panel to evaluate the potential of PFNA for use in an airport environment. The Panel on Assessment of the Practicality of Pulsed Fast Neutron Analysis for Aviation Security and the present report were the outcome of that request.

Congress directed the FAA to continue to support the development of PFNA technology and appropriated $2.5 million in Fiscal Year 2001 for this purpose.² Similarly, Congress instructed the U.S. Customs Service (USCS) to cooperate with the Department of Defense (DoD) in the development of PFNA and to test this technology for use in contraband (e.g., drugs and currency) detection. The FAA agreed that the U.S. Customs Service’s interest in PFNA is similar to FAA interests and supported the inclusion of that service in this study.

In this report, the capabilities of PFNA are compared with the capabilities of currently available x-ray–based equipment for detecting explosives and drugs. The panel also compares PFNA capabilities with those expected from the future development of current x-ray equipment.

The panel reviewed the physical basis for PFNA technology and evaluated its applicability to the inspection of full cargo containers. PFNA uses neutrons as the interrogating or probing radiation. For nonhydrogenous cargos, the neutrons have a range that permits them to penetrate deep into full cargo containers and return a signature that contains information on the elemental composition of the cargo. PFNA can determine the carbon, nitrogen, and oxygen content in an object, and the relative amounts of these elements can then be used to help discriminate explosive from non-explosive materials. The detection algorithms exploit the high nitrogen and oxygen content of most explosives and the high chlorine content and high carbon-to-oxygen ratio in certain drugs.³

The strengths of PFNA are its ability to penetrate deep into nonhydrogenous cargos, its ability to discriminate on the basis of the elemental composition of the interrogated material, and its ability to penetrate metal objects within the cargo. The weaknesses of PFNA are a limited depth of penetration for hydrogenous cargo, the need for a large radiation-shielded building in which to conduct the cargo scans, the logistics of moving the cargo to and from the scanning facility in a timely fashion, the long time required to detect a small-explosive-threat quantity in a cargo container, the reluctance of airport workers to accept a neutron-generating facility within airport boundaries, and the reluctance of the public to have their passenger bags irradiated by neutrons.

The panel reviewed the laboratory-demonstrated explosives-detection performance for PFNA. The January 1997 tests in Santa Clara, California,⁴ provided a reference performance benchmark and a proof of principle for PFNA detection of explosives. The exact types of explosive material that can appear in these two categories are defined in the Code of Federal Regulations and the Federal Register (58 FR 47804). The results of the test conducted by the FAA in January 1997 showed an overall detection rate of 75 percent and an estimated false alarm rate of 13 percent. At that time, SAIC stated that the existing DARPA PFNA system, with further hardware modifications, should be capable of achieving a detection efficiency of 90 percent or better for explosives of the size required for certification⁵ of luggage inspection systems, a throughput rate of two containers per hour, and a false alarm rate of less than 10 percent of containers inspected.

The FAA provided $11 million to upgrade the PFNA hardware and detection algorithm and to conduct a more detailed blind test of PFNA for the detection of explosives in containerized cargo⁶ in October 2000. The promised improvement in threat detection was not achieved in this subsequent testing. The October 2000 test provides the latest and most statistically valid performance metrics for PFNA. This test used an improved blind test protocol (Fleisher, 1998a) for containerized cargo that was developed with the intention of comparing the performance of existing conventional x-ray cargo-scanning systems with that of PFNA. Using a strict adherence to the preestablished blind test protocol, the FAA evaluation of the October 2000 containerized cargo tests yielded an average probability of detection (P_d) of 50 percent and a probability of false alarm (P_fa) of 20 percent (Row 1 of Table 3 from Ancore Corporation, 2001a). The average PFNA measurement time for cargo containers was 90 minutes, but this time showed a strong correlation with the cargo type, ranging from 48 minutes for electronics to 133 minutes for printed matter (Bell and Green, 2001). The single fastest scan time was 25 minutes; the slowest was 197 minutes (Section 1.2 of Ancore Corporation, 2001a). These scan times are not driven by the fundamental physics of the PFNA detection but are the result of trade-offs made in the implementation. Trade-offs include the selection of accelerator current, target pressure, and target cooling, which affects neutron source strength, and the collimator and exposure room, which affects the signal-to-noise ratio of the detectors.

The panel also reviewed the existing laboratory-demonstrated drugs detection performance for PFNA. The Eurotunnel test series is indicative of the ability of PFNA to detect large-threat quantities of drugs or explosives in full cargo containers. These results are well summarized by the findings of the final April 1996 double-blind testing. In that test, the double-blind protocol was used for the PFNA operators and operations. In addition to fixed and limited-threat objects, the cargo types were restricted to eight prearranged categories, and baseline performance data could be gathered on the cargo types.

A cargo manifest was always available to the PFNA operators, because manifest verification was one of the Eurotunnel evaluation metrics. This permitted the PFNA operators to select appropriate scan parameters (time) for the neutron attenuation of the cargo. This practice was representative of the anticipated Eurotunnel operational protocol. The availability of these data was unlikely to have affected the ratio of detection to false alarms (P_d/P_fa detection metric⁷) for the PFNA system. Fast and direct scans were performed from both sides of the cargo container. False positives, true positives, false negatives, and true negatives were recorded. In order to take PFNA operator experience into account, metrics were gathered for trained Eurotunnel operators as well as for experienced SAIC operators.

The results of these tests (Lacey, 1999) showed a very high probability of detection for both SAIC and Eurotunnel operators. The probability of a false alarm was statistically indistinguishable from zero. The most difficult cargos were water, paper, and rice, in which the threats were located in the center of the highly neutron-attenuating cargo. In summary, the Eurotunnel PFNA testing was judged to be a success by the Eurotunnel operations. Issues of cost (both operational and initial capital investment), cargo throughput, and value of detection were evaluated. Eurotunnel operations decided not to implement a PFNA detection capability in the Eurotunnel but to preserve the operational compatibility with a future implementation if circumstances change.

The most important detection characteristic for the inspection of cargo containers is the ability of the probing radiation to penetrate the container. Existing x-ray systems (computed tomography, transmission, and backscatter) with an electron endpoint potential of less than 450 kV have a photon energy too low to penetrate cargo containers. High-energy (>450 kV) x-ray–based systems lack sufficient contrast and are useful only for anomaly detection or inspection of empty trucks. Another important attribute of an explosives-detection system is its ability to detect explosives while maintaining a low false alarm rate. Existing data for explosives detection (FAA-conducted blind tests) indicate that the x-ray systems are not effective in the inspection of full cargo containers to detect explosives and that random selection would be equally useful.⁸ It is clear from these data that PFNA is the only existing detection system worthy of being considered for explosives detection in containerized cargo.

Another task for the panel was to compare the demonstrated and projected capabilities of PFNA with those of currently available x-ray systems for the inspection of break bulk cargo.⁹ The first and most important consideration is a comparison of the detection performance of the PFNA and x-ray systems.

For a PFNA system, it is assumed that the cargo was inspected in the original container and was repackaged into small containers for inspection. Repackaging is done to prevent an "opaque" or "shield" alarm where material near the container perimeter may result in unreliable characterization of material in the interior. Taking this repackaging into account, the most appropriate performance metric to use for PFNA detection of explosives in break bulk cargo is a 61 percent probability of detection with a 9 percent probability of false alarms. Selection of these metric gives Ancore credit for what it learned from the blind test but also requires it to detect Category A explosives. For x-ray computed tomography (CT) in break bulk cargo, where a shield alarm is considered to be a false alarm (because it fails to clear the bag), the system performance metric has a much larger (poorer) P_fa.¹⁰ Both performance metrics should be considered to be unacceptable for any system applied to the detection of explosives in break bulk, owing to the inadequate probability of detection and the long time required to resolve the high level of false alarms.

Although PFNA has the greatest potential for detecting explosives in containerized cargo, the PFNA technology not only fails to meet the current FAA cargo detection metric¹¹ but also comes with a high implementation cost. The high cost comes not only from the time and money required to construct the initial inspection facility but also, and more importantly, from the logistics of the inspection process, where it would have a severe impact on the air cargo industry.

The operational impact is so severe that only when the alternative is an unpacking and hand inspection of the cargo would the industry be expected to voluntarily consider PFNA. Even then, the industry might consider alternatives, such as cargo-only flights for much of its cargo transportation. Because neither a hand inspection nor the application of PFNA would be expected to meet the cargo detection metric for passenger aircraft, it would be very difficult to construct a cost/benefit analysis to justify implementing either inspection system.¹²

If an airline had to select one of the two break bulk cargo explosives inspection technologies (x-ray CT or PFNA) as operated in the automated mode without any specified alarm resolution procedure, most airlines would select the x-ray CT inspection because it is much more easily installed, is currently available, and has a documented baseline throughput that has been demonstrated rather than optimistically projected. If, however, alarm resolution is involved in the technology selection criteria, the PFNA technology would almost certainty be selected, because x-ray CT alarm resolution procedures would have to be applied to about 33 percent of the scanned items.

In the case of PFNA, opaque regions,¹³ which can be interpreted as shield alarms, could be eliminated by rescanning in a new, less densely packed container

configuration, and only 9 percent of the overall containers would be expected to give a nonshield-based false alarm that would require hand inspection. If the PFNA alarm could target the specific bulk piece part for a directed trace and hand inspection, only about 0.3 percent of the overall break bulk parts would require a hand inspection.

For the detection of drugs in full cargo containers, the large, low-energy backscatter or transmission x-ray systems can only detect anomalies, but this method can be very effective in finding drugs in empty containers. Because approximately 50 percent of commercial containers crossing the U.S.-Mexico border are empty (Hoopengardner et al., 1994), even the low-energy x-ray systems have a role in drugs detection. When we restrict our attention to filled containers with large heterogeneous materials, the low-energy x-ray systems typically do not have the penetrating power to be effective for scanning containerized cargo, and the high-energy x-ray systems do not have sufficient contrast to permit an operator to identify and resolve anomalies.

PFNA has performed better in this role because it targets the elemental composition of the drugs. High-energy photon systems have a reasonable performance based on anomaly detection when the scan image is compared against a cargo manifest. Both systems have a perceived problem with safety given the possibility that potential smugglers will attempt to conceal themselves in the container. Heartbeat detectors or thermal imaging may be required before the drug inspection is conducted to preclude the irradiation of humans.

Both high-energy photon (>450 kV) and PFNA systems can be subject to countermeasures by drug smugglers. The smugglers can either avoid the channels protected by the systems or alter the drug insertion schemes to defeat these detection systems. Both x-ray systems and PFNA can be implemented in a system that can be relocated.¹⁴ A PFNA system that can be relocated is more difficult to implement than a high-energy x-ray system because of regulatory requirements and the desire to use a preinstalled building and platform rather than to rely solely on distance for radiation shielding.¹⁵ PFNA offers the promise of a more flexible response to changes in the insertion mode, but this flexibility needs to be tested against a set of possible countermeasures and quantified in terms of time to implement an altered detection algorithm and detection effectiveness. High-energy photon systems will have better public acceptance and are expected to have an easier installation process and a faster scan time.

The decision of Eurotunnel to implement photon scanning systems while maintaining the capability to upgrade to a PFNA system would likely prefigure the decision by most airports and ports of entry if they were asked today to implement a contraband (drugs or large quantities of explosives) detection system for cargo containers. Once operational data (on, for example, installation time, scan time, maintenance issues, or worker acceptance) are available from several installed PFNA systems, the decision might be different.

The primary condition is that the PFNA system must be modified to detect Category A explosives; to remove the opaque region in the center of full LD-3 cargos, including those containing printed matter or seafood; and to detect the FAA explosive threats in palletized cargo.

The second condition is that the scan time for a typical cargo container (LD-3) must be reduced to no more than 5 minutes.

The third condition is that PFNA must either reduce its false alarm rate to less than 1 percent (while maintaining the FAA cargo detection metric) or be associated with an acceptable false alarm resolution procedure that can clear a cargo container of a false alarm within 5 minutes.

The fourth condition is that a federal agency must take the lead in establishing an approved PFNA installation procedure that meets all federal and applicable state regulatory requirements (Ryge and Bar-Nir, 1992) (e.g., safety, radiation generation, and waste disposal) to ensure that PFNA installation issues do not unduly influence deployment decisions.

Each of these conditions must be met before PFNA can be considered a viable technology for airport installation. Once they are met, the limited space for a separate inspection facility might still restrict use in some airports.

Current x-ray–based inspection systems for cargo do not have the potential for being improved to the point where they can be effectively used to detect small-threat quantities of explosives in full containers. So, while the potential for PFNA to be capable of eventually detecting threat quantities of Category A explosives is not clear, PFNA has better potential for cargo scanning than any other explosives-detection technology currently being considered.

The main role for PFNA is the detection of explosives in full cargo containers. The following sections offer findings and recommendations for this bulk

explosives-detection mission and prioritize supporting research activities.

For containerized (LD-3) cargo scanning of FAA-defined explosive threats, PFNA has demonstrated a more effective detection (P_d and P_fa) capability than any other technology. However, this detection ability is not at the FAA-specified performance level for containerized cargo.

The most recent PFNA blind testing showed that for one threat class, Category A, the system failed to unambiguously detect any threat objects. Thus, PFNA cannot currently meet the FAA cargo detection metrics. There is no reason to move explosives testing for a future PFNA system to an airport environment unless and until that system can adequately detect this important threat class.

Because the difficulty the PFNA system has in detecting Category A explosive would be clear to anyone investigating the publicly available reports on PFNA, and because Category A explosive is known to be readily available to terrorist organizations, terrorists would be expected to easily exploit this vulnerability. Accordingly, fielding a PFNA system in an operational capacity at an airport would not be feasible until this deficiency is addressed. Deploying this system, which has no Category A detection capability, without a complementary detection system would not meet systems goals and could give a false sense of security. However, the system may still provide value when compared to currently deployed equipment.

In addition, the most recent testing revealed that for hydrogenous cargos, an opaque zone makes up nearly one-third of the volume of the standard LD-3 cargo container. In this zone, neither Category A nor Category B explosives in FAA-defined threat quantities could be reliably detected. This opaque zone is a significant shortcoming of the current PFNA technology and must be corrected before a future PFNA system can be considered for testing in an airport environment.

A critical limitation of PFNA is the requirement for a separate building to house the scanning equipment. The separate building is required because of the shielding needed for safety reasons and because facilities of this size could in any case not be integrated into existing terminals. Moreover, the combination of cost and facility size makes it unlikely that most airports would be receptive to building even one PFNA facility, and building the multiple PFNA facilities that may be required for efficient throughput would currently be impossible. Additionally, transporting containerized cargo to and from this separate building for testing would probably impose considerable delays in cargo transportation on passenger aircraft.

If the scan times could be significantly reduced from those currently demonstrated (an average of 0.67 containers per hour) and if only containers from unknown shippers were scanned,¹⁶ the delays added just from transporting the containers to and from the inspection facility would probably push much of the cargo currently carried on passenger aircraft to all-cargo aircraft or, in some cases, to surface transport. If the scan times cannot be significantly reduced¹⁷ below those currently demonstrated, then it is even more likely that most cargo from unknown shippers would be transferred to all-cargo aircraft.

Current blind testing protocols used in cargo testing, although carefully designed with test time and cost in mind, are not sufficient when the physics underlying the detection algorithms are not clearly understood. The current PFNA prototype has two different detection algorithms, one based on a neural net and one on a discriminate analysis. In neither case has the sensitivity of the detection metric to the physically meaningful parameters been clearly established, nor have the robustness and stability of the detection algorithms been sufficiently addressed.

The record from previous PFNA blind tests shows considerable variation in detection metrics that is at least in part due to the inadequate sample size and lack of diversity in the cargos used in the testing protocol. During the post-test analysis of the latest blind test, a special detection algorithm specifically for fish was developed to account for differences between previous calibration cargos and the test cargos. In addition, significant differences in the density of the paper in the test cargos caused problems with the detection algorithm; the variation in the nitrogen content of the pentaerythrite tetranitrate (PETN) explosive caused detection problems; and the increased density and high packing fraction in some of the cargos resulted in opaque regions where the detection algorithms were insufficient. In summary, the variations in the cargo composition/packing and in the explosives composition are still causing significant changes in the detection algorithms.

The interrelationship of the blind test cargos with the cargos used to refine the detection algorithm underlines the need for a careful match between test cargos and actual airline cargos and the need for complete coverage of the expected cargo composition. The problems that have arisen due to an insufficient match between bags used in the EDS checked baggage certification testing and the passenger bags seen in operational airport environments provide a lesson that should result in careful attention being paid to the cargo test protocol.

The FAA has considered the range of previously demonstrated explosive threats in setting the current test protocol. However, in the absence of a large variation in test configurations, the FAA also needs to take into explicit account the physics underlying the detection basis of the PFNA technology and the implemented detection algorithms when it sets up test protocols. Cargo threat-insertion methods should permit the selection of innocuous materials surrounding the explosive with the spoofing of the detection algorithm in mind. Multiple threats in a single container should also be part of this testing. This approach goes significantly beyond the test rigor used for x-ray–based explosives-detection systems but is reasonable considering both the limited range of cargo testing that can be carried out within the cost constraints and the problems uncovered with the previous passenger-bag-specific EDS test protocol.

These insertion technologies should only be used in PFNA blind testing, and no record of the insertion methods or of the resulting PFNA detection data should be retained for, or available to, any neural net training. All neural net training data should have a completely independent pedigree (including being constructed by a different set of people). The level of sophistication in the threat insertion should be indicated by the test developers and correlated with the blind test performance. Great care must be taken in the detection algorithm development and in the testing protocol to provide the neural net with statistically unbiased input samples. In such cases, the user must define both input and output spaces in the algorithm so that effectively no room exists for "creative" discovery by potential terrorists outside those previously set specifications. This can be very difficult to implement in the case of explosives detection, where the terrorist controls, to some extent, the input space.

Performance of the PFNA system must be assessed in laboratories rather than in an airport environment. The logistical and safety issues surrounding the use of actual explosives in an airport testing protocol drive this requirement.

Quantification of the probability of false alarms in PFNA detection requires a large and diverse cargo test suite and will have to be conducted in an airport environment. The range and diversity of air cargo and the expense of putting together blind test cargo prevents the gathering of statistically significant data under laboratory test conditions. Airport testing must be completed before any airport operational deployment is planned. This testing should only be conducted with a mature combination of PFNA system hardware and detection algorithms that are expected to have some value in a deployed mode, even if the deployment is only considered in the future under significantly different threat conditions. A two- to threefold increase in the rate of false alarms experienced in the field over that seen in laboratory certification testing—as was the case for some of the x-ray CT checked-baggage explosives-detection systems—is not acceptable.

Category A explosives are a proven threat, and the existence of an opaque region in standard aircraft cargo containers renders PFNA of little or no value in certifying the absence of a threat to the level required by the FAA detection criteria. PFNA is still generally in the development phase, except for the ability to detect Category A explosives, where it may still be in the research phase.

Thus, priority should be given to moving clearly from a research phase to an engineering phase by demonstrating proof of concept for the detection of Category A explosives and by the elimination of the opaque region. These two remaining problems must be solved jointly rather than separately before proceeding with any hardware upgrades to improve throughput, particularly since design trade-offs may be involved in addressing these problems.

If PFNA were able to meet the explosives-detection criteria, the strong selling point for this nuclear technology would be its potential flexibility for adapting to new threats using existing hardware and updated detection algorithms. This potential is implied by its element-specific detection capability but has not yet been demonstrated. In fact, the apparently strong dependence of the current explosives-detection algorithm on the nitrogen content of an explosive suggests that this multielement detection capability is not being fully exploited by the current explosives-detection algorithms. While it may be that the natural variation in the explosive composition of current threats prevents other element-specific detection metrics that do not focus on nitrogen from playing an important role, this multielement detection capability should be quantified and its potential importance in detecting other non-nitrogenous–based explosives and nonchlorinated contraband materials (e.g., currency, mustard and nerve chemical agents, and hazardous industrial chemicals) should be quantified. In addition, the time needed to adapt the detection algorithm to these new contraband materials should also be quantified.