Enhancing BioWatch Capabilities Through Technology and Collaboration Proceedings of a Workshop (2017) / Chapter Skim
Currently Skimming:
3 Recommendations from the Government Accountability Office Report and the Department of Homeland Security Response
3 Recommendations from the Government Accountability Office Report and the Department of Homeland Security Response
Pages 13-42
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 13... ...
• Review available technological and non-technological capabilities for biosurveillance and reporting, including autonomous systems, as well as the assessment and testing of such systems, with a focus on the BioWatch program. Presenting the findings in Biosurveillance: DHS Should Not Pursue BioWatch Upgrades or Enhancements Until System Capabilities Are Established (GAO, 2015)
|
From page 14... ...
The only factor that GAO is against, said Persons, is ineffective government. Since 2003, Persons explained, DHS has focused on acquiring an autonomous detection system to replace the current BioWatch Generation 2 (Gen-2)
|
From page 15... ...
He noted that DHS officials told GAO that the system's operational objective is to detect catastrophic attacks, which DHS defines as attacks large enough to cause 10,000 infections, and they stated that the system is able to meet this objective. However, while DHS has commissioned some testing of the system's performance characteristics, department officials told GAO they have not developed technical performance requirements that would enable them to interpret the test results and draw conclusions about the system's ability to meet its operational objective.
|
From page 16... ...
designed to produce estimates of key performance characteristics, including sensitivity, or to support conclusions about the types and sizes of attack the system can reliably detect. Persons explained that the outdoor demonstration, performed by the Naval Surface Warfare Center Dahlgren Division, involved releasing a simulant for one of the BioWatch threat agents and showed that the Gen-2 technology could successfully detect this simulant in an open-air environment.
|
From page 17... ...
In the absence of technical performance requirements for Gen-2, DHS officials said they have used modeling and simulation studies, commissioned from multiple national laboratories, to link test results to conclusions about the system's ability to detect attacks. However, said Persons, these studies were not designed to directly and comprehensively assess Gen-2's operational capabilities, and none of the studies that were
|
From page 18... ...
They also analyzed ranges of hypothetical system sensitivities rather than incorporating the results of the four key tests of the performance characteristics of Gen-2. DHS commissioned additional modeling and simulation studies for the purpose of selecting sites for Gen-2 collector units.
|
From page 19... ...
Based on those conclusions, GAO recommended that DHS should not pursue upgrades or enhancements to the Gen-2 system until it • establishes technical performance requirements, including limits of detection, necessary for a biodetection system to meet a clearly defined operational objective for the BioWatch program by detect ing attacks of defined types and sizes with specified probabilities; • assesses the Gen-2 system against these performance requirements to reliably establish its capabilities; and • produces a full accounting of statistical and other uncertainties and limitations in what is known about the system's capability to meet its operational objectives. Finding 3 During a June 4, 2013, meeting conducted in collaboration with the National Academies, GAO identified eight best practices for developmental testing of binary threat detection systems.2 For the most part, said Persons, the BioWatch program partially aligned with these best practices when it was testing the Gen-3 system (see Table 3-1)
|
From page 20... ...
He noted, too, that DHS responded positively to the recommendation to incorporate best practices and has done so with its technology development projects, including those aimed to enhance BioWatch Gen-2 capabilities. Finding 4 The fourth finding, which Persons characterized as a miniaturized technology assessment, was that PCR is the most mature technology available for an autonomous detection system, but there are still uncertainties about
|
From page 21... ...
The GAO report noted that while autonomous detection may provide benefits that include reduction in casualties and clean-up costs and greater cost efficiency, the potential benefits of an autonomous system for BioWatch depend on specific assumptions, some of which are uncertain. In 2014, a National Academies workshop titled Technologies to Enable Autonomous Detection for BioWatch: Ensuring Timely and Accurate Information for Public Health Officials explored several technologies that could be used in an autonomous BioWatch system (IOM and NRC, 2014)
|
From page 22... ...
DISCUSSION The discussion began with moderator Gerald Parker, associate vice president for public health preparedness and response, Center for Innovation in Advanced Development & Manufacturing, Texas A&M University Health Science Center, crediting the BioWatch program for bringing together public health, emergency management, law enforcement, and intelligence at the local, state, and federal levels to learn how to deal with what at first is low-confidence information that increases in confidence as more information comes into the system once a BAR is declared. "This is probably one of the best examples of how a program and a technology has driven that collaboration in understanding how to deal with low-confidence information in a confident way," said Parker.
|
From page 23... ...
BioWatch's testing program looked at the independent performance of its collectors, and that, he explained, entailed conducting a whole system test against a killed agent to examine the entire process, from collection through analysis and identification, and a separate operational demonstration on a military base using a simulant to show that the system could detect and identify it. Nonetheless, Walter said, BioWatch was "deemed as unreliable by the GAO report because we had never conducted a live agent test in an operational environment to prove the system
|
From page 24... ...
Matthew Davenport, program manager, DHS S&T, added that he sees the GAO report as directed more toward DHS, not BioWatch, in that BioWatch has operational requirements that set the stage for what DHS should be doing in the research and development environment and how it should approach testing the technologies it develops for deployment in the field. In the case of BioWatch, said Brinsfield, no one seems to be able to say what the optimal test for BioWatch should be, in part because the science does not yet exist to say what the perfect metric should be for detecting a biological agent in a way that would protect a certain percentage of the population.
|
From page 25... ...
" asked Gunn. Mark Buttner, professor, Department of Environmental and Occupational Health, School of Community Health Sciences, University of Nevada, Las Vegas, said Las Vegas did not embrace BioWatch at first, but since it has the city believes the program is a great success.
|
From page 26... ...
and asked why user testing was missing at such a late stage of development; in most technology development efforts, users are involved along the way. He also suggested that a deep organizational communication study examining how these systems are used in different cities, why false positives happen, how the cities handle false positives, and other aspects of process and outcome could yield some important insights that would improve the overall program.
|
From page 27... ...
Concluding the discussion session, Brinsfield said, "What I get loud and clear from this discussion is that while we lack the perfect test and clearly lack the ability to do the perfect test, one way we could look at some of those issues is to figure out what data we have within the program that we could release to some of the partners and see what studies they would like to run on their own data." She noted there will be jurisdictional issues to resolve for that to happen, but she did not see that as a major obstacle given the partnerships that exist between the BioWatch program and the local public health communities. DHS RESPONSE TO THE GAO RECOMMENDATIONS AND AN ASSESSMENT OF GAPS IN CURRENT BIOWATCH CAPABILITIES Providing a response of how the BioWatch program has addressed GAO's recommendations, Walter first said that the GAO investigation was something to be taken advantage of because it is always beneficial to have someone outside of the program and who does not have a vested interest in any particular aspect of the program to look at the way things are done and identify places that can be improved.
|
From page 28... ...
• Develop and document the new program performance measure, modeling, and outcomes, and provide a description of uncertainties and limitations associated with BioWatch program measures. With regard to the fourth GAO recommendation, which was to use best practices to inform test and evaluation actions, Walter said BioWatch has and will continue to adhere to acquisition management guidance provided by DHS, and is ensuring the recommended best practices are incorporated into its approach and documentation.
|
From page 29... ...
Though the national laboratories came up with the Pd-10K metric, they concluded that there is no significant difference between Fp and Pd-10K performance measures when the new, experimentally derived Dugway Proving Ground sensitivity values for limits of detection are used versus the sensitivity values used in current BioWatch models and simulations. The national laboratories noted that Pd-10K is not a meaningful metric for indoor facilities and subways because attacks that large would likely infect the entire facility or system and thus any collector architecture would detect it.
|
From page 30... ...
Once this tool is optimized to BioWatch's satisfaction, said Walter, it will be used whenever possible to determine the Pd-10K value for all BioWatch jurisdictions and report system performance using both Fp and Pd-10K. When possible, BioWatch will also use the experimentally measured sensitivity values from the Dugway Proving Grounds test, along with confidence intervals, for all applicable modeling and simulation.
|
From page 31... ...
When Gunn suggested it might be appropriate to reappraise every jurisdiction according to a planned timetable, Walter said that would be a good idea, but the program lacks sufficient funds to do so. Brinsfield added that if the local BioWatch program partners want a systematic r eappraisal, the funds to do so would have to be requested in future budget requests.
|
From page 32... ...
Gordon-Hagerty asked Walter if there was a plan to expand the BioWatch system and if so, if there is a standard procedure for adding new jurisdictions that want to participate in the program. Walter replied that there is not a plan and that the BioWatch program needs to create a standard operating procedure on how it would expand using the deployments it has made for various national events such as the Super Bowl or political conventions as the model.
|
From page 33... ...
With regard to addressing some of the issues GAO raised about developmental testing, Davenport said that S&T has funded work at the U.S. Army Edgewood Chemical Biological Center (ECBC)
|
From page 34... ...
, where he worked 31 years prior to joining DHS, the procedure for any new system that might be developed was to first develop the requirements for that system -- what its function will be, what its performance characteristics will be, and how that performance would be measured. "That is not just for aircraft carriers or nuclear submarines," said Fischer.
|
From page 35... ...
He added that the BioWatch program office is now developing an operational requirements document that would be used to develop key performance parameters and metrics. "Without those documents, there are no requirements," said Fischer, who noted that within 1 year, the answer to that question will then be yes.
|
From page 36... ...
Smith said that Georgia's experience with BioWatch has been a very positive one and that Georgia has come a long way since the early days of the program, when there was no plan for a response in the event of a BAR. The BioWatch national meetings have acted as great sources of information that she then uses to educate her stakeholders on the BioWatch advisory committee, whose 70-some members include officials from five public health jurisdictions and other partners.
|
From page 37... ...
Gunn agreed that data and information are the critical pieces of the larger BioWatch system. A gap that was obvious to her was that DHS might not be getting all of the critical local, and perhaps state, information given the current decentralized data system, and even if that information does get to DHS, it may have gotten out to the local public by that point.
|
From page 38... ...
Mark Buttner said that deploying detectors indoors in arenas and convention centers is something his community would like to see. The challenge from a BioWatch advisory committee's perspective, though, is what to do with a BAR when a private company stakeholder needs to interpret that BAR and make decisions.
|
From page 39... ...
Such information might allow him to model and anticipate the impact of a prolonged power outage, high temperatures, and the socioeconomic and health status of the population, then deploy limited resources to areas where they would have the greatest impact. Although such detailed demographic information about the distribution of chronic illnesses or other factors that would make someone more susceptible to a biological agent may not exist today, it should in the near future and would help guide a response not only to a BAR but also to emerging infectious diseases such as Zika.
|
From page 40... ...
Modeling was one item, and what will be needed is to determine which models are useful to state and local public health officials before, during, and after an event. Another request Brinsfield heard was the need for more tools and technology, particularly with regard to data collection.
|
From page 41... ...
RECOMMENDATIONS FROM THE GAO REPORT AND THE DHS RESPONSE 41 are validated, and how they are presented to the user. "Creating another six platforms will not necessarily get us where we need to go," said Brinsfield.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.