Review of the Department of Homeland Security's Approach to Risk Analysis (2010) / Chapter Skim
Currently Skimming:
5 The Path Forward
5 The Path Forward
Pages 88-114
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 88... ...
5. Adopt strong scientific practices and procedures, such as careful documentation, transparency, and independent outside peer review.
|
From page 89... ...
Decision support requires the ready availability of such resources, to ensure that specific risk assessments are carried out according to the appropriate guidelines and their results are clear to risk managers. Such a staff would also be responsible for the development and periodic revision of risk analysis guidelines.
|
From page 90... ...
, it was not always clear that DHS leadership fully understood the fundamental scientific concerns with the risk analysis methodology chosen, the underlying assumptions used, the data inputs required, or how to use the analysis outputs. Some risk models developed by DHS contractors remain proprietary, which introduces additional problems.
|
From page 91... ...
HITRAC works with key contractors to access technical staff while waiting for the DHS Human Resources recruiting process. Both RMA and HITRAC are actively educating job candidates and working on candidate communications to manage the longer lead times required in recruiting and signing technical talent.
|
From page 92... ...
However, it could also be an indication that RMA is not adding value to DHS's distributed efforts in risk analysis. Discard the Idea of a National Risk Officer The director of RMA suggested to the committee that the DHS Secretary, who already serves as the Domestic Incident Manager during certain events, could serve as the "country's Chief Risk Officer." establishing policy and coordinating and managing national homeland security risk efforts.2 A congressional staff member supported the concept of establishing a Chief Risk Officer for the nation.3 The committee has serious reservations.
|
From page 93... ...
Figure 5-1 shows a spectrum of more traditional risk analysis methods, from the more qualitative or subjective methods on the left to methods that rely heavily on data of high quality, volume, and objectivity on the right. The figure also attempts to capture the rapidly developing capabilities in modeling terrorism and intelligent adversaries, through tools such as attacker-defender (or defender-attacker-defender)
|
From page 94... ...
In general, a risk analysis is intended to combine data and modeling techniques with subject matter expertise in a logical fashion to yield outputs that differentiate among decision options and help the decision maker improve his or her decision over what could be accomplished merely with experience and intuition. Such models can be used to gain understanding of underlying system behavior and how risk events "shock" and change the behavior of the system of interest; to evaluate detection, protection, and prevention options (risk-based cost-benefit analysis)
|
From page 95... ...
. For those many areas where DHS models risk as a function of T, V, and C, the use of non-proprietary models would mean that the best models for each of these elements (T, V, and C)
|
From page 96... ...
, The Department of Homeland Security's Risk Assessment Methodology: Evolution, Issues, and Options for Congress (Masse et al., 2007) , this conceptual framework has evolved in that context through three stages of development: a first stage when risk was generally equated to population; a second stage when risk was, primarily in additive form, assessed as the sum of threat, critical infrastructure vulnerability, and population density.
|
From page 97... ...
This is a particularly important issue in DHS risk-related work, especially for terrorism, where uncertainty is particularly large. As many authoritative studies have noted, it is essential to assess the levels of uncertainty associated with components of the risk assessment and to communicate these uncertainties forthrightly to users and decision makers.
|
From page 98... ...
Until these deficiencies are improved, only low confidence should be placed in most of the risk analyses conducted by DHS. The FY 2009 Homeland Security Grant Guidance describes the DHS approach to risk assessment as (DHS, 2009)
|
From page 99... ...
Risk methods should not prejudge the answers to trade-off questions that are inherently political or preclude input by decision makers and other stakeholders. Based on these concerns, the committee makes the following recommendations: Recommendation: DHS should rapidly change its lingua franca from "Risk = T × V × C" to "Risk = f(T,V,C)
|
From page 100... ...
DEVELOP A STRONG SOCIAL SCIENCE CAPABILITY AND INCOPORATE THE RESULTS FULLY IN RISK ANALYSES AND RISK MANAGEMENT PRACTICES A particular concern of the committee's is an apparent lack of expertise in social sciences, certainly in RMA, but apparently also in other DHS units responsible for risk analysis. Social science expertise is critical to understand terrorism risk and to properly model societal responses to any type of hazardous scenario, and this absence poses a major gap in DHS expertise.
|
From page 101... ...
, DHS's consequence modeling is in general too limited in what it considers. That is not always wrong for a particular stakeholder's needs, but it is misleading if the modeling should illuminate the full extent of homeland security risk.
|
From page 102... ...
Yet, despite that, the almost exclusive concentration among DHS risk analysts is on damage to critical infrastructure and the need to "harden" facilities, leaving this important domain of consequences unassessed. Accordingly, the partial approach to risk analysis employed at DHS carries the risk that DHS is working on the wrong problems, because terrorists might be aiming for an entirely different set of consequences than those that are driving DHS priorities.
|
From page 103... ...
An increased reliance on such capabilities can upgrade DHS efforts to use quantitative modeling for anticipating a broader range of consequences of catastrophic events than in the past, particularly those consequences that lead to large-scale social and economic disruptions. To improve preparations for managing a broad range of consequences, quantitative risk analyses should take into account the diverse ramifications to the extent possible.
|
From page 104... ...
Recommendation: DHS should have a well-funded research program to address social and economic impacts of natural disasters and terrorist attacks and should take steps to ensure that results from the research program are incorporated into DHS's risk analyses. Recommendation: In characterizing risk, DHS should consider a full range of public health, safety, social, psychological, economic, political, and strategic outcomes.
|
From page 105... ...
Effective risk communication is quite difficult. It should be done by staff who understand the issues and details of the risk analyses.
|
From page 106... ...
Of particular importance is the need for DHS to specify with complete clarity the specific uses to which risk analysis results will be put. This echoes the first recommendation of the NRC's 2007 Interim Report on Methodological Improvements to the Department of Homeland Security's Biological Agent Risk Analysis, which called for DHS to "establish a clear statement of the long-term purposes of its bioterrorism risk analysis" (NRC, 2007c)
|
From page 107... ...
The goal is to implement technology transfer, from universities to the homeland security workforce, while keeping those universities grounded in the real needs of DHS. Improving risk modeling at DHS will require commensurate building up of academic ties, training of DHS people, and tech transfer routes to the DHS user community.
|
From page 108... ...
For example, Secretary Napolitano included the following in her terms of reference for the 2009 Quadrennial Homeland Security Review: Development and implementation of a process and methodology to assess national risk is a fundamental and critical element of an overall risk management process, with the ultimate goal of improving the ability of decision makers to make rational judgments about tradeoffs between courses of action to manage homeland security risk. Other federal organizations that have mature risk cultures and processes, such as the EPA and Nuclear Regulatory Commission, took years or decades to mature.
|
From page 109... ...
Environmental Protection Agency in developing capabilities, over a 20-year period, for using risk analysis to inform strategic decision making and decisions about risk reduction. Such guidelines, when well developed, provide both the scientific basis for methods and guidance on the sources and types of data needed to complete each type of analysis.
|
From page 110... ...
Without that discipline, it is very difficult to know precisely how DHS risk analyses are being done and whether their results are trustworthy and of utility in guiding decisions. As illustrated in the sections on uncertainty and avoiding false precision in Chapter 4 and in the discussion about how T, V, and C are combined to measure risk in different circumstances, it is not easy to determine exactly what DHS is doing in some risk analyses because of inadequate documentation, and the details can be critical for determining the quality of the method.
|
From page 111... ...
Good scientific practice for model-based scientific work includes the following: Clear definition of model purpose and decisions to be supported; Comparison of the model with known theory and/or simple test cases or extreme situations; Documentation and peer review of the mathematical model, generally through a published paper that describes in some detail the structure and mathematical validity of the model's calculations; and Some verification and validation steps, to ensure that the software implementation is an accurate representation of the model and that the resulting software is a reliable representation of reality. In the absence of these steps, one cannot assess the quality and reliability of the risk analyses.
|
From page 112... ...
Recommendation: DHS should adopt recognized scientific practices for its risk analyses: DHS should create detailed documentation for all of its risk models, including rigorous mathematical formulations, and subject them to technical and scholarly peer review by experts external to DHS. Documentation should include simple worked-out numerical examples to show how a methodology is applied and how calculations are performed.
|
From page 113... ...
In one sense, DHS riskrelated processes can be helpful in this regard: in most cases, those who are close to the risk management function will have to be involved in producing vulnerability analyses. This is certainly the case for CIKR sectors, and it is also true for users of TRAM, MSRAM, and probably other risk packages.
|
From page 114... ...
. Recommendation: To maximize transparency of DHS risk analyses for decision makers, DHS should aim to document its risk analyses as clearly as possible and distribute them with as few constraints as possible.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.