Review of the Department of Homeland Security's Approach to Risk Analysis (2010) / Chapter Skim
Currently Skimming:
4 Evaluation of DHS Risk Analysis
4 Evaluation of DHS Risk Analysis
Pages 52-87
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 52... ...
For example, the interim Integrated Risk Management Framework (IRMF) , including the risk lexicon and analytical guidelines (primers)
|
From page 53... ...
DETAILED EVALUATION OF THE SIX ILLUSTRATIVE RISK MODELS EXAMINED IN THIS STUDY Natural Hazards Analysis There is a solid foundation of data, models, and scholarship to underpin the Federal Emergency Management Agency's (FEMA's) risk analyses for earthquakes, flooding, and hurricanes which uses the Risk = T × V × C model.
|
From page 54... ...
For other problems, such as riverine flooding in the absence of structural protection, the vulnerability assessment requires little more than ascertaining whether flood waters rise to the level of a building. Assessing consequences of extreme events of natural hazards has typically focused on loss of lives, injuries, and resulting economic losses.
|
From page 55... ...
To achieve this last goal, studies are frequently conducted retrospectively to compare predictions with actual observed outcomes. A second indicator that risk analyses for natural hazards are fairly reliable is that the limitations of the constituent models are well known and adequately documented.
|
From page 56... ...
uncertainty is identifiable and quantifiable. This is the case for most natural hazards risk assessments.
|
From page 57... ...
They are present in deterministic design studies as well as in risk assessments, although in the former they are masked by factors of safety and other traditional means of risk reduction. Conclusion: DHS's risk analysis models for natural hazards are near the state of the art.
|
From page 58... ...
Analyses of Critical Infrastructure and Key Resources (CIKR) DHS has processes in place for eliciting threat and vulnerability information, for developing consequence assessments, and for integrating threat, vulnerability, and consequence information into risk information briefings to support decision making.
|
From page 59... ...
DHS recognizes that the use of generic attack scenarios (threats) based on today's knowledge can leave risk analyses vulnerable to the unanticipated "neverbefore-seen" attack scenario (the black swan)
|
From page 60... ...
The committee cannot assess the impact of all these efforts to increase the information and dialogue between national and local levels with respect to DHS risk analysis, and specifically with regard to threat assessments and probabilities. The majority of information gathered by the fusion centers is on criminal activities, not foreign terrorism.
|
From page 61... ...
However, there also need to be specific, consistent, repeatable exchanges, and other actions focused just on risk modeling. These interactions with experts from the broader intelligence community -- including those responsible for collection operations -- should be focused on building a common terminology and understanding of the goals, limitations, and data needs specific to DHS risk assessments.
|
From page 62... ...
Examples of such clusters are New York City bridges, facilities surrounding Exit 14 on the New Jersey Turnpike, the Chicago Financial District, RaleighDurham Research Triangle, and the Tennessee Valley Authority. However, the complexity of the RRAP methodology seems incommensu5 Critical Infrastructure Protection: Sector Plans and Sector Councils Continue to Evolve, GAO-07-706R (July 10, 2007)
|
From page 63... ...
The weighted PMIs were shown to four significant figures in a presentation to the committee, and different values are estimated for physical security, security forces, and security management. Those three weighted PMIs are averaged to obtain an overall index for a piece of critical infrastructure.
|
From page 64... ...
. Recommendation: DHS should ensure that vulnerability and consequence analyses for infrastructure protection are documented, transparent, and repeatable.
|
From page 65... ...
Recommendation: The committee recommends focusing specific research at one of the DHS University Centers of Excellence to develop risk models and metrics for terrorism threats against national icons. The range of consequences considered is also affected by the mandate of the entity performing the risk analysis.
|
From page 66... ...
Given each threat scenario and the associated rate of attack and mortality estimates (vulnerability estimates) , the NISAC modelers calculated reduction in labor in critical infrastructure sectors (i.e., absenteeism rate and duration [population consequences]
|
From page 67... ...
Such activities include the HITRAC Level 1/Level 2 program for differentiating high-risk targets, 18 Individual Sector Risk Profiles, development of a National Critical Foreign Dependencies list, deployment of PSAs to support Site Assistance Visits (SAVs) , and the Enhanced Critical Infrastructure Protection Initiative (ECIP)
|
From page 68... ...
plan around capabilities needed to respond to and recover from a variety of possible disruption events, whether terrorist attacks, natural hazards, or industrial accidents. DHS recognizes the need for analysis and planning for both short-term protective measures and longerterm risk-based investments in prevention, protection, response, and resiliency.
|
From page 69... ...
Understand event impacts of terrorism attacks, industrial accidents, and natural hazards using a single common model (integrated all-hazards approach where threat is any disruption event that impacts system ability to function as intended)
|
From page 70... ...
,13 and managing enterprise production capacity.14 This type of approach has also been used successfully in conjunction with more traditional site-based facilities management and security or fire protection operations15 and is recommended by the American Society of Civil Engineers' Critical Infrastructure Guidance Task Committee in its recently released Guiding Principles for the Nation's Critical Infrastructure (ASCE, 2009)
|
From page 71... ...
Five of these programs, covering more than half of FEMA's grant money -- the State Homeland Security Program, the Urban Areas Security Initiative, the Port Security Grant Program, the Transit Security Grant Program, and the Interoperable Emergency Communications Grant Program -- incorporate some form of risk analysis in support of planning and decision making. Two others inherit some risk-based inputs produced by other DHS entities -- the Buffer Zone Protection Program, which allocates grants to jurisdictions near critical infrastructure if they are exposed to risk above a certain level as ascertained by the Office of Infrastructure Protection; and the Operation Stonegarden Grant Program, which provides funding to localities near sections of the U.S.
|
From page 72... ...
FEMA staff told a committee delegation on a site visit that this coarse approximation is relatively acceptable to the entities supported by the grants programs. It appears that the choice of weightings in these risk assessments, and the parameters in the consequence formulas, are chosen in an ad hoc fashion and have not been peer reviewed by technical experts external to DHS.
|
From page 73... ...
Initiative as explained in Chapter 2. C2C is a "model" to "develop, test, and implement a method for strategically managing a portfolio of grant programs at the local, state, and federal levels." It is intended to "create a culture of measurement, connect grant dollars to homeland security priorities, and demonstrate contributions of preparedness grants to the national preparedness mission."16 The C2C model replaces "vulnerability" with "capability," in a sense replacing a measure of gaps with a measure of the ability of a system or community to withstand an attack or disaster or to respond to it.
|
From page 74... ...
Overall, TRAM produces a threat rating for various attack scenarios that is calculated as the product of an SME rating of the attack likelihood and an SME rating of the scenario likelihood. The presentation of notional results at the committee's first meeting showed scenario likelihoods for a range of assets to three significant figures, which is unrealistic, and overall threat ratings for each of those assets (the product of attack likelihood and scenario likelihoods)
|
From page 75... ...
This complexity also makes it very difficult to evaluate the model. On a site visit to the Port Authority of New York and New Jersey, a committee delegation asked the TRAM contractor team whether the tool had been validated.
|
From page 76... ...
Contractor experts said they can estimate the uncertainty in these risk analyses, although the output becomes complicated, and that information is normally interpreted by contractor staff and not presented to the end users. The Port Authority of New York and New Jersey (PANYNJ)
|
From page 77... ...
, or the 2008 integrated Chemical, Biological, Radiological, and Nuclear (CBRN) assessment that builds on BTRA and CTRA.18 DHHS is responsible for the nation's preparedness to withstand and respond to a bioterror attack, and in order to learn more about how DHS coordinates with another federal agency in managing a homeland security risk, a delegation of committee members made a site visit to DHHS's Biomedical Advanced Research and Development Authority (BARDA)
|
From page 78... ...
BARDA manages Project BioShield, which includes the procurement and advanced development of medical countermeasures for chemical, biological, radiological, and nuclear agents, as well as the advanced development and procurement of medical countermeasures for pandemic influenza and other emerging infectious diseases that fall outside the auspices of Project BioShield." BARDA provides subject matter input to DHS risk analyses and relies on DHS for threat analyses and risk assessments. One fundamental reason that BARDA declined to rely on these DHS products is that it received only a document, without software or the primary data inputs, and thus could not conduct its own "what-if" analyses that could guide risk mitigation decision making.
|
From page 79... ...
Integrated Risk Management Framework The committee can develop only a preliminary impression of DHS's adoption of the Integrated Risk Management Framework because, as a developing process rather than a model, it is not yet in its final state. That is normal: instantiations of Enterprise Risk Management (ERM)
|
From page 80... ...
However, with the exception of risk analysis for natural disaster preparedness, the committee did not find any DHS risk analysis capabilities and methods that are yet adequate for supporting DHS decision making, because their validity and reliability are untested. Moreover, it is not yet clear that DHS is on a trajectory for development of methods and capability that is sufficient to ensure reliable risk analyses other than for natural disasters.
|
From page 81... ...
Some DHS risk work reflects an understanding of uncertainties -- for example, the uncertainty in the FEMA floodplain maps is well characterized, and the committee was told that TRAM can produce output with an indication of uncertainty (though this is usually suppressed in accordance with the perceived wishes of the decision makers and was not shown to the committee)
|
From page 82... ...
weapons. The next generation of TRAM is being developed to include the ability to represent a range of hazards -- human-initiated, technological, and natural -- and measure and compare risks using a common scale.23 More generally, RMA created the Integrated Risk Management Framework in order to support disparate types of risk assessment and management within DHS and eventually across the homeland security enterprise.24 The DHS Risk Steering Committee's vision for integrated risk management is as follows: "…to enable individual elements, groups of elements, or the entire homeland security enterprise to simultaneously and effectively assess, analyze, and manage risk from multiple perspectives across the homeland security mission space" (DHS-RSC, 2009)
|
From page 83... ...
Integrated risk analysis collects analyses for all potential risks facing an entity, here DHS, and combines those risks into one complete analysis using a common metric. This is contrasted with comparative risk analysis which omits the last step.
|
From page 84... ...
of the Statement of Task, the committee makes the following recommendation: Recommendation: The risks presented by terrorist attack and natural disasters cannot be combined in one meaningful indicator of risk, so an allhazards risk assessment is not practical. DHS should not attempt an integrated risk assessment across its entire portfolio of activities at this time because of the heterogeneity and complexity of the risks within its mission.
|
From page 85... ...
In these cases, the same risk management option might have the ability to reduce risks from a number of sources such as natural hazards and terrorism. The analysis of the alternative risk management options that could mitigate risks to a set of activities or assets could be analyzed in a single quantitative model in much the same way that cost-effectiveness analysis can be used to select the least-cost investment in situation in which benefits are generally incommensurate.
|
From page 86... ...
One of the key assumptions in integrated or enterprise risk management (particularly for financial services firms) is that there is a single aggregate risk measure such as economic capital (Bank for International Settlements, 2006)
|
From page 87... ...
It is worth noting that most nonfinancial services firms implementing ERM adopt the philosophical concepts but have several metrics for comparative risk across operations. For example, nonfinancial services firms evaluate risks using comparative metrics such as time to recover operations, service-level impact over time, potential economic loss of product or service, number of additional temporary staffing (reallocated resources)
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.