United States federal agencies must satisfy specific requirements for analyzing proposed major or economically significant regulatory actions, such as those expected to have an annual effect on the economy of $100 million or more.231,236 As required under Executive Order 12866 and other executive branch authorities,43,284 OMB sets out a hierarchy of methods:231
- benefit-cost analysis (BCA), in the ideal, monetizes all the benefits and costs of a proposed regulatory action to arrive at a net balance;
- cost-effectiveness analysis monetizes the cost of meeting a policy objective, most typically stated as a numerical target, and can be used when costs are known and important categories of benefit can be quantified, but not monetized; and
- breakeven or threshold analysis asks how much value society must place on the benefits or costs of an action for the benefits to outweigh the costs, and can be used when benefits or costs cannot be monetized or quantified.
As noted below, DHS has employed breakeven analysis in much of its recent analyses of major regulatory actions to address terrorism threats.
BCA, cost-effectiveness analysis, and breakeven analysis have gained acceptance in state and local policy making and among nongovernmental entities, but all three methods have limitations. They do not, for example, address concerns about equity, the distribution of benefits and costs, and the possibility that some members of society might gain while others would not. OMB states, “Regulatory analysis should provide a separate description of distributional effects (i.e., how both benefits and costs are distributed among sub-populations of particular
concern) so that decision makers can properly consider them along with the effects on economic efficiency.” 231 Moreover, for regulations intended to confer benefits under circumstances of extreme uncertainty, the analytical requirements pose special challenges not limited to data deficiencies.
GUIDANCE ON REGULATORY ASSESSMENT
OMB specifies that a good regulatory analysis, regardless of method, should include “(1) a statement of the need for the proposed action, (2) an examination of alternative approaches, and (3) an evaluation of the benefits and costs—quantitative and qualitative—of the proposed action and the main alternatives identified by the analysis.”231 Before recommending federal regulatory action, an agency must demonstrate that the proposed action is necessary. Questions to consider include the following:
- Is the action intended to address a significant market failure—a situation in which the market for goods or services, alone, cannot provide a satisfactory outcome—or to meet some other compelling public need?
- Could other types of actions—federal or not—be as, or more, effective than the action?
- Could the compelling public need, identified above, be met through state or local regulation, voluntary programs (e.g., programs sponsored by trade associations), public-private partnerships, insurance, or tort liability?
OMB further specifies that a prospective evaluation of benefits and costs should
- explain how the proposed requirements are linked to the expected benefits;
- identify a baseline, typically no action, against which to measure the effects of the proposed regulatory action, and draw comparisons to alternatives; and
- identify and capture the expected undesirable side effects and ancillary benefits of the proposed regulatory action and the alternatives.
OMB also provides the policy community with context, warnings, and caveats. It explains that “the motivation for the analysis is to (1) learn whether the benefits of an action are likely to justify the costs or (2) discover which of various possible alternatives would be the most cost-effective,” 231 but it notes that efficiency might not be the only or overriding public policy objective. OMB also warns that good regulatory analysis is not strictly formulaic; rather, it requires competent professional judgment, transparency, and well-documented supporting evidence, including a discussion of the sensitivity of the results to embedded assumptions. OMB asks agencies to quantify anticipated benefits and costs to the extent possible, but acknowledges that quantification, particularly monetization,
may not be feasible across the board. When it is not possible to monetize benefits or costs, an agency may calculate them in terms of physical units; when calculations of physical units are also impossible, an agency may, instead, proceed qualitatively. In particular, OMB231 recognizes that,
It will not always be possible to express in monetary units all of the important benefits and costs. . . . If the non-quantified benefits and costs are likely to be important, you should carry out a “threshold” analysis to evaluate their significance. Threshold or “break-even” analysis answers the question, “How small could the value of the non-quantified benefits be (or how large would the value of the non-quantified costs need to be) before the rule would yield zero net benefits?”
None of the methods specified by OMB—BCA, cost-effectiveness analysis, or breakeven analysis—inherently addresses OMB’s concerns for either the links between regulatory actions and anticipated benefits or the possibility of unintended consequences and ancillary benefits, nor do any deal directly with matters of equity. The section that follows provides an overview of some of the challenges of assessing control strategies, particularly their benefits.
Assessing the benefits of actions to prevent or reduce losses from terrorist attacks (e.g., estimates of potential reductions in deaths and injuries, property damage, or various social effects) under extremely uncertain conditions poses substantial challenges.236,237,285,286 Such an analysis would require an explicit characterization of the underlying risk of a terrorist attack and an understanding of the ways in which the regulations might affect that risk, which would, in turn, present analytical obstacles. In addition, a monetary estimate of benefits might involve evaluating society’s willingness to pay to avoid terrorism-related damages.287,288 Farrow cites specific challenges of interdependencies among defenses or targets; choices among damage- and probability-reducing strategies; the avoidance of unintended consequences in complex systems; ambiguity about the likelihood of events and consequences; the existence of intelligent and adaptive adversaries; and the path-dependent and irreversible nature of some defenses and attack consequences.289
Two challenges stand out in this context: (1) a lack of data both for typifying scenarios, even those with a historical basis, and for projecting the nature and likelihood of future incidents; and (2) an inability to anticipate how terrorists will adapt to changes in the security environment.236,245,290,291 Relevant, reliable, and accurate data are scarce—or unavailable—because terrorist attacks in the United States and comparable settings are infrequent and highly variable and because sensitive information about attacks requires special handling or protection. Moreover, a change in policy that affects the security environment might lead to dis-
placement; in particular, a terrorist could choose a different method of attack or redirect his or her efforts to other settings.292,293 If, for example, a terrorist finds that is has become too difficult to obtain precursor chemicals to construct IEDs in one location, he or she might choose to work with a vehicle, gun, or knife or to operate elsewhere. The scarcity of data compounds the problem of characterizing potential adaptations and outcomes.
Some recent regulatory assessments have used break-even analysis to circumvent the challenges of benefits estimation. For example, for the proposed Ammonium Nitrate Security Program, DHS compared the anticipated costs of the program to the costs—to people and property—of the bombing of the Murrah Building in Oklahoma City and then calculated the frequency with which similar events must happen to offset the cost of the security program. Members of the policy community have recognized DHS’s adoption of break-even analysis as a step forward in the evolution of analysis of terrorism security policy, but have called for further advancement.237 In a DHS-sponsored workshop held at the RAND Corporation in 2011, the agency itself sought to identify novel approaches to valuing benefits and, potentially, to move beyond break-even analysis.236 For example, DHS staff presented a fault tree approach that can be used to map existing security programs and policy mechanisms and to estimate event failure probabilities, such as the failure of existing efforts to stop the entry of terrorists into the United States. In developing these models, expert judgment is essential. Quantitative judgments—including quantitative statements of a model’s uncertainty—can be combined in these risk models to develop distributions (or ranges) of both baseline risks and changes in risks with the adoption of additional programs and policy mechanisms.
Cost assessment tends to be less fraught because it presents greater certainty. Generally speaking, regulatory agencies propose specific measures that entail known—or knowable—administrative requirements and operational burdens that can be estimated with established accounting and engineering methods. Nevertheless, some elements of cost assessment, such as those that depend on evaluations of businesses’ or consumers’ behavior or involve unintended consequences, can present uncertainties and pose analytical challenges.