The U.S. Environmental Protection Agency (EPA) has estimated that thousands of premature deaths and numerous cases of illness, such as chronic bronchitis and asthma attacks, could be prevented by reducing exposure to air pollution. These estimates come from regulatory health benefits analyses, which attempt to quantify changes in the expected cases of mortality and illness that are likely to result from proposed air pollution regulations. The estimates are often controversial, and the methods used to prepare them have been questioned.
In 2000, Congress recognized concerns about the methods used by EPA and emphasized the need for “the most scientifically defensible methodology in estimating health benefits.” It directed EPA to ask the National Academy of Sciences “to conduct a study of this issue and recommend to the agency a common methodology to be followed in all future analyses.”1
THE CHARGE TO THE COMMITTEE
In response to EPA’s request, the National Research Council (NRC) convened the Committee on Estimating the Health-Risk-Reduction Benefits of Proposed Air Pollution Regulations, which prepared this report. Mem-
bers were chosen for their expertise in risk assessment, exposure assessment, toxicology, epidemiology, biostatistics, health economics, and air pollution regulations. The committee was asked to accomplish the following tasks:
Consider issues important in estimating the health-risk-reduction benefits of air pollution regulations, including the scientific data, risk-assessment approaches, populations affected, baselines used, assumptions, analysis of uncertainty, and identification of key indicators of exposure and population health status.
Critically review methods used for recent estimates of regulatory health benefits.
Identify methods used by federal regulatory agencies and others, recommend standard good-practice guidelines and principles for estimating health benefits, and delineate the data-gathering required to better assess health benefits in the future.
Identify approaches to estimating regulatory health benefits when relevant information is limited.
Where applicable, recommend areas for further research and monitoring.
The committee was not asked to evaluate methods used to estimate other types of benefits, such as improvements in visibility, resulting from air pollution control. The committee also was not asked to review the methods used for economic valuation of health benefits or for regulatory cost analyses.
THE COMMITTEE’S APPROACH
To accomplish its charge, the committee heard, in public session, presentations from representatives of EPA, the U.S. Senate, the Office of Management and Budget (OMB), and other interested parties; reviewed materials submitted by EPA and others; and reviewed current literature relevant to health benefits estimation. The committee selected for detailed review the health benefits analyses contained in the regulatory impact assessments (RIAs) prepared by EPA for the following rule-makings: (1)“Particulate Matter and Ozone National Ambient Air Quality Standards”
(1997), (2)“Tier 2 Motor Vehicle Emissions Standards and Gasoline Sulfur Control Requirements” (1999), and (3) “Heavy Duty Engine and Vehicle Standards and Highway Diesel Fuel Sulfur Control Requirements” (2000). The committee also reviewed the health benefits analysis completed for EPA’s analysis of the benefits and costs of the 1990 Clean Air Act Amendments (CAAA) (1999). All of these analyses are described in Chapter 2 of this report (see Tables 2-1 and 2-5).
Ozone and airborne particulate matter (PM) were the primary focus of the EPA analyses selected by the committee for review. Therefore, the committee spent a considerable amount of time discussing these pollutants, especially PM, and did not address issues associated with the analysis of the hazardous air pollutants (HAPs). However, many of the findings and recommendations of the committee have broad applicability and are not limited to analyses conducted for PM.
THE COMMITTEE’S EVALUATION AND FINDINGS
Despite many inherent uncertainties, the committee concludes that regulatory benefits analysis can be a useful tool for generating information valuable to policy-makers and the public. Properly conducted analyses can help identify the type, magnitude, and relative importance of health benefits, highlight the sensitivity of the benefits estimates to assumptions made in the analysis, and indicate the areas of greatest scientific uncertainty. Information from the analyses can help focus future research efforts to reduce key uncertainties. The committee emphasizes, however, that estimates of health benefits and their economic valuation are only one part of the deliberative and political processes necessary for the development of sound policy.
Estimating the health benefits of a potential reduction in ambient air pollution involves a series of steps. First, the regulatory options to be evaluated must be clearly defined with regard to scope, timing, and implementation. Then, the boundaries of the analysis, such as the time period for which benefits are evaluated, must be established. In addition, the regulatory baseline (the description of conditions without the proposed regulation) must be defined. Once the analysis has been structured, future changes in pollutant emissions and resulting changes in ambient pollutant concentrations and population exposures can be predicted. Changes in health outcomes can then be estimated by applying concentration- or exposure-response
functions (derived from the health literature) to estimated changes in population exposures.
The committee finds that these basic steps provide a reasonable framework for conducting health benefits analysis and that EPA has generally used this basic approach when estimating the expected health benefits of proposed air pollution control regulations. However, on the basis of the analyses reviewed by the committee, EPA’s implementation of these steps could be improved. Recommendations for improvements in the process are described in the following pages.
The committee notes that analysis of health benefits for any regulation will require flexible, innovative, and multidisciplinary participation and guidance of scientific experts. Therefore, the committee did not attempt to write a detailed manual for conducting benefits analysis but instead addressed the key methodological issues and their importance in the EPA benefits analyses reviewed by the committee.
Regulatory Options, Boundaries, and Baselines
The health benefits that are estimated to result from reducing air pollution depend on the decisions made at the beginning of the analysis regarding the regulatory options to consider, the health outcomes to evaluate, the time frame over which benefits are estimated, and the assumptions made about conditions with and without implementation of the regulation. In three of the four EPA analyses reviewed by the committee, EPA focused on evaluating a single regulatory option. This approach conflicts with current OMB guidance on benefits analysis, which suggests consideration of a range of regulatory options and a variety of technical and economic interventions.
The committee acknowledges that EPA cannot evaluate every possible regulatory option, given time and resource constraints; however, a realistic range of options guided by expert opinion and technical feasibility should be represented in EPA’s benefits analyses. At the beginning of each analysis, EPA should describe this range of options and any preliminary analyses that were conducted to exclude certain options from the formal benefits analysis. This approach would strengthen analyses that might otherwise appear to serve the purpose of justifying EPA’s chosen regulatory option.
Once the regulatory options are selected, EPA must determine how
broadly to define the scope of the analysis, including the degree to which secondary or unintended effects of the regulation should be examined. For example, air pollution regulations can change not only ambient air pollution levels but also how fuels are made or how combustion devices are operated. These changes might affect human health through other pathways, such as through water pollution or occupational exposures. An analysis of health benefits that ignores those effects might result in a substantial misrepresentation of the potential impacts of pollution-control measures on society. Although the committee recognizes that assessment of secondary effects may be difficult, the benefits analysis should discuss whether such impacts appear to be important and, if so, should incorporate a plan for assessing them.
Although EPA usually evaluates the costs of regulatory options for the time period between introduction and full implementation of the regulation, the benefits of the regulation have often been examined for only a single year—typically the year in which the regulation will have been fully implemented. Evaluation of benefits for only a single year has two limitations. First, when the costs of the regulatory action decrease over time and the benefits increase, the comparison of benefits and costs in the distant future could be misleading. Second, choosing an evaluation point in the distant future, such as 2030, is likely to increase the uncertainty associated with estimating both benefits and costs. These limitations can make the analysis misleading. Therefore, benefits should be estimated at reasonable intervals, such as every 5 years, over the regulatory time frame, including both the period of implementation and the expected period of expression of all significant health effects.
To estimate the benefits of a proposed air pollution regulation, EPA makes predictions about conditions expected to occur both with the regulation (control scenario) and without the regulation (baseline scenario). Predictions concerning air emissions and the U.S. population are especially relevant to calculating the health benefits. Two issues regarding emissions predictions particularly concern the committee. First, many important components of an emissions analysis, such as number of vehicles in a class, average miles traveled per vehicle, and emissions per mile, are seldom summarized for the benefits analysis. This lack of information makes it difficult to judge the plausibility of the emissions estimates. Second, current emissions models fail to provide an assessment of uncertainty associated with the emissions predictions for the baseline and control scenarios, which
can be substantial. Comparison of emissions predictions to historical trends could help elucidate discrepancies that should be explained or formally incorporated into an uncertainty analysis and taken into account when estimating health benefits.
Predictions about future populations, such as numbers, age distributions, and baseline health status, are important aspects of EPA’s benefits analyses. However, it is difficult to make confident predictions about the characteristics of populations 30 years in the future. EPA should evaluate the uncertainty involved in these predictions and the impacts of these uncertainties on the benefits estimates. Some sense of the uncertainty in these predictions may be obtained by comparing the characteristics, such as age, sex, ethnic mix, disease, and mortality, of the projected future population with those of the populations studied in the epidemiological studies on which the benefits estimates are based.
A critical step in estimating the benefits of proposed air pollution regulations is determining the effect of emissions changes on ambient air quality. This has traditionally been accomplished using air-quality models of varying complexity. EPA’s approaches to exposure assessment evolved considerably over the period of the analyses reviewed by the committee as a result of continued improvement in the models and marked increase in available monitoring data for key pollutants. Overall, the methods used in the most recent EPA analysis reviewed by the committee (heavy-duty engine and diesel-fuel analysis) represent an appropriate and reasonably thorough application of the available data and models for exposure assessment.
Several issues, however, deserve to be mentioned regarding the models and the assumptions used in the exposure assessments. First, models are simplifications of reality. Estimating how well a model simulates pollutant concentrations in the ambient air resulting from emissions changes estimated at some future time is difficult and requires a systematic process of model testing and evaluation. Without such a process, it is difficult to know how much confidence to place in the predictions. The methods used to test the models also need to be clearly described in the benefits analysis. Second, many of the models used by EPA are time and resource intensive, thus limiting the modeling that can be conducted. The limitation is problem-
atic because it restricts the number of regulatory options that can be considered and the number of years for which benefits can be estimated.
A tacit or explicit assumption in exposure assessment is that pollutant concentrations in ambient air adequately represent human population exposures. Although ambient concentrations in many cases appear to be reasonable indicators of human exposure, EPA should more rigorously assess the relative contributions of different emissions sources to human exposures. For example, EPA should evaluate whether PM emissions from diesel-fuel vehicles have a greater impact on human exposure than those from stationary sources, because diesel exhaust is emitted closer to people.
Another assumption specific to the analyses reviewed by the committee concerns PM. PM is a heterogeneous mixture that varies in size, composition, and source of origin; therefore, the health effects of PM exposures in one area might be different from those in another area and might vary over time. For example, the health effects of agricultural PM, which are derived primarily from crustal, animal, and plant sources, may differ from the health effects of urban PM, which are derived primarily from combustion sources, such as power plants and automobile and truck traffic. Because scientific information on PM toxicity is incomplete, EPA has typically made the assumption of equivalent potency across particle types. The committee believes that benefits analyses would be strengthened by evaluating a range of alternative assumptions regarding relative particle toxicity in sensitivity or uncertainty analyses.
The appropriate selection and definition of adverse health outcomes is integral to any assessment of health benefits. A wide range of health effects, primarily related to the respiratory and cardiovascular systems, is linked to exposure to air pollutants. In the analyses reviewed by the committee, EPA appears to have carefully considered the majority of these effects. However, many health outcomes are not quantified because there are insufficient data or because inclusion of certain health effects in the primary analysis could lead to double-counting.
The committee identified several issues regarding the selection and definition of mortality and morbidity (disease and other adverse health effects) outcomes. Clinically diagnosed illnesses, such as chronic bronchitis
and asthma attacks, are typically evaluated in benefits analyses. A problem with these diagnoses is that they cover a wide range of severity levels and time courses. For example, chronic bronchitis can range from a chronic cough to a severe chronic airway obstruction that requires long-term care. The lack of clear categorization of outcome severity in benefits analyses has implications for quantification and valuation of the outcomes. Although EPA has made some attempt to deal with this issue, it needs to investigate and improve the methods used to reconcile differences between the severity of disease described in air pollution epidemiology and that commonly used to develop estimates of background disease prevalence and incidence.
In each benefits analysis reviewed by the committee, EPA used U.S. studies to provide data to estimate the health benefits. Data for many health outcomes in the U.S. studies are restricted to a specific age group. For example, the data for hospital admissions apply to persons 65 years or older, primarily because the data come from Medicare databases. For the benefits analyses, EPA did not extrapolate those data beyond the age ranges provided in the studies. The committee notes that recent studies conducted outside the United States provide information on certain health outcomes with broader age ranges and on outcomes not currently evaluated by EPA, such as levels of use of the primary-care system. EPA should use such studies when appropriate to extrapolate beyond the age ranges currently considered and to incorporate health outcomes not currently evaluated in the analyses.
Mortality is a well-defined health outcome that was evaluated in each EPA analysis reviewed by the committee. Mortality estimates tend to dominate the overall health benefits estimates when a dollar value is assigned to them. However, the committee notes that data on morbidity is less comprehensive and needs to be improved, especially if the value assigned to mortality decreases and morbidity outcomes begin to play a more dominant role in the benefits analysis.
Another important issue relates to the key assumption that there is a causal association between particular types of air pollution and adverse health outcomes. The EPA benefits analyses reviewed by the committee provided little information concerning this assumption. Although a comprehensive discussion of causality is not necessary for a benefits analysis, the evidence of causality should be summarized to justify the inclusion or exclusion of health outcomes and to assess the uncertainty associated with the assumption of causality. EPA should investigate and, if necessary,
develop methods of evaluating causal uncertainty relating to key outcomes so that this uncertainty can be represented in the final benefits estimates.
A primary element of health benefits analysis is the selection of the concentration-response functions, which describe the quantitative association between ambient air pollution levels and the corresponding health effects. Concentration-response functions can be derived from animal studies, human clinical studies, or epidemiological studies. In the analyses reviewed by the committee, EPA relied on epidemiological studies as the basis for estimating concentration-response functions. Because epidemiological studies involve the study of humans in real-world situations and, therefore, are more relevant to the assessment of health benefits than animal toxicity or human clinical studies, the committee supports the use of these studies to estimate concentration-response functions. However, the benefits analyses should reflect the plausibility and uncertainty of the concentration-response function, such as imprecision of exposure and response measures, potential confounding factors, and extrapolation from the study population to the target population in the benefits analysis.
For the analysis of mortality, EPA used cohort studies (epidemiological studies that evaluate health effects in a specific population over a period of years) to derive benefits estimates in each analysis reviewed by the committee. The committee agrees with that approach. Compared with time-series studies (epidemiological studies that provide estimates of health effects due to recent exposure), cohort studies give a more complete assessment because they include long-term, cumulative effects of air pollution. Furthermore, the particular advantage of cohort studies is that they provide data to estimate the number of life-years lost in a population, not just the number of lives lost, thus allowing for several valuation methods to be used.
Overall, the committee found that the epidemiological studies selected by EPA for use in its benefits analyses were generally defensible. However, the criteria and process by which EPA reached its decisions were not articulated in many cases, and at times, the study selection process appeared to be inconsistent. For example, estimates were derived from multiple studies in some cases and from single studies in other cases when
multiple studies were available. This selection process requires judgment on the part of the analyst, and EPA needs to document clearly the rationale for its selection of studies and concentration-response functions.
The committee concluded that EPA’s selection of the American Cancer Society (ACS) study2 for the evaluation of PM-related premature mortality was reasonable, given the size and precision of the study. However, those facts are not necessarily grounds for adoption of this study over others. For example, the Harvard six cities study3 has some advantages over the ACS study, such as the use of a random population sample and the careful placement of monitors for the study. Because several new studies have since been published, including an extended analysis of the original ACS study, a new U.S. cohort study, and other non-U.S. studies, EPA should review its selection of the most appropriate studies. Furthermore, EPA might want to consider derivation of a weighted-mean estimate from the cohort studies following review of the entire database.
Decision-makers may want to know the effects of a regulation on different subgroups of a population, such as groups with varying health or socioeconomic status. Health effects might vary because the regulation causes different reductions in exposures for different subgroups or because various subgroups may respond differently to a specific exposure reduction. Populations may respond differently because their baseline rates of illness differ or because their concentration-response functions differ. The committee encourages EPA to estimate and report benefits by age, sex, and other demographic factors, when possible. Any assumptions that might explain the differences among subgroups should be clearly stated.
Analysis of Uncertainty
EPA uses a two-part approach to assess uncertainty in its health benefits analyses. The first part of the approach is a primary analysis that
produces a probability distribution for each health outcome evaluated. For example, EPA provided a probability distribution for the number of avoided deaths in 2030 in the analysis conducted for the Tier 2 rule-making. Only one source of uncertainty (the random sampling error associated with the estimated concentration-response function) was incorporated into the analysis. EPA typically emphasizes only the mean value of the probability distribution. Because of the lack of consideration of other sources of uncertainty, the results of the primary analysis often appear more certain than they actually are.
The second part of the approach is ancillary uncertainty analyses, which include alternative and supplementary calculations for some uncertainties and sensitivity analyses for others. The ancillary analyses usually examine one source of uncertainty at a time and therefore do not adequately convey the aggregate uncertainty from other sources, nor do they discern the relative degrees of uncertainty in the various components of the health benefits analysis.
EPA should move the assessment of uncertainty from its ancillary analyses into its primary analyses to provide a more realistic depiction of the overall degree of uncertainty. This shift will entail the development of probabilistic, multiple-source uncertainty models based not only on available data but also on expert judgment. EPA should continue to use sensitivity analyses but should attempt to include more than one source of uncertainty at a time. EPA also should strengthen its efforts to identify the uncertainty sources that have the greatest influence on the final results. The committee emphasizes that cost estimates are also subject to great uncertainty, and the same standards should be applied to the assessment of the uncertainties in those estimates.
As more sources of uncertainty are incorporated into the primary analyses, the results inevitably will appear less certain, and the analyses might appear to be less useful to some. However, uncertainty should be described as completely and as realistically as possible for all regulatory options, recognizing that regulatory action might be necessary in the presence of substantial uncertainty. The regulatory decision process will be better informed by a fair assessment of the uncertainty and a realistic evaluation of the likely reductions in that uncertainty attainable through further research.
Accurately characterizing the uncertainties in estimates of health benefits for projected future human populations is difficult. Therefore, EPA should consider conducting preliminary analyses that estimate in current
populations the health benefits resulting from hypothetical changes in current levels of emissions. Such preliminary analyses would help EPA develop an idea of the lower bound on the range of uncertainty. These analyses also would have fewer uncertainties than analyses based on projected future population exposures and health outcomes.
Presentation of Results
A common complaint about EPA’s regulatory benefits analyses is that the methods, the rationale behind the decision-making, and the results are not clearly described or presented. After review of the EPA analyses, the committee agrees that the presentations should be improved. The committee is concerned that important factors that drive the results of an analysis are often buried in appendixes or technical-support documents, and the rationales behind key decisions are not clearly discussed. Furthermore, the amount of discussion devoted to some parameters often does not appear to be proportional to their importance to the analysis. For example, in the heavy-duty engine and diesel-fuel analysis, an interpolation method used in the exposure assessment is discussed at length, whereas the exclusion of modeling results for the western United States is acknowledged in only one sentence.
The committee concludes that many of the problems associated with EPA’s presentation of such analyses could be solved by inclusion of a detailed summary that presents the key information of the analysis in a straightforward manner. Such information includes the following:
Changes in ambient air quality.
Health outcomes evaluated.
Uncertainties associated with the estimates.
The summary should highlight all assumptions that have a substantial impact on the results of the analysis.
The results of health benefits analyses are typically used as inputs to cost-benefit or cost-effectiveness analyses. Therefore, EPA should provide benefits estimates in ways that provide useful input to these analyses. For example, benefits estimates should be presented when possible by age group to allow calculation of quality-adjusted life-years, a measure used in cost-effectiveness analysis.
The committee recognizes that some of the following recommendations will be easier for EPA to implement than others. However, with the exception of research needs, these recommendations should not require substantial new resources on the part of EPA, although EPA may need to change its approaches and allocation of resources to accomplish them. The committee acknowledges that some of the research needed is outside EPA’s jurisdiction and will require support from other agencies.
EPA should include in its regulatory benefits analyses comparative estimates of the benefits for several regulatory options that represent a realistic range of choices available to the decision-maker. If regulatory options are eliminated at an early stage, the rationale for the elimination should be provided.
EPA should examine whether unintended positive or negative impacts on human health or the environment might occur from implementation of the proposed regulation. For example, changes in fuels could result in water pollution, changes in occupational exposures, or reductions in greenhouse gas emissions. If important impacts are identified, a plan to assess them more completely should be included.
EPA should estimate potential benefits at reasonable intervals, such as every 5 years, over the regulatory time frame, including the period of regulatory implementation and the expected period of occurrence of all significant health effects.
EPA should present the information on which emissions estimates are based for scenarios with and without the regulation. This information will help readers judge whether the predictions are reasonable and will suggest which components are most important in driving the emissions reductions associated with the regulation.
EPA should clearly state the projected baseline statistics used in estimating health benefits, including those for air emissions, air quality, and health outcomes.
EPA should assess the degree to which modeled predictions agree with measured observations that have not been used to derive or calibrate the model. The results of those comparisons should be presented in the benefits analysis and used to help characterize the uncertainties associated with the resulting modeled predictions.
More emphasis should be given to the assessment, presentation, and communication of changes in morbidity and quality of life. Although often difficult to quantify, these factors may begin to play a more dominant role in benefits analysis if the value assigned to mortality decreases.
EPA should improve the methods used to account for the spectrum of severity of clinically diagnosed illnesses. When appropriate, EPA should also use data from non-U.S. studies in its benefits analyses to broaden the age ranges to which current estimates apply and to include more types of relevant health outcomes.
EPA should strive to present the results of its health benefits analyses in ways that avoid conveying an unwarranted degree of certainty, such as by rounding to fewer significant digits, increasing the use of graphs, and placing less emphasis on single numbers and more emphasis on ranges.
EPA should place the results of its health benefits analyses in context by referring not only to absolute numbers of avoided adverse health outcomes but also to total projected numbers of these outcomes and to population sizes. For example, an estimated number of avoided deaths in a future year should be accompanied by projections of the total number of deaths and the population size in that year.
EPA should begin to move the assessment of uncertainties from its ancillary analyses into its primary analyses by conducting probabilistic, multiple-source uncertainty analyses. This shift will require specification of probability distributions for major sources of uncertainty. These distributions should be based on available data and expert judgment.
To obtain expert judgment needed for its expanded primary uncertainty analyses, EPA should rely on internal expertise, as available, and external experts, as needed. In all cases, the experts whose judgments are used should be identified, and the rationales and empirical bases for their judgments described.
As EPA incorporates additional sources of uncertainty into its
primary analyses, it should analytically determine which uncertainty sources have the greatest influence on the mean and spread of the probability distributions. The uncertainty sources that have the greatest impact on the spread of the distribution should receive high priority for additional research.
In presenting the probability distribution for each health benefit estimated in the primary analysis, EPA should more clearly identify the sources of uncertainty that are not evaluated in the primary analysis.
Although the results of the benefits analyses may appear to be less certain, EPA should describe the uncertainty as completely and realistically as possible, recognizing that regulatory action might be necessary in the presence of substantial uncertainty.
EPA should consider providing preliminary analyses that estimate in current populations the health benefits resulting from hypothetical changes in current levels of emissions. Such preliminary analyses would help EPA develop an idea of the lower bound on the range of uncertainty. These analyses also would have fewer uncertainties than estimates based on projected future population exposures and health outcomes.
In all stages of the benefits analysis, EPA should justify and clearly describe the assumptions and methods used to estimate health benefits.
Each benefits analysis should be accompanied by a brief summary, such as 20 to 30 pages in length, that provides all critical elements of the analysis and the results, so that the reader can approximately estimate the benefits on a national level from the information provided.
To enhance the quality of future regulatory benefits analyses, a standing, independent, technical review panel should advise EPA in the initial stages of its benefits analysis. This panel should have expertise in regulatory options analysis, emissions and exposure assessment, toxicology, epidemiology, risk analysis, biostatistics, and economics and should be appointed with strict attention to avoiding conflict of interest, balancing biases, and ensuring broad representation. The panel should also be supported by permanent technical staff to ensure consistency of reviews over time. EPA should follow the panel’s guidance on the need for peer review.
In reviewing EPA's health benefits analyses, the committee identified several research needs. Some are relevant to improving the scientific basis for estimating the health benefits of further reductions of PM and other air pollutants. These research recommendations are mentioned in the body of the report. Others have to do with the development of improved methods for health benefits analyses in general. The research recommen-
dations include the need for improvements in the following areas: (1) methods for using expert judgment in support of health benefits analyses, (2) methods for characterizing uncertainty surrounding causal interpretation of epidemiological findings, (3) efficiency and characterization of uncertainty in the atmospheric fate and transport models used in support of health benefits analyses, (4) health surveillance systems to characterize morbidity outcomes, and (5) analysis of mixtures as well as the single pollutant.