Skip to main content

Currently Skimming:

Summary
Pages 1-17

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.


From page 1...
... Yet the agency has not sufficiently leveraged opportunities to improve its regulatory decisions by adopting a comprehensive strategy for periodically evaluating and refining its models. This report recommends a series of guidelines and principles that, if adopted, will improve environmental regulatory models and decisions made by the agency.
From page 2...
... convened the Committee on Models in the Regulatory Decision Process in response to a request from CREM to independently assess evolving scientific and technical issues related to the selection and use of computational and statistical models in decision-making processes at EPA. The full charge is provided in Box S1 at the end of the Summary.
From page 3...
... Regulatory model evaluation must consider how accurately a particular model application represents the system of interest while being reproducible, transparent, and useful for the regulatory decision at hand. Meeting these needs may require different forms of peer review, uncertainty analysis, and extrapolation methods.
From page 4...
... In particular, model evaluation should not stop with the evaluation activities that often occur before the public release of a model but should continue throughout regulatory applications and revisions to the model. For all models used in the regulatory process, the agency should begin by developing a life-cycle model evaluation plan commensurate with the regulatory application of the model (for example, the scientific complexity, the precedent-setting potential of the modeling approach or application, the extent to which previous evaluations are still applicable, and the projected impacts of the associated regulatory decision)
From page 5...
... Peer review for some regulatory models should involve comparing the model results with known test cases, reviewing the model code and documentation, and running the model for several types of problems for which the model might be used. Reviewing model documentation and results is not sufficient peer review for many regulatory models.
From page 6...
... Thus, although probabilistic uncertainty analysis is an important tool, requiring EPA to do complete probabilistic regulatory analyses on a routine basis would probably result in superficial treatments of many sources of uncertainty. The practical problems of performing a complete probabilistic analysis stem from models that have large numbers of parameters whose uncertainties must be estimated in a cursory fashion.
From page 7...
... Recommendations Quantifying Uncertainty In some cases, presenting results from a small number of model scenarios will provide an adequate uncertainty analysis (for example, cases in which the stakes are low, modeling resources are limited, or in
From page 8...
... Measurements help to provide the conceptual basis of a model and inform model development, including parameter estimation. Measurements are also a critical tool for corroborating model results.
From page 9...
... Adaptive strategies that rely on iterations of measurements and modeling, such as those discussed in the 2003 NRC report titled Adaptive Monitoring and Assessment for the Comprehensive Everglades Restoration Plan, provide examples of how improved coordination might be achieved. Recommendations Using adaptive strategies to coordinate data collection and modeling should be a priority of decision makers and those responsible for regulatory model development and application.
From page 10...
... Even more problematic are models that accrue substantial uncertainties because they contain more parameters than can be estimated or calibrated with available observations. Recommendations Models used in the regulatory process should be no more complicated than is necessary to inform regulatory decisions.
From page 11...
... When critical model parameters are estimated largely on the basis of matching model output to historical data, care must be taken to provide uncertainty estimates for the extrapolations, especially for models with many uncertain parameters. Proprietary Models A model is proprietary if any component that is a fundamental part of the model's structure or functionality is not available for free to the general public.
From page 12...
... usually will not require such scrutiny, although EPA should be cognizant of the costs that obtaining and using such software may impose on interested parties. MODEL MANAGEMENT Models and Rule-makings The sometimes contentious setting in which regulatory models are used may impede EPA's ability to implement some of the recommendations in this report, including the life-cycle evaluation process.
From page 13...
... Rather than requiring a response within 60 days, as the Information Quality Act does, the evaluation process would involve consideration of that information and response at the appropriate time in the model evaluation process. To further encourage life-cycle evaluation of models that support federal rule-makings, alternative means of soliciting public comment on model revisions need to be devised.
From page 14...
... The model documentation also should have elements in "plain English" to communicate with nontechnical evaluators. An understandable description of the model itself, justifications, limitations, and key peer reviews are especially important for building trust.
From page 15...
... A more formal process may be needed to ensure that CREM's models database is complete and updated with information that is at least equivalent to information provided for models currently contained in the database. Yet, even with a high-quality models database, EPA should continue to develop initiatives to ensure that its regulatory models are as accessible as possible to the broader public and stakeholder community.
From page 16...
... It will also examine case studies of model development, evaluation, and application to further elucidate guiding principles. The objective of the committee will be to provide a report that will serve as a fundamental guide for the selection and use of models in the regulatory process at EPA -- the goal is to produce a report on models similar to the NRC's 1983 "Red Book" on risk assessment (Risk Assessment in the Federal Government: Managing the Process)
From page 17...
... • How can the agency provide guidance on procedures for appropriate use, peer review, and evaluation of models that is applicable across the range of interdisciplinary regulatory activities undertaken by the EPA? • How can issues related to input data quality, model sensitivity, uncertainty, and the use of model outputs be addressed in a unified manner across the multiple disciplines that encompass modeling at EPA?


This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More information on Chapter Skim is available.