Improving Policy Analysis: Research on the Decision-Making Process
Scientific research is of little practical value if it does not address the issues that matter to decision makers and reach them in a useful form. Failure to demonstrate practicality can threaten support for research programs whose budgets are justified by policy concerns. That burden of proof must be borne not only by the more directly applicable research (e.g., short-term impact assessments), but also by the fundamental research designed to support applications. Fundamental research should provide decision makers with the information needed for long-term planning, by indicating the extent of existing uncertainties and the rate at which they might be reduced.
Concern for the practicality of environmental research was a focal topic in the committee's 1992 report (National Research Council, 1992) and a more recent report prepared for Congress (U.S. Congress, Office of Technology Assessment, 1993). Such concern underlies the expansion of the USGCRP to promote integrated assessment and recent planning on the human dimensions component of the USGCRP (Cantor et al., 1993). The practicality of global change research depends on bridging the gap between decision makers, who need answers about how global change might affect their decisions and how their decisions might affect global change, and scientists, who create knowledge about global change that ideally provides the answers that decision makers seek. This gap is a normal consequence of the differing priorities of decision makers and scientists, and it does not resolve itself automatically. We believe that global change research is an area in which the gap is particularly problematic and that it is likely to remain a source of difficulty for the research program unless special efforts are taken.
Consequently, we see bridging the gap as a critical research problem. We propose a systematic research program, building on work already done in the field (Fischhoff and Furby, 1983; Fischhoff, Svenson, and Slovic, 1987; Rubin, Lave, and Morgan, 1992), to help ensure that relevant environmental science is conducted and that its results are provided to decision makers in usable form.
Aspects of these problems recur in all decisional domains in which large, semiautonomous research programs have developed around major policy issues (e.g., AIDS, health care reform). Global change, however, presents an extreme degree of complexity and uncertainty. Not only is the relevant scientific knowledge imperfect, but also a considerable effort is needed to assess the extent of its uncertainty. Moreover, it is characterized by well-publicized disagreements among specialists, whose respective credibility is difficult for decision makers to judge. In such a situation, special efforts are needed to enhance mutual understanding between the producers of knowledge about global change and the consumers of that knowledge. Better communication will guide scientists toward producing more useful knowledge and decision makers toward making better use of what is produced.
We do not believe that effective communication will occur merely as a result of exhortation to try harder and a commitment to doing so. But systematic research using methods from decision theory and related fields can help by specifying the sources of communication failures and offering methods for bridging the gap (Raiffa, 1968; von Winterfeldt and Edwards, 1986). We propose a research program to improve global change decision making by building on systematic analyses of particular decisions and decision contexts. The research program would include both formal and descriptive analyses and would require the participation of all the social and natural sciences needed to characterize particular decisions, as well as representatives of the decision makers.
Research would begin from the recognition that a body of scientific knowledge may be relevant to decision makers operating at quite different levels. For example, decision makers who need answers to questions about the potential for coastal zone flooding include homeowners, municipal officials, managers of Superfund sites in coastal zones, developers, local citizens' groups, Federal Emergency Management Agency officials, and flood insurance providers. These decision makers will vary in what issues interest them and what level of detail they need. Estimates of sea level rise and storm surges will be particularly relevant to decision makers with long time horizons, such as regional planners, investors in long-lived capital equipment, and insurers. In order to be useful to such a range of actors, scientific knowledge needs to be developed and delivered with their specific decisions in mind.
Analyses of decision needs will typically show that for some decisions existing environmental science is relevant, whereas for others it is not; that scientific knowledge that is relevant today may not be relevant later (and vice versa);
that the future relevance of knowledge will depend on the policies that are pursued in the interim—some of which will be affected by current research. Analyzing these possibilities requires intimate knowledge of both the decisions and the relevant environmental science. Without that link, decision making will suffer, by being deprived of relevant results, and environmental science will suffer, by being less able to make the case for its usefulness.
RELATION TO USGCRP PRIORITIES
A focused research effort on decision making and global change will support the wider needs of the USGCRP in at least the following ways:
it will support the development of methods for integrated systems assessment, as planned within the human dimensions segment of the USGCRP, by connecting those assessments with specific decisions;
it will assist the wider USGCRP initiative on integrated assessment by developing ways to translate decision makers' questions into priorities for global change research;
it will complement research on the economics of global change (e.g., work on the value of global change information) with a perspective that addresses a wide range of decisions as well as identifying barriers to effective decision making; and
it will establish communication channels between the environmental sciences and decision makers, leading to the better use of existing science and better focus for future science.
TIMELINESS OF EFFORT
Focused research on decision making and global change is particularly timely now for at least the following reasons:
developing methods for integrated assessment and improving links between research and policy have assumed pressing importance within the USGCRP;
techniques of decision analysis, not widely used now within the USGCRP, are available and relevant; and
a decision-making perspective provides a reasoned and orderly framework for the environmental sciences to address the increasing demands for accountability, by demonstrating the practical differences that the results of their research make.
Research would address such issues as:
What are the specific details of the decisions that various actors face with regard to anticipated global changes? What options do these decision makers perceive (and overlook)? How well do they understand the consequences of those actions? What weight do they assign to different consequences? What time frame do they consider? What do they extract from reports of scientific findings? What cognitive, political, cultural, and institutional factors shape their thinking about decision alternatives or constrain their decision making?
How can research be assigned priorities according to its potential value to different decision makers?
How do decision makers' information needs depend on the level of aggregation of the information and the scope of their decisional power?
How can complex data be summarized in ways that are faithful to the underlying science and pertinent to decision makers? How can scientists assess the limits to current knowledge and the opportunities for reducing uncertainty?
The box (opposite page) suggests one research approach that may be used in various decisional domains, such as land use, energy policy, and the evaluation of environmentally benign technologies. In this approach, an actual set of decisions is analyzed so as to describe the decisions as seen by key decision makers, relate them to available knowledge, identify the gaps between the two, and present the scientific community with an analysis of those gaps. A goal of each project in the decision-making research area would be to show how pending research can best be directed to address decision makers' questions. Each project would consider the variety of decision makers (e.g., officials of national, state, and local governments and international organizations, corporate executives, households, leaders of nongovernmental organizations and social movements) that influence policy, the potential conflicts among them, and the institutional obstacles to effective action (e.g., perverse incentives, jurisdictional obstacles). In addition to providing insight into the relations of science and policy in particular domains, the projects would identify research needs in the social and decision sciences. Over the long term, the research would develop general methods for improving the match between scientific production and decision makers' needs. That effort would require fundamental research on such topics as: (a) the cumulative effects of imperfections in individual decision making on societal processes, (b) procedures for eliciting scientists' beliefs regarding the definitiveness of their research, (c) procedures for comparing environmental outcomes with other (e.g., economic) impacts, (d) methods for determining people's current and desired time horizons, (e) factors that shape decision makers' ways of thinking about their decisions, (f) analytical procedures estimating the practical
IN-DEPTH ANALYSIS OF GLOBAL CHANGE DECISION MAKING
One strategy for decision-making research is to analyze in depth a suite of interrelated decisions in which a global environmental change (and the associated sciences) play a central role. Examples include decisions about land use, energy policy, the implementation of market and price-based mechanisms to secure sustainable development, the evaluation of environmentally benign technologies, and the quantification of environmental effects.
The initial research goal would be to create detailed descriptions of the decisions faced by key decision makers in a particular domain. To fully understand the dynamics by which decisions are made, the full range of participants in a decision-making process will have to be considered. The key decision makers will vary from one context to another but will typically include government officials at various levels, corporate officials, leaders of social movements, and members of the public. The descriptions would incorporate the perspectives both of the decision makers and of expert observers of their circumstances. At the core of each description would be alternative characterizations of the decision, in decision analytic terms, created with sufficient precision to allow formal analyses. Surrounding this formal core would be narrative accounts of the assumptions made in creating the analytical characterizations, the weaknesses in these representations, and the issues that are omitted altogether.
The second goal of each example would be a systematic analysis of the implications of existing environmental research for these specific decisions. That analysis would begin by identifying discrepancies between expert and decision maker perspectives on the decision. It would proceed to a diagnosis of the reasons for the existence of these disagreements, accepting the possibility that both perspectives might be in error. The analysis would continue by evaluating the impacts of bridging the gaps by providing better scientific information, including summaries of research in decision-relevant terms. Given the reporting conventions of most sciences, this step would involve eliciting judgmental assessments from substantive experts, in addition to reviewing published studies. An estimate of the potential usefulness of existing research could then be derived by comparing the expected value and definitiveness of decisions taken with and without the benefit of various results.
The research would then pose the informational needs of decision makers to environmental scientists, asking what they could do to satisfy those needs. Scientists' answers might include both new projects and the reworking of existing data. These scientists would be asked to focus on both short-term and long-term projects. The emphasis would be on usefulness of research results for specific decisions and not just their general ability to inform. Attention would also be directed to the fundamental research needed to achieve decision-relevant results. However, the inferential chain would have to be laid out explicitly (e.g., we need to build x, in order to learn y, which will help z to decide whether to take action a or action b). In addition to its substantive results, the process would also help the individuals involved learn to speak to one another.
yield of possible research programs, and (g) ways for setting priorities among scientific information in terms of its practical usefulness.
RELATION TO INTERNATIONAL RESEARCH
There is currently no comparable international research effort. There are, however, decision scientists in most of the Western democracies and the former Soviet bloc who could conduct comparable research. An international effort would help in judging research priorities in terms of the needs of foreign, international, and transnational decision makers.
At least initially, this research should be done in interdisciplinary teams, involving both natural and social scientists, with the latter in a lead role. A strong argument can be made for much of this research to be conducted by groups in individual institutions with long-term support because of the difficulty of creating a comprehensive perspective on the decisions being studied, as well as a common language among the participants. There would also be possibilities for smaller investigator-initiated projects. Each might pursue issues that are neglected by the larger teams, fundamental research topics that have emerged from them, and applications in new domains. In the interests of transfer of technology, researchers involved in major research projects should develop working relationships with relevant agency personnel.