5
Methods for Evaluating Scenarios: Reconciling Quantities and Values
Richard Bernknopf, research professor at the University of New Mexico, presented a collaborative project among his research group, the U.S. Geological Survey (USGS) and Sandia National Laboratory. The Net Resource Assessment (NetRA) is an assessment of the societal tradeoffs and impacts on ecosystem services with a resource extraction or economic development activity. The assessment is driven by the USGS Energy and Minerals Science Strategy and Department of the Interior Secretarial Order 3330. The goals of the USGS Energy and Minerals Science Strategy aim to provide inventories and assessments of energy and mineral resources, and to understand the effects of energy and mineral development on natural resources and society.1 Additionally, Secretarial Order 3330 states that “the Order will ensure consistency and efficiency in the review and permitting of new energy and other infrastructure development projects . . . while also providing for the conservation, adaptation and restoration of our nation’s valuable and natural and cultural resources.”
Dr. Bernknopf said that the NetRA is an analytical component of a multi-resource analysis (MRA) and is accompanied by a decision support tool. It takes account of current societal decision-making demands and is an expansion of natural resource assessments that include ecosystem services. The current USGS resource assessments, he said, can be broadened by developing a Decision Support Tool (DST)-oriented conceptual framework that assesses the benefits, costs,
_________________
1 Ferrero, R.C., Kolak, J.J., Bills, D.J., Bowen, Z.H., Cordier, D.J., Gallegos, T.J., Hein, J.R., Kelley, K.D., Nelson, P.H., Nuccio, V.F., Schmidt, J.M., and Seal, R.R. 2013, U.S. Geological Survey energy and minerals science strategy—A resource lifecycle approach. U.S. Geological Survey Circular 1383–D, 37 p. [Available at: http://pubs.usgs.gov/circ/1383d/circ1383-D.pdf].
and societal tradeoffs associated with the collocation of natural resources and ecosystem services. The DST-oriented conceptual framework also provides a tool for policy analysis with simulation capabilities. The tool for policy analysis is a system dynamics model that allows for the simulation of multiple resource development scenarios and provides a comparative analysis of the outcomes of different policies and practice.
The NetRA is initialized with USGS natural resource and ecosystem data, and integrates multiple collocated resources to generate usable development scenarios, he explained. It contains both spatial and temporal components to estimate the net societal benefits for a scenario of regional natural resource development. It also applies the DST systems dynamics model to evaluate specific resource development scenarios resulting from landscape conversion. Dr. Bernknopf said that there was a site selection process with USGS to establish a set of formal criteria to evaluate active or the potential development of a resource. The criteria included collocation of ecosystem services and tradeoffs of interest, any public land involved, available data, and potential decisions for the Bureau of Land Management (BLM). Of five candidate sites (the Greater Green River Basin, San Juan Basin, Piceance Basin, Bakken Shale, and the Uranium Time-Out Area), the Piceance Basin in northwestern Colorado was selected, in part, because it was less controversial than the other sites.
Resources in the Piceance Basin are intertwined in a way that extracting any one resource would have an effect on the overall ecosystem services. Natural gas extraction by hydraulic fracturing was chosen as the resource assessment unit. There are diverse opinions among stakeholders on hydraulic fracturing and the potential for development in the Piceance Basin; however, there is currently no natural gas being extracted. The group decided that the ecological assessment units defined would have geographic boundary, surface and subsurface components, and clearly identifiable and measurable ecosystem services. Habitat fragmentation, water quality, and elk migration were chosen. In addition to the complexity of the resources, Interstate 70 cross-cuts the study site and the Colorado River is down slope from the site. Comparing elk migration patterns with shale borehole locations and leases reveals that the elk migration corridors are collocated with the gas resources. Any development of the gas resources would have an impact on the elk regardless if the migration patterns were permanent or only seasonal.
Dr. Bernknopf said that the DST would incorporate multiple disciplines including biology, ecology, geology, hydrology, economics, and engineering; integrate natural resource and economics value concepts, methods, and data; and couple models and scientific data with market and nonmarket prices. Additionally, it would estimate the net societal benefits of all natural resources developed and preserved. It can be applied at multiple scales (e.g., interregionally or intraregionally), used temporally as well as spatially, and operated in multiple regulatory constraint frameworks. The assessment aims to find societal benefits
impacted by environmental impacts, development costs, and natural gas production. Scenarios can be developed to demonstrate how changing one component affects the others. For example, by focusing more on development, the environmental impacts and net societal benefits can be assessed. This is done through a series of models that form a network of stocks and flows—the stocks are the resources in place and the flows are the development through time and space. He said that assessing feedbacks between the stocks and flows is also important.
Dr. Bernknopf presented the production function model for the extraction of natural gas by hydraulic fracturing. There are multiple components in the model, however, that are still being developed and that do not have data available. Additionally, there are sub-models to the production function model. The geology flow rate sub-model captures characteristics such as number of wells, depth of wells, and permeability of the rock. It captures the characteristics that are used in geological assessments to develop a flow rate, which is a well-known indicator used in practice. This geologic sub-model feeds into a hydraulic fracturing sub-model, which assesses the development of the resource with number of wells, distance of horizontal expansion, pad density in the area, and number of roads. This assessment then feeds into a cost sub-model to estimate the total cost of extraction. He also presented an ecological model that would assess the percentage of elk population that would be displaced and how the migration patterns would shift. Inputs into this model included development characteristics, such as road sizes, road density, pad sizes, and pad density in the area. The outcomes from the different models are condensed in the RAU, which quantifies the fuel or mineral development, and the EAU, which quantifies the environmental and ecosystem impacts (Figure 5-1). Reserves and resources can be derived from the RAU and water quality and habitat fragmentation can be derived from the EAU—both are ultimately used to assess the total social benefit of development.
Dr. Bernknopf said that the NetRA project is in the proof-of-concept stage to demonstrate the conceptual framework’s feasibility. To do this, however, there are still data for inputs and issues with model compatibility that need to be solved. Several data gaps exist, including natural resource stocks, engineering economics for resource extraction, biophysical and ecological data for ecosystem services stocks, market prices, regulations, and nonmarket values. The output of the proof-of-concept, he said, will be a modeling framework that demonstrates the functionality of the decision support tool. The decision support tool will be a probabilistic model that will provide a distribution of outcomes, expected values, and uncertainty. The conceptual framework and DST will attempt to meet development needs (e.g., natural gas resources) and ecosystems needs (e.g., limit impacts on terrestrial ecosystems) by objectively providing a probability distribution of outcomes that allows the decision maker to compare one scenario to another.
Karen Jenni, founder and president of Insight Decisions, is developing an integrated MRA with the USGS for a defined geographic region in the Powder
FIGURE 5-1 Outcomes from different models are condensed in the RAU and EAU, which are used to assess the total social benefit of development.
SOURCE: Richard Bernknopf, Presentation, National Academies of Sciences, Engineering, and Medicine Workshop, June 2, 2015, Washington, D.C.
River Basin in Wyoming and Montana, which she highlighted in her workshop presentation. She is evaluating how to combine an analysis of existing resources under current conditions as well as under different future scenarios. An MRA, she said, consists of three main components: integrated information on the current status of multiple natural resources (including ecosystem services), models describing the interrelationships among collocated natural resources, and analyses evaluating the impacts and tradeoffs to the natural resources in biophysical and socioeconomic terms (Figure 5-2).
For the Powder River Basin, Dr. Jenni conducted a regional inventory and assessment of multiple natural resources, and included summary-level descriptions of resources suitable for different audiences. She also has conceptual models and analyses that address the relationships between the natural resources in the region; however, she is still early in the process of developing the models. One area of importance, she emphasized, is the resource valuation and economic implications of changes in resources under different future scenarios. An important component for stakeholders is a “portal” to access information (or multiple “portals” but with resource information available through each portal). The current interface is a Geographic Information System (GIS)-based system that provides detailed resource information until ultimately a source document,
FIGURE 5-2 The three main components of a multi-resource analysis (MRA).
SOURCE: Karen Jenni, Presentation, National Academies of Sciences, Engineering, and Medicine Workshop, June 2, 2015, Washington, D.C.
such as a USGS report, is provided. This was a critical feature to USGS and to other stakeholders—maintaining transparency and traceability of data used in the models.
Dr. Jenni described effective approaches for measuring and comparing values provided by multiple resources within a landscape-based approach. Involve decision makers and stakeholders early and often, she said, and start with the objectives of the decision makers or stakeholders, and allow them to specify their values. She also suggested acknowledging difficult-to-quantify values, and to the degree possible, separating “technical” questions (e.g., biophysical outcomes) from “value” questions. She described an example of the efforts with the Landscape Conservation Cooperatives in the Pacific Northwest that engaged tribes and First Nations in Canada to recognize traditional ecological knowledge and values that were important to their tribal communities. Although some of the traditional knowledge was difficult to value, it was still important to recognize. The values of first foods, for example, were challenging. Salmon is an evident food that had value assigned to it, but there were other first foods important to the tribal stakeholders that did not fit into an evaluation framework. Lastly, she said, it is important to maintain transparency throughout the process when aggregat-
ing values, with no “black boxes.” In engaging stakeholders in the Powder River Basin, she said many are interested in knowing more about the whole process and even in how modeling is conducted. It is important but challenging to engage stakeholders at such a level of granularity within a project conducted on a large scale assessing multiple resources.
Dr. Jenni stated that decision analysis as a quantitative methodology brings together three important characteristics. The first is that decision analysis provides a process and structure to addressing a resource management issue. For example, decision analysis can help define the scope that would be addressed under a multi-resource assessment. Clearly defining the decisions to be supported and outcomes that are important to decision makers will limit the overall scope, as well as scope-creep, in a resource management project. Bringing many researchers together on a project can result in more information than what is needed to address a particular issue; it is important to keep focus on a set of outcomes that decision makers have expressed are important.
Second, decision analysis offers many tools for quantifying uncertainty. It is used extensively in modeling efforts to reduce overall complexity of models. Expert assessment can be used when not every step in a process is modeled, meaning that judgment can be used to capture what is known and not known for modeling. Uncertainty is used to summarize what is known in the model and to move beyond the unknown steps. It helps to explain how certain modelers are about their outcomes and to infer what may happen if they are wrong.
Third, decision analysis allows for multi-attribute or multi-criteria approaches. Dr. Jenni said that often stakeholders do not want separate outcomes for different priorities combined into just one value, but instead want to see how different decisions may impact each of the priorities separately. It is important for stakeholders to have common goals so that decisions can be identified to best reach those goals. Such agreement is a challenge, she said, especially when decisions are on a large, landscape-based scale.
Dr. Jenni said that the steps necessary to link biophysical quantities to values, and to ensure biophysical measures or models are consistent with quantification, start with decision makers’ and stakeholders’ desired outcomes. It is important to start with understanding what decision makers and stakeholders want and then working backwards in the process. Identify the values linked to a decision framework and to scenarios of interest, and then identify biophysical measures that relate to those values. Also, she said, it is important to document the whole decision analysis process, which will also limit the over-modeling that often occurs.
Robert Johnston, director and research professor of the George Perkins Marsh Institute at Clark University, discussed methodologies for linking quantities and values, which he described as very challenging to do empirically even if simple to do conceptually with existing frameworks. Dr. Johnston started with a few key points. A landscape-based analysis is only as good as its weakest link, he stated. For example, if a decision support tool is used, but the science incorpo-
rated into it is not very good, then the entire analysis can be considered limited. Also, there are often arbitrary constraints and expectations on an analysis. Often, an analysis is approached with an idea of how an issue would be incorporated into a decision support tool, a requirement to assess values across an entire landscape, or with a specific way in which an agency would address an issue due to cultural or political reasons. Such constraints often make the analysis more difficult and do not necessarily comport with how systems would actually behave.
Dr. Johnston said economic evaluation as a multi-resource decision framework is designed to evaluate tradeoffs across multiple different outcomes and is explicitly designed to link the different biophysical outcomes to values. One advantage of economic valuation is that it provides a formal quantitative approach and enables internally consistent and generalizable results. Expressing outcomes in dollar values allows them to be more easily compared in a meaningful way and provides a consistent metric to measure changes in social welfare; however, the evaluation of tradeoffs does not have to be expressed as dollar values but can be other units of measure. Another advantage of expressing outcomes in dollar values is that it measures values realized by the general public, which may not be aligned with the values held by policymakers or other stakeholders. Market values and ecosystem service values, Dr. Johnston said, are subsets of economic values or benefit cost analysis. Determining tradeoff ecosystem-service values versus resource-extraction values versus market values are all the same to an economist, he said. They can be viewed as goods and services, but there are differences in how they are measured. Economic values are only one part of the assessment, and do not dictate the direction decision makers will take. They are used with other information, including other values measure using other approaches, in the overall decision-making process.
Dr. Johnston provided as an example a case study of Kitts Hummock, Delaware, which focused on eroding beaches (Figure 5-3).2 The management questions pertained to spending money to maintain beach or allowing various types of retreats of the beach. Considerations included habitat for shorebirds and horseshoe crabs, as well as housing that lined the beaches. The first step in addressing the management issue was to consolidate the considerations in the valuation process. This included compiling the cost of sand, fill, and demolition of structures, buying out homes from homeowners under retreat scenarios, recreational benefits, property transfers, and housing services. Values can be assigned to all the considerations and a cost and benefit analysis emerged. When doing an objective cost and benefit analysis, he said, the outcome can sometimes be surprising. For example, the option that provided the most benefit was to allow the beach to erode naturally.
_________________
2 Johnston, Robert J., Mahesh Ramachadran, and George R. Parsons. 2014. Benefit Transfer Combining Revealed and Stated Preference Data: Nourishment and Retreat Options for Delaware Bay Beaches. [Available online: http://works.bepress.com/cgi/viewcontent.cgi?article=1041&context=george_parsons].
FIGURE 5-3 An assessment of management options to address eroding beaches in Kitts Hummock, Delaware.
SOURCE: Robert Johnston, Presentation, National Academies of Sciences, Engineering, and Medicine Workshop, June 2, 2015, Washington, D.C.
It is standard practice, he said, to compare economic values provided by diverse market and nonmarket resources across landscapes. Substitutions of resources (tradeoffs) must be realistic and plausible. This becomes very challenging when cultural values of First Nations or native communities are involved. There are many perceived challenges when doing economic analysis, because often people have an objection to the idea of monetization and the degree to which it is applicable—a fear of economic imperialism. There is also a misunderstanding and misapplication of methods, particularly with mapping and decision-support tools. This can be true with benefit transfer methods where a value is calculated in one location and then applied in another.
External constraints are also a challenge to the economic valuation process. Often researchers impose their own constraints on the analysis. For example, he said, a researcher may set out to map value and needs to determine a value for every resource. The challenge is that the value of a particular resource in a given area is a factor of everything else around it and meaning is lost when a static value is attached to it. There are also subjective distinctions made between values. He said that different agencies may have enabling legislation or regulations that state only certain types of values can be considered. This can lead to distinctions between valuation tools or a push for one-size-fits-all decision support tools; however, he said, such arbitrary distinctions are often not supported by the scientific literature.
Typically, Dr. Johnston said, only a subset of primary values can be measured empirically. Economists often categorize values as either market or nonmarket values (Figure 5-4). Nonmarket values are often associated with observable behavior and nonuse values, such as the values of existence and for passing on a resource to future generations. Dr. Johnston noted that although the diagram for market and nonmarket values is fairly simple, it is well established. The diagram does not, however, contain ecosystem service values explicitly.
Ecosystem service values are not often a useful distinction from other types of values. Rather, he said, it is more useful to determine what biophysical elements affect human well-being, and then to label the element as an ecosystem service value if it helps communication. Often though, such arbitrary distinctions can confuse rather than help the analysis. One of the challenges is how to link biophysical models with economic ones, which can be done when natural scientists measure the right elements for valuation.
Economists do not measure the values of total ecosystems, but rather the value of changes in specific measures. It is important to determine what to count as a basis for valuation (i.e., developing causal chains or means-ends diagrams). This is done by first identifying the beneficiaries by fiat (e.g., political jurisdiction) or by analyzing where values exist (e.g., economic jurisdiction). Then, management or policy actions being considered are linked to biophysical changes in the landscape while also accounting for human behavioral changes as necessary. The benefit-relevant indicators along the chain are then identified (e.g., the
FIGURE 5-4 How economists categorize values as either market or nonmarket.
SOURCE: Robert Johnston, Presentation, National Academies of Sciences, Engineering, and Medicine Workshop, June 2, 2015, Washington, D.C.
fundamental things that people value, such as recreation, hunting, or property value). Once the indicators are identified, they should be qualitatively described to represent what people value without ascribing an actual value, after which economic valuation or another multi-criteria decision analysis tool can quantify the social value. This links biophysical changes to primary changes in social welfare.
The panel was asked a question about the user interface of decision support tools and modeling, and if there is a way to make them more interactive with non-modelers. The questioner said that often when there is a conflict on the landscape, the stakeholders want to understand a decision support tool that is being developed to address the conflict, but due to the complexity of the modeling, they come to see it as a black box. Stakeholders want to be part of the decision-making process, but there is concern their values are not being incorporated into the process. Dr. Bernknopf explained that they are trying to develop a decision support tool that allows all users to create a range of scenarios and realize outcomes without having to understand all the equations and statistics incorporated into the model.
Dr. Johnston added that uncertainty with models and the stakeholders’ comprehension of that uncertainty provides an ongoing challenge. The need to provide simplified information needs to be balanced with the need to provide accurate information. For example, when a decision support tool results in a map, there is a tendency for stakeholders to take the map as truth and not understand the extent of uncertainty underneath the representation of data. Dr. Jenni said that there is value
in using simplified, cartoon models to communicate information. Often when there are multiple disciplines involved in an assessment of a large area, such as the Powder River Basin, the modeling becomes too complicated when all interests are incorporated and cannot be explained in a more simplified way. Dr. Bernknopf added that a decision support tool can assess a range of outcomes, which helps to address some of the uncertainty.
The panelists were asked about the complexity of models and if there were any attempts to reduce the amount of complexity to determine the least amount of information needed to successfully use the models presented. It was noted that the more complexity the models contain, the fewer stakeholders would be able to use them, and the less successful implementation would be. Dr. Bernknopf replied that they are working with a range of different types of scientists and trying to accommodate everyone by keeping the models manageable for all involved yet scientifically robust. Carl Shapiro, director of the Science and Decisions Center at USGS, commented that as the sponsor for Dr. Bernknopf and Dr. Jenni’s projects, they aimed to do a proof of concept of two multi-resource analysis projects in two places to see if they could be developed in a way that was relevant and useful to decision makers. Dr. Shapiro said that they are not yet at the stage of trying to eliminate complexity from the model as they are still early in the process of modeling to see if the efforts are feasible.
A related question to the panelists was if the decision tools being developed are able to factor in variances in the decision space associated with different resources. The example given was for resources that had explicit trust laws that limited the number of possible decisions. Dr. Bernknopf responded that they are trying to find constraints over a large region, and to determine the probability of the amount of resources capable of being extracted in that region. They have not yet been able to address constraints at such resolution as specific regulations that would protect a wilderness area. Another participant commented there are two conflicting goals—the development of a sophisticated decision support tool that aggregates a tremendous amount of information for a decision maker but that is also simple enough for a nontechnical user to manipulate and understand. The participant said that there is a significant challenge in being able to have the sophistication necessary in a decision support tool while also keeping it simple enough for all stakeholders to use.
The panelists were asked about how to address stakeholders when they assign different weight to different values in a multi-resource analysis context. The participant explained that when stakeholders are provided with a range of scenarios with different outcomes and are asked to make policy or management decisions, the challenge is that they value different attributes of the scenarios and could potentially become gridlocked in making a decision. It may be useful, the participant added, to provide the multi-stakeholder decision with all the outputs disaggregated by values to compare against what stakeholders think would happen in the absence of a management decision.
Dr. Bernknopf responded that they hope to develop a decision support tool that would essentially allow for that process by providing stakeholders with virtual dials that they can use to change inputs to see how the outcomes change accordingly. By allowing all the stakeholders to create a range of outcomes over multiple scenarios, a distribution of values is provided. He said it is a probabilistic approach to the decision-making process. Dr. Jenni added that it is important to allow different stakeholders to provide their input and express their tradeoffs in the modeling process. She said there is also a challenge in identifying who the decision makers are in a large, landscape-based analysis. Many stakeholders are interested in the outcomes of scenarios, but they are not necessarily the decision makers that need to use the decision support tool.
Dr. Wainger said that developing models and scenarios is aimed to influence specific decision makers, such as landowners. The goal may be to have them install a structure, such as a buffer strip to protect water quality. The question she posed was if landowners needed to understand all the benefits they are providing to society or if they only need to understand their own personal benefits or own opportunity costs. Much of that discussion, she added, is about how much detail or data is needed for the analysis when many of the attributes of the analysis are fundamentally incompatible with each other. The other questions she posed were about when quantities are useful and their limitations for helping understand tradeoffs. Researchers often quantitatively analyze the attributes of a decision analysis but are still unable to resolve tradeoffs, which results in negotiated solutions that may or may not be based on the data available. Dr. Jenni provided an anecdote from a recent meeting where a scientist presented on a decision analysis preformed with stakeholders that resulted in a decision not based on the analysis but on the stakeholders’ own perspectives. Decisions, she said, are often not based on facts or data but on the values held by the decision maker.
Dr. Hitzman added that there are factual-based (data-based) models and value-based models. He said that there is a challenge when it comes to the decision maker using a factual-based model, because ultimately decisions are often political. He said that it is critical to consider the political reality when building decision support tools, although it may not be used until the model or decision support tool is communicated to stakeholders. The communication step, he added, needs to be separated out from the model itself. Dr. Shapiro provided perspective on the complexity of the models discussed. He said that one of the reasons for developing complex models is due to a relatively complex set of issues when conducting a multi-resource analysis with multiple stakeholders over a large landscape. The process begins by moving from a single resource analysis to trying to understand what happens to other resources in the same region, and then ultimately moving to a valuation step. There will be a necessary tradeoff, he said, between how much information will be available to inform a model and the complexity of a model so that it is not so complex that a decision maker is unable use it.
