3

Evaluating the Landscape Conservation Cooperatives Network Strategic Plan

The committee was asked to assess the scientific merit of the Landscape Conservation Cooperatives (LCC) Network and its goals. In this chapter, the committee discusses its approach to program evaluation in general, followed by an evaluation of the goals and objectives outlined in the LCC Network Strategic Plan (LCC, 2014; Appendix F). The committee evaluates this plan’s goals and objectives based on value-focused thinking (Keeney, 1992). Furthermore, the chapter briefly discusses the structure and function of the LCC Network. The committee was also tasked to review the network’s evaluation process, which is addressed in Chapter 4.

EVALUATING THE GOALS AND OBJECTIVES OF THE LANDSCAPE CONSERVATION COOPERATIVES

Meaningful evaluation requires clearly articulated and measurable objectives that reflect the mission of the program being evaluated, and that are sensitive to the context for which the program was initiated. Three kinds of objectives are relevant to evaluation: ends, means, and process objectives (Keeney, 1992; see Box 3.1).

The description of the LCC Network¹ can be parsed to show ends, means, and process objectives:

• Develop and provide integrated science-based information about the implications of climate change and other stressors for the sustainability of natural and cultural resources;
• Develop shared, landscape-level, conservation objectives and inform conservation strategies that are based on a shared scientific understanding about the landscape, including the implications of current and future environmental stressors;
• Facilitate the exchange of applied science in the implementation of conservation strategies and products developed by the Cooperative or their partners;
• Monitor and evaluate the effectiveness of LCC conservation strategies in meeting shared objectives;
• Develop appropriate linkages that connect LCCs to ensure an effective network.

Distinguishing properly between means and ends is important because there may be many ways of successfully achieving a desired end, and it would be unfortunate to preclude achieving such end objectives by too closely prescribing the means by which an ends objective should be pursued. In the LCC context, this is particularly important

___________________

¹ See http://lccnetwork.org/about.

because of the considerable autonomy of the regional LCCs in deciding how to operate within the overall LCC mission. It is important that the LCCs view certain process objectives as ends in themselves, as well as means, because of the emphasis that the overall LCC mission places on collaboration and partnerships in both the structure and the function of LCCs.

A hierarchical organization of ends objectives (and sub-objectives) is useful (1) because it helps to eliminate redundancy and identify gaps in program goals, and (2) because it leads directly to an efficient set of metrics that can be used for program evaluation (see discussion on metrics in Chapter 4). The LCC Network description quoted above includes only a few words about ends. To use these high-level, broad ends objectives to guide program evaluation requires elaboration of the aspects of each of the high-level objectives into sub-objectives. The LCC language offers some clues for sub-objectives, mentioning “economy, land, water, wildlife, and cultural resources” as being desirable aspects of sustainable landscapes. Each of those sub-objectives might be divided further, answering questions such as, What aspects of the economy are important in this context? Other possible sub-objectives are not so clear; for example, what are the elements of “sustainability” that are important? A fully expanded objectives hierarchy would identify the aspects of each higher-level objective or sub-objective, culminating in very specific criteria (i.e., metrics and measures) for evaluating the performance of programs to enhance achievement of the objectives in the hierarchy (where the activities of those programs are the means).

The LCC Network Strategic Plan (LCC, 2014; hereafter referred to as the strategic plan) is organized around four primary goals: conservation strategy, collaborative conservation, science, and communication. The strategic plan usage of “goals” versus “objectives” is quite common, but different from the value-focused usage of these terms described above. The strategic plan uses “goal” to describe high-level objectives, some of which are ends and some of which are means in the value-focused thinking. For each enumerated goal, the strategic plan identifies multiple objectives. The strategic plan uses “objective” to mean specific actions or targets needed to accomplish a goal; this usage is close to the means objectives of value-focused thinking.

The critique of the strategic plan that follows draws on the principles of value-focused thinking to analyze redundancies and potential inconsistencies in the strategic plan’s formulation of goals and objectives as a first step toward developing an efficient set of evaluation metrics. The process of setting metrics is discussed in detail in Chapter 4. For each of the LCC Network’s four goals (conservation strategy, collaborative conservation, science, and communication), the committee first analyzes the goal itself with respect to its status as an end in itself or a means to a desired end. Then,

we present the objectives associated with that goal in a table, followed by analysis of the status of those objectives with respect to ends, means, and process.

Conservation Strategy

Devising a conservation strategy can be considered the overarching goal (end) for the LCC Network to pursue because the purpose of the entire program is to improve conservation effectiveness. The remaining three goals are really “means” to achieving this end, although to some degree they also can be viewed as ends in themselves. The full goal, as expressed in the plan, is to attain “[a]n ecologically connected network of landscapes and seascapes adaptable to global change—such as climate change—with the ability to sustain ecological integrity and health to meet the needs of society at multiple scales.” We note that this is not a strictly scientific goal—“ecological integrity” and especially “ecological health” are not definable without reference to human value judgments (De Leo and Levin, 1997). Indeed, meeting human needs, which are associated closely with values, is part of the overarching purpose of the LCCs. This is an appropriate task of landscape-scale conservation, and it indicates the importance of obtaining broad agreement on which human needs and values will be considered when choosing actions and how progress toward meeting those needs will be measured. The objectives included in the strategic plan address the need to consider and identify a range of stakeholder priorities (and are means to the desired end; see Table 3.1).

All five objectives describe actions to be taken to accomplish the goal of landscape-level conservation and they are all means to this desired end.

Objective 1-1. As part of this objective, the strategic plan (see also Appendix F) specifies two example tactics: “establish conservation objectives at LCC and other applicable scales” and “roll-up LCC objectives to identify Network-scale objectives.” While these tactics do not add much information, they do suggest a sequence of activities: first develop objectives at the LCC scale and then see if network-scale objectives emerge or can be extracted. Objective 1-1 is critically important. As discussed in Chapter 2, how well this particular means objective is achieved plays a critical role in realizing conservation outcomes.

Objective 1-2. The strategic plan’s example tactics to promote resiliency and adaptation include working with various partners and indigenous peoples to identify “flagship regions” and priority areas for conservation; to provide tools and guidance for implementing and supporting conservation designs; and to acknowledge the extensive knowledge and practices of indigenous people. Objective 1-2 describes ways of communicating and supporting the results of Objective 1-1. As discussed in Chapter 6 and Appendix C, this objective is important to achieving the broader conservation goal of the program. It is not clear what the role of stakeholders is in disseminating this information, nor is it clear how the extensive knowledge and practices of indigenous peoples will be incorporated into the conservation designs, rather than simply being acknowledged.

Objective 1-3. Objective 1-3 is the program’s means to scale up Landscape Conservation Designs to achieve conservation goals at the network scale. This task is an important rationale for creating the LCCs: to improve capacity for addressing broad-scale and cross-boundary problems that are unlikely to be addressed effectively through existing institutions and mechanisms. Further emphasis on Objective 1-3 will also help clarify the connection between each LCC’s goals and regional- to network-scale goals (see discussion below). This consideration deserves more attention than has been provided to date, as discussed below.

TABLE 3.1 Brief Description of LCC Network Objectives for the First Strategic Goal, “Conservation Strategy”

^aLandscape Conservation Design is an iterative, collaborative, and holistic process that provides information, analytical tools, spatially explicit data, and best management practices to develop shared conservation strategies and to achieve jointly held conservation goals among partners.

Objective 1-4. This objective is critically important because work cannot be done without resources, which are means to the desired ends. However, given the current budget environment, resources dedicated to LCC activities through specific appropriation from Congress can be perceived as reducing funds available to other goals deemed important by partner agencies. Therefore, objectives under this goal might benefit from a careful review and adjustments in order to achieve broad stakeholder buy-in on Objective 1-1 prior to proceeding with Objective1-4 (see also discussion on program life cycle, below).

Objective 1-5. Objective 1-5 is an important process objective, as it expresses a commitment to adaptive management. A critically important part of that is self-evaluation, an assessment of whether and to what degree the program has been effective in (a) developing the conservation design and (b) yielding desired outcomes from the conservation design. This process objective is one of the key factors to successful landscape-scale conservation as outlined in Chapter 2 and discussed in greater detail in Chapter 6. Note that the type of adaptive management described in this objective is “passive,” rather than “active,” adaptive management, where contrasting management strategies are employed in a quasi-experimental manner in order to more efficiently learn how management actions create resulting ecological conditions.

Collaborative Conservation

The full statement of the collaborative conservation goal is “facilitated alignment of partnership efforts within and amongst LCCs, including planning efforts and resources that improve conservation outcomes across LCCs and the Network.” This goal is intended to implement the “collaborative” element in the program’s title. It is a means to achieving the conservation ends of the LCC program, although achieving collaboration also is an end in itself, given the LCC emphasis on partnership. It is a process objective in the value-focused thinking framework. The goal has six objectives (that are themselves means). As mentioned in Chapter 2, “collaborative conservation” is a central element of large-landscape conservation, and achieving this process objective and developing high-quality collaboration will be critical to the success of the program. Under this goal, the strategic plan lists six objectives (see Table 3.2).

Objective 2-1. It is important to develop and maintain a functioning network with the individual LCCs. Example tactics from the strategic plan include identifying barriers to collaboration and seeking to break them down, and continued pursuit of partnership opportunities. This objective reflects a continuing effort to develop the collaborative network that was announced by Secretary of the Interior Ken Salazar in 2010.

Objective 2-2. This objective includes a sample tactic to identify opportunities for cooperation across multiple LCCs, an approach that would help address conservation problems that frequently span multiple LCCs. It is unclear to the committee what process is in place to facilitate the identification of such opportunities and how the individual LCCs or the network as a whole would measure progress toward this objective.

Objective 2-3. This objective lists an example tactic to develop common definitions and performance metrics for key qualitative and quantitative outcomes that demonstrate value. Measuring effectiveness of the network is essential. However, as discussed in greater detail in Chapter 4, the LCC Network has not yet developed an effective process to monitor or evaluate the value and effectiveness of the LCC Network. As a result, it is difficult to scale from the effectiveness of individual LCCs to the network-level value and effectiveness. Achieving this process objective will be critical to sustaining support for this network.

Objective 2-4. This means objective appears to only minimally support the collaborative conservation goal, and does not appear to be a measurable objective.

Objective 2-5. The sample tactic indicates that this means objective focuses on seeking opportunities to leverage LCC funding. It is unclear how this would contribute to the goal of collaborative conservation.

Objective 2-6. In contrast, Objective 6 has no example tactic. It is related to the first two objectives under Goal 1, but

TABLE 3.2 Brief Description of LCC Network Objectives for the Second Goal, “Collaborative Conservation”

focuses on operational needs rather than conservation goals. Objective 1-3 and this objective help with scaling up efforts and outcomes beyond the individual LCCs and are discussed in greater detail in Chapter 4.

Science

The full statement of the goal regarding science and the LCC Network is the following: “Natural and cultural resources are conserved at large landscape and seascape scales, guided by the collaborative application of science, experience, and cultural or traditional ecological knowledge and the generation of new conservation knowledge.” The first phrase expresses an end and the remaining phrases express means of achieving that end. This goal has three means objectives (see Table 3.3).

Objective 3-1. Example tactics specified in the strategic plan include completing, disseminating, and implementing the LCC Science Plan; identifying and developing spatial, biological, and cultural data and evaluation tools across the network; and supporting assessment tools for climate change and climate adaptation planning for important cultural and subsistence resources. The goal identifies science as important, and the objective, and especially the example tactics, focuses on cultural and subsistence resources.

Objective 3-2. The example tactic to promote collaborative science is to include resource managers, cultural practitioners, and indigenous peoples in framing resource conservation issues and management objectives. This objective reflects an important recognition that environmental problems are defined relative to particular stakeholders; they are not abstract and self-contained issues. The objective and example tactic come closer than those mentioned earlier to recognizing the importance of incorporating the various stakeholders and areas of knowledge and expertise in all stages of the process.

Objective 3-3. The example tactic focuses on cooperating, sharing, cooperative synthesis of data, and communication. The objective is similar to previous ones that focus on evaluating effectiveness of the program. However, this objective includes both means and ends components because it aims to evaluate the effectiveness (means) and improve the effectiveness (ends) of LCC science. Perhaps most crucially, implementing this objective requires performance metrics to assess “the value of the LCC science.” These metrics ideally align with the ends objectives, but that is complicated when means and ends are intermingled, as is the case here.

Communications

The full statement of the communication goal is to: “advance the knowledge of, support for, and engagement in landscape-scale conservation across the LCC Network.” In other words, the goal appears to be to build a constituency for the LCC Network across all of its parts. It appears to be a means to the overall end of improving landscape-scale conservation. The goal has five means objectives (see Table 3.4).

Objective 4-1. The example tactic given in the strategic plan includes use of the LCC Network website and other tools as platforms for sharing news and the value of the program. This objective is a restatement of Goal 4, and in this restatement, the broader user community outside the LCC Network is included, as well as the network and the individual LCCs. Clearly, implementing this goal is important for at least two reasons. First, no matter how good the products of the LCC Network might be, their usefulness will be diminished to the extent that they are not widely known. Second, without outside knowledge of the program and its usefulness, continued support of the program will be difficult. The LCC Network has a challenge in promoting its successes and usefulness without detracting from the activities of other programs that it facilitates, partners with, and depends on.

Objective 4-2. The example tactic is to identify new, strategic target audiences whose interests might intersect with conservation interests. As discussed in Chapter 2, landscape-scale conservation depends critically on a large number of stakeholders, the majority of which are private landowners.

Objective 4-3. The example tactics are to communicate LCC successes and make LCC products more widely available. This objective is very similar in concept and practice to Objective 4-1.

TABLE 3.3 Brief Description of the LCC Network Objectives for the Third Goal, “Science”

TABLE 3.4 Brief Description of the LCC Network Objectives for the Fourth Goal, “Communications”

Objective 4-4. This appears to be more of the same, basically an iteration of Objective 4-1.

Objective 4-5. This objective appears to belong under Goal 2, Collaboration. Clearly, one major advantage of having a network is to facilitate learning across wider time and space scales than would likely occur without a network; doing that is an aspect of collaboration.

TIMING OF EVALUATIONS IN THE PROGRAM LIFE CYCLE

The evaluation literature distinguishes between summative and formative evaluation (e.g., Wholey, 1996), both of which are retrospective evaluations of programs that have already been at least partly implemented. Summative evaluation takes place after a program has been completed or, in the case of ongoing programs, after a program has been fully implemented. The measurement criteria used for summative evaluation should be those associated with the ends and process objectives that appear in the objectives hierarchy, capturing the desired qualities the program is intended to enhance. However, for reasons of cost or convenience, proxy measures may be needed for some objectives, and those might be more closely associated with means objectives than with the ends that are fundamentally of interest. For example, if it is considered too difficult to directly measure the population status of a bird species dependent on large parcels of mature hardwoods, an evaluation might instead focus on the easier-to-measure extent and structure of the types of forest on which the species depends. Restoring forest habitat is a means to the desired end of enhancing the abundance of the bird, but because adequate habitat is a necessary, if not sufficient, condition for increased abundance, it can be used as a proxy measure.

Formative evaluation takes place partway through program implementation, or periodically during the course of an ongoing program, to assess progress and direct mid-course corrections. It is common for formative evaluation to use measurement of means as a proxy for accomplishment of ends because it may be too soon for even a successful program to have had its anticipated effect on ends. This is the case for the current evaluation of the LCCs, where the management of both natural and cultural resources that LCC activities are intended to influence has considerable inertia and may not show the effects of improved management for many years. In such circumstances, successful implementation of means, such as development of a nationally integrated land-use classification system, is an appropriate metric for formative evaluation.

Additional perspective on appropriate evaluation metrics for the LCC Network and for individual LCCs comes from the literature on the life cycle of governance structures for collaborative networks (Imperial et al., in press). These governance structures are viewed as having a natural evolution of both form and function. One model of this evolution lists four stages: (1) activation, where membership may be changing, relationships among members are developing, and the organization is searching for its focus; (2) collectivity, where the network has created processes for nurturing member interactions, has framed problems in a way that engages members and helps garner resources, and has codified decision-making procedures; (3) institutionalization, where the structure and role of the network have stabilized and the focus has turned to creating value through its expenditure of resources; and (4) stability, decline, or restoration, where a network may continue to garner resources and create value for some time, but may also need to adapt to address new problems or even disband, if the problems for which it was convened are now being addressed in another way. This is entirely consistent with focusing early evaluation on the means of achieving ends and on the processes by which decisions are made. In the longer term, when the LCC Network and individual LCCs have reached the institutionalization phase, it will be appropriate for evaluations to focus on creation of value as measured by achievement of the conservation ends for which the LCC Network was conceived. It is also worth remembering that reconfiguration or even dissolution can be part of the natural evolution of a collaborative network and that such an end casts no shadow on the value created by the network during its lifetime.

RELATIONSHIP OF THIS FRAMEWORK TO PERFORMANCE MEASUREMENT AND EVALUATION OF GOVERNMENT PROGRAMS

The federal agencies participating in LCCs are obliged to conduct performance measurement and program evaluation in the manner prescribed by the Government Performance and Results Act of 1993 (GPRA, 1993) and the Government Performance and Results Acts Modernization Act of 2010 (GRPA, 2010).²Chapter 4 critically examines an existing set of evaluation metrics used by the FWS to assess performance of the LCC system thus far (the Science Investment and Accountability Schedule); these metrics follow the Office of Management and Budget (OMB) guidelines. Chapter 4 also uses terms (e.g., process, outputs, outcomes) consistent with OMB guidance to propose new metrics for assessing LCC system performance in the short, medium, and long term. Because these terms are used commonly to discuss government programs, it seems worthwhile to note where the value-focused thinking framework used here, with its emphasis on distinguishing between ends and means, does and does not correspond to OMB terms.

OMB uses outputs to refer to products and services delivered by a program (see also Box 4.1); this terminology corresponds roughly to means in the value-focused thinking framework. OMB uses outcomes to refer to the results of those products and services; this corresponds roughly to ends in the value-focused thinking framework. The charge to this committee referred to outcomes and outputs consistent with OMB guidelines. In describing different types of program evaluation, OMB includes measurement of both outcomes and outputs as components of what it calls outcome evaluation. In contrast, value-focused thinking urges evaluation of programs with respect to their effects on ends, rather than with respect to their employment of particular means. However, value-focused thinking does recognize that means could appropriately be the focus of program evaluation, either as a proxy measure, where it is too difficult to directly measure accomplishment of ends, or as an interim measure, where it is too soon to expect to see any effect on ends, even for a successful program. OMB uses process to mean the type or level of activities conducted by the program, including adherence to regulatory requirements and professional standards. This is rather different from the value-focused thinking usage of process to refer to qualities of the way that a program undertakes its business (e.g., collaboratively).

CONCLUSIONS

In summary, similar to the language in the Secretarial Order, the strategic plan speaks mainly about means rather than about ends. The strategic plan does call attention to process objectives, as is appropriate for a program that intends to promote collaboration. The discussion above points out instances where goals appear to be means, rather than ends; where objectives seem to be misplaced under their respective goals; and where there are significant redundancies both among objectives under the same goal and among objectives that appear under different goals. We suggest reworking the language that describes the LCCs’ overarching goals, vision, and mission with the aims to distinguish ends from means, to clarify process objectives that should be considered ends in themselves, and to elaborate sub-objectives. Such a reworking using the framework of value-focused thinking would lead clearly, and without redundancy, to evaluation metrics (see Chapter 4 on metrics). The same restructuring applied to the strategic plan would also facilitate development of succinct and operationally meaningful metrics for guiding implementation of the LCC system and evaluating its success. Upcoming revisions of the strategic plan offer opportunities for this restructuring.

The same advice about distinguishing ends and means, organizing ends objectives hierarchically, and using that hierarchy to develop an efficient set of metrics applies to individual LCCs and subgroups of LCC partners who are developing strategic plans under the LCC umbrella. These groups will be developing and revising strategic plans (including goals and objectives), choosing and implementing science and management activities, and evaluating their success in advancing the overall goals of the LCC system. Using a systematic approach to these recurring tasks will help make program evaluation more efficient and more transparent. It will also facilitate the essential task of relating the goals of the regional LCCs and their partner organizations to those of the national-level LCC Network.

Linking Individual LCCs’ Goals with the Network’s Goals

The strategic plan includes several goals and objectives that aim to facilitate coordination and collaboration across the individual LCCs and create a “high-functioning organizational culture for LCCs and the Network” (see above, Objectives 2-1, 2-2, and 2-3). Objective 2-3 of the LCC Network’s goal—to develop conservation strategy—points to a need to improve the capacity for addressing broad-scale and cross-boundary problems. Identifying and addressing issues that span multiple LCCs or the entire network are unlikely to be addressed effectively through existing institutions and mechanisms. However, the network does not explicitly describe a process or structure that assists in identifying priorities at the larger or network scale. Nor are there metrics in place to aggregate outputs and results at the network scale (see Chapter 4 for further discussion). As discussed in Chapter 2, the landscape approach requires a bottom-up approach. The LCCs have each engaged in a strategic planning effort and begun to identify their strategic conservation priorities. As discussed in Chapter 4, each LCC is currently evaluated

___________________

² See https://www.whitehouse.gov/omb/mgmt-gpra/index-gpra and http://gao.gov/new.items/d11646sp.pdf.

using the Science Investment and Accountability Schedule evaluation tool, which is based on goals and objectives outlined in the Strategic Habitat Conservation Handbook (National Technical Assistance Team, 2008). The current difficulty in evaluating the LCC Network (discussed in Chapter 4 in greater detail) arises from an apparent lack in a process that clearly links the individual LCCs’ strategic goals and missions to the LCC Network’s strategic plan.

Many cross-LCC efforts are under way. However, developing an actionable strategic plan with clear national conservation priorities based on the conservation priorities of the individual LCCs has yet to be completed. The LCC Network staff have developed a table that illustrates how the goals of each LCC compare to the goals of the LCC Network Strategic Plan (see Appendix G). However, an evaluation process that can aggregate the accomplishments of individual LCCs to the network goals does not yet exist. To be able to do that, the evaluation tool for the individual LCCs has to change or the network’s strategic goals/objectives have to more closely map to each LCC’s goals (see additional discussion in Chapter 4).

The goals and objectives in the current strategic plan include most of the critical elements of the landscape approach discussed in Chapter 2. Overall, the strategic plan is consistent with the latest research on conservation at the landscape scale. The plan includes such critical elements as stakeholder engagement, adaptive management, and delivery of landscape-scale designs at the regional level with the aim to scale up at the LCC Network level. As discussed in Appendix C and elsewhere in this report, the guidance to implement the strategic plan needs to be improved. Developing science and information needs collaboratively with resource managers will be a critical process objective (and both an end in itself and a means of securing other ends), as discussed in Chapter 2.

Because the LCC Network was established in 2009, and some individual LCCs more recently, formative rather than summative evaluation is most appropriate at this stage. Formative evaluation can consider implementation of means (activities, outputs) as interim metrics of program success. Summative evaluation, concentrating on the effects that LCC activities have had on ends, will become more appropriate as the LCC program matures. As discussed in the section on timing of evaluations in the program life cycle, it is important for evaluation that the criteria being used to assess the health of the network during each phase in the program is tied to the structures and tasks appropriate to that stage. The LCC Network and its component LCCs are very much in the activation and collectivity stages of development, where evaluation is appropriately focused on engaging members; building relationships; and developing procedures for garnering resources, selecting projects, and carrying out activities.

Recommendation: When developing and revising strategic plans for the national network and for regional LCCs, the plans should distinguish ends, means, and process objectives, and organize ends objectives hierarchically to facilitate creating (and maintaining) an efficient set of evaluation metrics.

Recommendation: The LCC Network staff should conduct formative evaluation on an ongoing basis (e.g., annually) to guide LCC program implementation at regional and national levels.

Recommendation: The LCC Network should conduct summative evaluation periodically (perhaps every 5 years) for LCC programs that have been in existence long enough to have had a perceptible impact on ends (outcomes) to assess the values being enhanced by LCC activities.