PURPOSE OF THE ANALYTIC FRAMEWORK
As discussed in Chapter 1, science underpins the achievement of Environmental Protection Agency (EPA) strategic goals by providing information to support activities as varied as enforcement of laws and regulations to the development of new regulations and identification of future threats to public health and the environment. To support its science-based activities, EPA operates laboratories for various purposes.
Given the multiple and complex needs for scientific data and insights to fulfill its mission, an analytic framework is needed to determine how EPA can effectively and efficiently deliver the scientific information that will support decision making for health and environmental protection. This chapter will describe that framework.
TOUCHSTONES OF EFFICIENCY AND EFFECTIVENESS
The analytic framework includes general criteria for evaluating EPA’s laboratory enterprise and shaping EPA’s investments, processes, and planning to ensure the enterprise is aligned with the agency’s highest-priority needs. More-detailed criteria are provided in Chapter 4. In its 2008 report Evaluating Research Efficiency in the U.S. Environmental Protection Agency, the National Research Council characterized two criteria—efficiency and effectiveness—as central in evaluating the performance and results of EPA’s research and development program. Those criteria are also of central importance in analyzing EPA’s laboratory enterprise. In defining efficiency and effectiveness, we draw on the 2008 report but expand the definitions to apply beyond EPA’s Office of Research and Development (ORD) to the full array of laboratory science programs supporting EPA’s mission.
Efficiency is the laboratory enterprise’s ability to produce relevant, high-quality, timely, and cost-effective results to fulfill the agency’s mission. It includes both investment efficiency (Is the laboratory enterprise doing the right work and doing it well?) and process efficiency (Is it doing this work in a timely and cost-effective way?) (NRC 2008). Investment efficiency requires planning and budgeting to ensure that laboratory resources (both physical and human) are deployed to address the priorities and projected needs of EPA’s program offices and regions at a high level of quality. EPA’s laboratory assets are distributed among many parts of the agency, investment efficiency requires a coordinated agencywide process that can direct resources and activities to where they are needed most. It also requires adaptive adjustments to reflect new scientific information, sources, and methods and changes in agency priorities. The 2008 report concluded that investment efficiency cannot be assessed quantitatively and should be the subject of expert review. Process efficiency focuses on the management of the laboratory enterprise, particularly its ability to produce timely results with little waste. Unlike investment efficiency, it can be measured according to quantitative benchmarks (or standards) of dollars spent or hours worked.
In describing efficiency, the 2008 report distinguished among inputs, outputs, and outcomes. Inputs are resources—funds, facilities, and people—that support EPA’s science-related activities. Outputs are activities or products of science programs. They include not only the products of EPA’s research laboratories—such as published papers, new scientific methods, and enhancement of research capacity—but mon-
itoring of results, compliance tests, enforcement investigations, and other outputs of regional and program laboratories (NRC 2008). Outcomes are the benefits resulting from the science programs. They encompass both intermediate (programmatic) and ultimate (environmental or health) outcomes. The 2008 report concluded that ultimate outcomes—environmental or health end points—do not constitute a useful criterion for evaluating EPA research, because ultimate outcomes usually cannot be reasonably estimated in advance, may occur long after research is completed, and usually depend on actions taken by others. Programmatic benefits, however, constitute a useful and important criterion. In our analysis, programmatic benefits include not only advances in scientific knowledge or comprehensive assessments necessary for agency policy-making but advances in monitoring and testing capability and other outcomes that are necessary for effective implementation and enforcement.
Effectiveness is the ability to achieve useful results (NRC 2008). It focuses on the utility of laboratory enterprise outputs and outcomes to the agency’s science consumers. It captures the importance of the link between the work of EPA’s laboratories and the agency’s mission. The work needs to be not only relevant, of high quality, and cost-effective but of maximum usefulness in meeting the priorities and expected needs of the program offices and regions.
Those general criteria can be elaborated in their application to the diverse functions and processes of EPA’s laboratory enterprise. As discussed in Chapter 1, EPA laboratories serve various functions, from laboratory science and assessments to compliance certifications to field investigations and enforcement. Efficiency and effectiveness are relevant to each function but are measured differently for each. For example, measures of the usefulness of a laboratory study in supporting a major rule-making will differ from measures of its usefulness in sampling groundwater to characterize risks at a contaminated site.
Each laboratory function goes through multiple stages—from planning and budgeting to implementation to production of outputs and outcomes. Efficiency and effectiveness are relevant at each stage but vary in content among them. For example, efficiency concerns in the planning phase (such as alignment with strategic goals and program priorities) differ from those in implementation (such as cost-effective deployment of assigned resources).
Assessing effectiveness and efficiency requires a number of more specific criteria, and these will help to serve as the basis of the analytic framework in the next sections, and further elaborated in Chapter 4. To be effective and efficient, the various laboratories have to provide a “critical mass” of capabilities that include physical facilities and intellectual capital. The physical facilities need to support modern scientific efforts, for example, ultratrace analysis free of background contaminants. Most ORD laboratory facilities have been in operation for many years. They may have a legacy of residual contamination from work conducted when ambient concentrations were high, and they may not be able to support new types of equipment or systems that would permit the scientific staff to pursue studies of importance to the agency. The availability of space that can be readily adapted to a changing set of science priorities would offer the opportunity for research staff to continue to operate at the frontiers of science. However, it is not clear that EPA research must be conducted only in facilities dedicated to EPA activities. Given the panoply of federal and university laboratory facilities that have been developed to perform environmentally relevant research, it might make sense in some instances, for example, for some EPA employees to work in a Department of Energy (or other) laboratory where they could take advantage of unique capabilities. It would be useful to explore a wider array of approaches that take advantage of other research organizations that would permit EPA to perform the science that it requires to make informed decisions and implement science-based regulatory policies (see Chapter 4).
Like ORD laboratory facilities, program and regional office laboratory facilities need state-of-the-art equipment. Regional laboratories and program laboratories have as much need to keep equipment up to current standards as do ORD laboratories. In general, it appears that ORD laboratories have adequate access to equipment to maintain their capabilities. Assessments may be needed to determine equipment requirements for program and regional office laboratories to ensure that their equipment base is adequate to support their activities (see Chapter 3).
A critical issue facing EPA is the state of its scientific personnel. For a laboratory to be effective, it has to have employees with sufficient expertise for major projects that provide policy-relevant infor-
mation or data to support regulatory action. The number of full-time-equivalent laboratory staff has decreased over the last 15–20 years, and many senior scientists are approaching retirement age. EPA staff has been augmented by on-site contractors and postdoctoral associates, but there is a potential for loss of critical institutional knowledge as key staff members retire without hiring people to replace them and the time to transfer the senior staff members’ knowledge to junior personnel. This concern applies throughout the laboratory enterprise, from regional laboratories to ORD research facilities.
FRAMEWORK FOR ALIGNING THE LABORATORY ENTERPRISE WITH STRATEGIC AGENCY GOALS
Overall
The purpose of the analytic framework is to develop principles and guidance to ensure that the laboratories operate efficiently and effectively to advance the agency’s goals. The laboratory enterprise makes up a subset of EPA’s broad array of science-related activities (described in Chapter 1). Its primary function is to serve the program and regional offices by supporting the agency’s strategic goals and overall mission. EPA seeks to ensure that this function is provided through the process shown in Figure 2-1, which includes planning and resource allocation, implementation of laboratory activities, production of outputs and outcomes, and evaluation and assessment for planning and implementation—all in the service of programmatic activities reflecting the strategic goals.
Figure 2-1 depicts the iterative nature of the process. Every 5 years, the agency develops a strategic plan that encompasses the five overall goals (1) of the agency and each program office. Those overall goals drive the activities and goals of the laboratory enterprise (2), which consists of the ORD laboratories and centers, the regional office laboratories, and the program office laboratories. All the laboratories support the strategic goals by implementing a variety of short-term and long-term activities (3) that are selected in consultation with national program office and regional managers. Implementation of the activities is monitored (sometimes by both external and internal reviewers) and adjusted. Implementation yields outputs and short-term or intermediate outcomes (4) delivered to the programs and regions. The products of the laboratory enterprise not only support programmatic decisions and actions but may contribute to long-term outcomes (5), such as improvements in public and ecosystem health, clean air, and increased climate stability.
FIGURE 2-1 The overall process, in generalized form, that connects EPA strategic goals with the laboratory enterprise. See text for explanation of the numbers. Source: Adapted from NRC (2008).
The outputs and outcomes of the laboratory enterprise are reported to senior EPA management, including leaders in the program and regional offices, depending on the connection of the research activities to a given issue, and are used to inform decision-making (7). Assessment of the results informs annual cycles of planning and budgeting for laboratory activities. It also informs the 5-year revisions of the agency’s strategic plan, including updates that reflect new science, new issues, and new priorities (8).
The portion of the process that is of most immediate concern to the committee’s statement of task is represented as a conceptual analytic decision framework, shown in Figure 2-2. The figure focuses on the planning and budgeting, implementation, and assessment of outputs and outcomes for the laboratory enterprise and more particularly for the three main laboratory components of the enterprise. As explained above, the three laboratory components serve the agency’s mission in different ways and, although the criteria of efficiency and effectiveness apply to each, they may be applied differently for each. The three components all serve the agency’s strategic goals, and all need to be measured in the end by their contributions to reaching the goals. Moreover, although they can be broadly distinguished, some functions of the different types of laboratories are overlapping. For example, research by an ORD laboratory may be necessary to develop a method that a program or regional laboratory can use in carrying out a successful enforcement investigation. Some level of information-sharing and coordination within the laboratory enterprise is essential for its overall success.
We define the analytic framework further below for each of the three laboratory components and distinguish where the processes differ. Recommendations for optimizing laboratory performance within the framework are presented in later chapters.
FIGURE 2-2 The portion of the overall process shown in Figure 2-1 that focuses on the role of the laboratory enterprise. Three major steps are shown: planning and budgeting, implementation, and assessment of outputs and outcomes. Those apply to all three of the components of the laboratory enterprise. The dashed red arrows indicate internal communication among laboratory components, and the dotted green arrows indicate where the laboratories would benefit from external communication with other scientific entities. Both kinds of communication and coordination need to be enhanced (see Chapter 4). Other communication directions are not intended to be prohibited. Note: PO = program office; RO = regional office.
Framework for Office of Research and Development Laboratories
The application of the framework to ORD is expanded and depicted in Figure 2-3, which includes the three steps—planning and budgeting, implementation, and evaluation of outputs and outcomes. Specific criteria are included at appropriate places in the framework to address and maximize efficiency and effectiveness (see Chapter 4). The ORD element of the laboratory enterprise is perhaps the most complicated but has the most advanced planning process. In 2010, ORD began the process of realigning its research efforts with the concepts of sustainability. That effort involved the reduction of 13 research programs in ORD into six more integrated national programs (Anastas 2012).
An ORD 4-year strategic research plan is developed for each national research programs developed collaboratively to link agency goals to ORD goals and objectives. That is necessary because agency goals are not specific enough to inform or direct the workplans of the ORD national research programs. The plans are developed by ORD’s national program directors (NPDs) in consultation with ORD senior management.1 NPDs consult with the laboratory directors, but the issues driving the strategic plan come mostly from the NPDs in consultation with the program offices and regions. Annual project planning uses a matrix management approach whereby the NPDs and laboratory directors collectively determine priorities and how work will be done. Annual congressional appropriation funds received by ORD laboratories and programs contribute to determining the planning process. The matrix planning paradigm has been successfully implemented in recent years, but the overall process could be enhanced if the criteria for assessing efficiency and effectiveness were addressed systematically as discussed in Chapter 4.
The process for developing the budgets (for the executive branch’s budget request to Congress) is also collaborative, but inasmuch as it is done for future fiscal years, budgeting is not coupled directly to annual operational planning, which focuses on the current fiscal year.
FIGURE 2-3 Aligning agency strategic goals to the ORD portion of the laboratory enterprise. Note: NPD = national program director; LD = laboratory director; PO = program office; RO = regional office; BOSC = Board of Scientific Councilors.
_________________________
1Each of ORD’s six national research programs is led by an NPD. The NPDs are responsible for ensuring that the science conducted is relevant and of high quality. (EPA 2014c)
Once research goals are established, ORD implements programs through a variety of short-term and long-term research activities. The activities result in short-term advances and outcomes and eventually in longer-term environmental benefits. Ideally, effectiveness and efficiency can be optimized by considering the specific criteria discussed earlier in this chapter and addressed in Chapter 4. Short-term outputs of research can be evaluated for efficiency (for example, Was it done on time, on budget? Did the research unknowingly duplicate other scientific efforts?) and effectiveness (for example, Are results of sufficient quality to be useful? Does the research address the problem but lack suitability for implementation?) and then be reported to the NPDs and the senior management of the program offices depending on the connection of the research activities to given program office issues. The outputs and outcomes are shared with the program offices and the senior administration at headquarters to inform decision-making over the next annual cycle. In 4-year time horizons, the strategic plans are altered to reflect new science, new issues, and new priorities.
Regional Office Laboratories
The application of the framework for assessing regional office laboratories is different from that for assessing ORD laboratories owing to their mission of applying science to regulatory enforcement. The approach is outlined in Figure 2-4. Regional office laboratories support primarily short-term objectives related to specific enforcement actions in a region or projects developed specifically to meet the needs of the region as designated by the regional administrator. They are generally not involved in exploratory science and are typically restricted to using narrowly-defined procedures and strict quality-assurance and quality-control practices to ensure the reliability and acceptability of delivered data. The current structure provides flexibility and support for a regional administrator to undertake projects that are seen as important to the region.
However, this approach for regional office laboratories may not provide maximally efficient and effective laboratory capabilities in that some of the facilities will be relatively small in terms of staff and equipment. For recommendations that address this, see Chapters 3 and 4.
FIGURE 2-4 Aligning agency strategic goals with the regional office laboratories. Note: RA = regional administrator; RO = regional office.
Program Office Laboratories
Program offices maintain laboratories that directly support regulatory implementation, compliance, and enforcement efforts at the national level. There are eight such laboratories (two of which are referred to as centers). For example, they include a national enforcement and investigations laboratory in the Office of Enforcement and Compliance Assurance; two laboratories in the Office of Air and Radiation, of which one is devoted to testing of vehicle emissions, fuel economy, and transportation fuels and the other to assessing radiation risks; and laboratories in the Office of Chemical Safety and Pollution Prevention that support the agency’s pesticide registration and enforcement program (see Chapter 1).
An analytic framework for the program laboratories appears in Figure 2-5. Unlike the ORD and regional laboratories, these laboratories are under the direct control of the managers of the national program offices responsible for implementing and enforcing EPA’s regulatory programs. They follow the same sequence of planning, implementation, assessment, and feedback as EPA’s other laboratories, but the work of each program office laboratory is focused on the needs of one program client, and they are managed as part of that client. Integration of the program laboratories in the management processes of their program client will likely facilitate alignment with program needs and strategic goals. Although location of these laboratories in the program offices may simplify some management issues, there is a need to enhance information-sharing and coordinating with other components of the laboratory enterprise (see Chapter 4).
FIGURE 2-5 Aligning agency strategic goals with the program office laboratories. Note: PO = program office.
