Attributes of a First-in-Class Environmental Program: A Letter Report Prepared for the Bureau of Ocean Energy Management (2022)

Division on Earth and Life Studies
Ocean Studies Board

December 14, 2021

William Brown
Chief Environmental Officer
Bureau of Ocean Energy Management
45600 Woodland Road
Sterling, VA 20166

Dear Dr. Brown,

The Bureau of Ocean Energy Management’s (BOEM’s) Office of Environmental Programs requested that an ad hoc committee of the National Academies of Sciences, Engineering, and Medicine organize workshops and produce a letter report on the topic of BOEM’s aspirations toward a first-in-class environmental program. The report containing the committee’s guidance and advice is attached here.

BOEM’s environmental program supports the agency’s mandate to oversee development of energy and mineral resources of the U.S. Outer Continental Shelf (OCS) by ensuring that environmental protection is a critical element of BOEM’s decision making. Environmental studies supported by BOEM inform its environmental assessments, and the development of both fall under the environmental program. BOEM’s decisions in turn affect external entities including partner agencies and tribes, the energy industry, other industries operating in the OCS, and Indigenous and other coastal communities. These elements define a complex system that drives the needs, the outputs, and the impacts of the science produced by the organization.

The report describes attributes identified by the committee of a first-in-class, use-inspired, management-oriented science program (in this case BOEM’s Environmental Studies Program), including its relationship to management activities (BOEM’s environmental assessments and agency decision making). The committee provides a framework within which BOEM can evaluate whether its program meets the attributes of a first-in-class program, and from there, determine where improvements are needed. This framework takes a systems approach to program evaluation that looks across a program in all its operational phases, the relationships among these phases, and the relationship to the broader organization or external entities. In this report, these phases are the processes of a program, its

outputs, and the resulting impacts, as well as a strategic focus on innovation that underlies all phases. The attributes of a first-in-class program developed by the committee, organized by organizational components, are summarized in the table included with this letter.

The committee recommends that BOEM develop procedures and conduct regular evaluations to assess whether and how well its environmental program meets the attributes of a first-in-class program and to identify areas for improvement. The report contains guidance for conducting an evaluation, drawn from a workshop series and other information-gathering efforts.

The information in this report is drawn from foundational literature related to program evaluation, lessons drawn from other science programs, and further insights into first-in-class qualities from the U.S. Department of the Interior’s Office of Planning and Performance Management and the National Institute of Standards and Technology’s Baldrige Performance Excellence Program. The guidance contained in this report is offered to BOEM as a starting point for developing more detailed processes for evaluating and improving its program.

Sincerely,

TABLE 1 Attributes of a First-in-Class Program, Including the Positive Outcomes to a Program Expected to Be Achieved by Meeting These Attributes

INTRODUCTION

The Bureau of Ocean Energy Management (BOEM) is charged with managing the energy and mineral resources on the U.S. Outer Continental Shelf (OCS). Effective management that maximizes utility of the resources while minimizing environmental harms depends on relevant and defensible science. In pursuit of its interest in conducting an environmental program that is “first-in-class,” BOEM engaged a committee of the National Academies of Sciences, Engineering, and Medicine to organize workshops and write a short report addressing this aspiration (see Appendix A for the complete Statement of Task). The task required the committee to determine its specific approach to the workshops and report, described below.

The Goal of First-in-Class Status

BOEM’s Environmental Studies Program (ESP)—one component of BOEM’s broader environmental program—is guided by the following vision statement (BOEM, 2020a):

“BOEM’s long-term vision is for the ESP to be ‘first in class’—the best research program possible in the context of BOEM’s mission and constraints.”

The committee used this vision as a starting point for what first-in-class status would mean for BOEM. Based on this vision, being first-in-class is dependent on the quality of the science produced, with specific attention to the role that science has on achievement of BOEM’s particular mandate.

The committee did not take first-in-class to mean that BOEM’s program could or should be directly compared to other agency programs. BOEM and other agencies are each meeting unique missions and working within their own constraints. BOEM’s “class” can be defined in a number of ways; in one sense, BOEM is in a class of its own because no other agency is charged with conducting a science program for the same purposes as BOEM. From this perspective, first-in-class status is an aspiration for an agency or individual agency program to be the best it can be, in this case in its practices and continuous improvement toward pursuit of the production of highest-quality, informative, and influential scientific knowledge.

For the purposes of exploring practices that could contribute to BOEM’s first-in-class aspirations, the committee considered the “class” of peer agencies as being those that produce—whether intramurally or extramurally—managementoriented (or use-inspired), applied science that is designed to inform an agency’s environmental decision-making process. Agencies with similar science programs include the National Oceanic and Atmospheric Administration (NOAA), the U.S. Geological Survey (USGS, which produces science for other U.S. Department of

the Interior [DOI] bureaus), the U.S. Environmental Protection Agency (EPA), the U.S. Army Corps of Engineers (USACE), and the U.S. Department of Defense. Each agency may have one or more science programs, some of which were represented at the committee’s workshop, described below.

Committee’s Approach

This report contains guidance developed by the committee in support of BOEM’s interest in achieving first-in-class status for its environmental program. The committee focused on the ESP, as specified in the Statement of Task, in order to give emphasis to developing quality science. However, the priorities, outcomes, and impact of the ESP are driven by needs of the broader environmental program, particularly the environmental assessments. In this way, both of these aspects of the environmental program were considered as part of the committee’s approach to its task.

The committee relied on several sources of information in producing the guidance and recommendations found in this report, described here.

• Consultation with BOEM’s Office of Environmental Programs (OEP). The committee heard a presentation by representatives from BOEM’s OEP, the headquarters organizational unit responsible for the environmental program, during a virtual meeting early in the study. The presentation described the structure and processes used by the Division of Environmental Sciences (which administers the ESP), the Division of Environmental Assessment, and the Center for Marine Acoustics (all components of the OEP) to guide the development and use of BOEM’s funded science. Notably, the representatives emphasized the importance of having ways to measure whether or not their program was first-in-class. The OEP also participated in the workshop series described below.
• Consultation with DOI’s Office of Planning and Performance Management (OPPM). As a DOI agency, BOEM is guided by the procedures set by DOI to meet the requirements of the Government Performance and Results Act (GPRA) of 1993, the GPRA Modernization Act of 2010, and the Foundations for Evidence-Based Policymaking Act of 2018 (for which plans for implementation are still in progress). The committee heard a presentation from DOI’s OPPM in order to learn more about the context, procedures, and requirements BOEM is subject to regarding performance management for DOI.
• Consultation with the Baldrige Performance Excellence Program. The Baldrige Performance Excellence Program is a program of the National Institute of Standards and Technology. One of its main products is the Baldrige Excellence Framework, which provides a guide for purchase by any organization to self-evaluate its performance based on

recommended performance excellence criteria. The criteria are a set of questions that provide a systematic perspective across an organization (meaning an organization is represented by individual, but related, components) that looks at both processes and results and allows an organization to identify areas of strength and those needing improvement.

• Examples from other science programs. The committee organized a virtual workshop series for invited representatives from other science programs to share practices related to how they plan and evaluate their program design, their products, and their impact. Presenters in the “peer agency” category, meaning they were in BOEM’s class, represented science programs within federal agencies that produced science for environmental decision making. The workshop series also included informative cases from non-federal peers (tribal government, international cases, and industry) and non-peer agencies (i.e., those focusing on “discovery” science). The agenda for the workshop series, which lists participating programs, can be found in Appendix C. Additionally, this report contains examples from programs that were not discussed at the workshop but which the committee considers to be informative to BOEM.
• Select relevant literature. A systems approach to evaluation is well established in literature and is a common approach used by many organizations. The report draws from some key sources of information on a systems approach as well as its subcomponents related to process, outputs, impacts, and innovation.

This report presents a framework that BOEM can use to evaluate the ESP and its connection to the broader environmental program. The details of the framework are described with BOEM’s mission and constraints in mind but would be applicable to a large degree to similar management-oriented, use-inspired programs. The framework is not prescriptive in regard to the procedures, evaluation methods, or measures of success BOEM would use within the framework, which are more appropriate for BOEM itself to address. The committee focused on lessons drawn from other programs (including peer agencies). The examples from other agencies show that there is no single correct way to conduct an environmental studies program. Furthermore, to conduct the actual evaluation of BOEM’s environmental program, particularly whether or not it can be considered first-in-class, is outside the scope of this study. Thus, the guidance contained in this report is a starting point upon which BOEM can build more comprehensive program and evaluation processes, including ways of measuring success.

BOEM’S OPERATIONAL CONTEXT

BOEM is responsible for managing energy and mineral leasing and development on the OCS of the United States, an area of nearly 2.5 billion acres—comparable in size to the land area of the United States (see Figure 1). Its three focus areas are the development of oil and gas, renewable energy (from wind, waves, and currents), and marine minerals (sand, gravel, and other minerals). BOEM manages this huge geographic range and diverse resource portfolio with a staff of under 600 in a headquarters office and 3 regional offices (in California, Alaska, and Louisiana) and a budget of $191 million in FY2021. In FY2019, BOEM provided $5.5 billion in oil and gas royalty and lease bonus revenues to the U.S. Department of the Treasury. Over recent years, BOEM’s focus has been shifting from its historical emphasis on oil and gas to rapidly expanding the renewable energy program and meeting new demand for marine minerals. Significant contributions of electric power are expected from offshore wind leases in the Atlantic OCS in the coming decade, and pilot projects are under way to evaluate wave energy resources in the Pacific. Future areas of focus will include exploration of offshore critical minerals (BOEM, 2020b) and may include offshore carbon sequestration (Vidas et al., 2012).

BOEM’s environmental program informs all of BOEM’s activities related to energy and mineral development on the OCS. The environmental program is responsible for achieving BOEM’s environmental mission by supporting BOEM decision making and compliance under regulatory and statutory requirements (including the National Environmental Policy Act [NEPA], the Endangered Species Act, the Marine Mammal Protection Act, and others). The goal is not

just to have science be informative to agency decisions, but for environmental protection to be a “foremost concern and an indispensable requirement” in decision making (BOEM, 2020a). In this way, the environmental program is meant to be not just informative but influential. The program is managed in the national office by the OEP, which coordinates with regional offices (see Figure 2 for the OEP organizational structure). The chief environmental officer leads OEP and reports to the BOEM director and deputy director. OEP includes the Division of Environmental Sciences, the Division of Environmental Assessment, and the Center for Marine Acoustics. While the Division of Environmental Sciences has lead responsibility for the ESP, coordination with the Division of Environmental Assessment is necessary and expected. Program funding for the ESP has averaged $30 million per year in the past decade; since its inception in 1973, more than $1 billion has been invested in the ESP by BOEM and its predecessor agencies. To undertake its environmental studies, BOEM often partners with other federal agencies and nonfederal organizations to leverage resources and supplement BOEM’s internal capabilities. For example, during the period of FY2017–2021, 50% of procurement funding was for interagency agreements.

Section 1346 of the Outer Continental Shelf Lands Act (OCSLA) details research mandates for the ESP, which is to conduct studies “needed for assessment and management of environmental impacts on the human, marine, and coastal environments” within areas or regions leased under BOEM’s jurisdiction. Accordingly, studies to inform BOEM assessments and management decisions, both short and long term, are the key priority. This same part of OCSLA authorizes and establishes three goals for the ESP: baseline studies, impact studies, and monitoring studies. BOEM also considers broader, longer-term studies critical to addressing the scientific questions necessary to assess and manage potential harms to the environment of the OCS from energy and minerals development (BOEM, 2020a). BOEM shared with the committee three long-term environmental goals for its OEP program that are intended to help “drive BOEM’s environmental program to be first-in-class among federal peer agencies,” which can be found reflected throughout the report.

1. Protecting Ecosystems in the Context of Climate Change: Provide robust research, assessment, regulatory measures, products, and services to avoid and mitigate harm to ecosystems from OCS development; consult and coordinate with stakeholders, and assess impacts from climate change combined with steps to redress adverse effects.
2. Excelling in Consultation and Collaboration with Tribes and Alaska Native Claims Settlement Act Corporations: Substantially advance effectiveness of engagement.
3. Advancing Environmental Justice: Advance environmental justice groups historically disadvantaged by injustice, inequality, or exclusion, including initiatives advancing employment and economic enterprises.

A robust capability for program evaluation allows BOEM to assess whether its statutory requirements and strategic goals are being met and identify areas for improvement. This is particularly critical as programmatic priorities and strategic goals evolve. BOEM contributed to the achievement of the DOI strategic goals found in DOI’s 2020–2021 Annual Performance Plan and 2019 Report related to energy and economic security, access to mineral resources, and revenue production (DOI, 2019). BOEM contributes to these goals by meeting time-based performance measure targets through the activities of all program areas, including the environmental program, to ensure that these activities are implemented in a manner protective of the environment. However, BOEM will also have bureau-specific performance measures and broad strategic goals to meet; while some may be outside the scope of DOI performance management, DOI procedures may still provide tools that BOEM can use to structure evaluation and engage in continual learning.

The ESP is guided by a Strategic Framework (BOEM, 2020a) that considers three questions: (1) What does BOEM need to know? (2) What strategic questions should be posed? (3) What criteria should be used to prioritize studies for addressing these questions? The Strategic Framework also identifies 10 strategic science questions that BOEM seeks to address in its studies (e.g., how to assess cumulative impacts, understanding effects of sound on marine life, and understanding the effects of hydrocarbon exposure on marine life). The ESP Strategic Framework discusses the role of BOEM science and technical peer review of the annual study plan, including the use of science and technical review teams of BOEM subject-matter experts, informal feedback by members of the National Academies Standing Committee on Offshore Science and Assessment (COSA) whose meetings also serve as a forum for feedback by nonmembers, and by BOEM senior management. BOEM’s goal is to revise this strategic framework every 5 years, including an assessment of the effectiveness of its studies in meeting BOEM’s needs.

Some methods currently used by BOEM to measure the effectiveness of its environmental program include surveys and other forms of invited feedback from staff on the studies development plan process; use of the ESP Program Assessment Tool for staff to evaluate whether studies met their objectives, have a use, were on time and in budget, and were peer reviewed; and use of the Environmental Studies Program Information System (ESPIS) for tracking and sharing study outputs. The OEP is currently undertaking a study to understand how ESP-funded research contributes to BOEM’s assessments and vice versa (Industrial Economics, Inc., 2020). This Evaluating Connections study takes a close look at the current status of the connections between these programs. In contrast, this report provides higher guidance for how such connections are essential for a first-in-class program.

A FRAMEWORK FOR EVALUATION AND IMPROVEMENT

The committee’s task relates to approaches for program evaluation or program performance assessment with the goal of informing whether a program’s performance can be considered first-in-class. The U.S. Government Accountability Office (GAO) defines program evaluation as periodic study of how well program objectives are being achieved (GAO, 2011). It is related to—but distinguished from—performance measurement, which is the ongoing monitoring and reporting of progress toward established goals based on measurable performance standards. Both are meant to assess program effectiveness, but an evaluation will examine a broader range of information in addition to that which is captured by performance measures. Regardless of the terminology an agency uses, this makes the point that an evaluation is an opportunity to look beyond what is normally captured by ongoing monitoring, such as connections between a program and its societal or policy context. In this report, the committee outlines a framework for conducting a program evaluation.

The committee adopted the systems model for program evaluation developed by Rossi et al. (1999, 2018) and Worthen et al. (1997) that looks across a program in all its operational phases, the relationships among these phases, and the relationship to the broader organization or external entities. GAO (2011) describes a similar model that distinguishes between four kinds of program evaluation: process evaluations, outcome or output evaluations, impact evaluations, and cost–benefit analyses. This framework has been adopted by the Centers for Disease Control and Prevention (CDC, 2011) and has been used to evaluate medical delivery programs, educational programs, and distance learning programs (Rossi et al., 2018; Rovai, 2003; Worthen et al., 1997). The systems approach can also be used to represent programs that fund or undertake scientific studies (e.g., Bartlett, 2016; Morton, 2015; Samuel and Derrick, 2015).

The committee started with an initial conceptual model based on the literature described above on systematic evaluation that included the Process, Outputs, and Impact phases of a program. Workshop series speakers were asked to describe their organization’s program design and evaluation activities in each area. Several workshop presentations highlighted the importance of innovation as part of a use-inspired program throughout the entire program process. The committee agreed that innovation was essential to first-in-class aspirations and was important to consider as its own component of the conceptual model, particularly because it requires its own strategic processes. Evaluation occurs within and across all components of the model, and it is based on the results of this evaluation that improvements can be strategized. The committee’s conceptual model is described next and depicted in Figure 3.

• The Process phase of the framework focuses on how the studies program identifies, prioritizes, and funds science needs and monitors studies while they are in progress.

• The Outputs phase of the framework focuses on study reports, findings, and how those findings are communicated within BOEM and to other communities.
• The Impact phase of the framework focuses on whether individual studies or the overall program make a difference.
• Innovation is a feature throughout all of the phases of a studies program.

A scientific studies program that aspires to be recognized as first-in-class will make a commitment to quality and quality improvement that encompass every

phase of funding, conducting, and completing a study (Dale, 2003; Takeuchi and Quelch, 1983). Programs recognized as high performing have been described in the literature as having three essential features: (1) they will produce studies that are consistently noteworthy (salient) because of their rigor, methods, analyses, and findings/inferences; (2) they will produce studies that are relevant to its target audience and a broader audience; and (3) they will communicate actively, consistently, and effectively about their studies—their sample designs, analytical methods, study findings, and the relevance of those findings—to the larger scientific and regulatory communities and the public (Coglianese, 2012; EPA, 2011; Ferguson et al., 2014; Makkar et al., 2016). These are all components of the Outputs phase of the committee’s conceptual framework.

The three features described in the preceding paragraph can be used to evaluate a studies program in relation to its peers and the nature of its potential impact. The Process phase of the conceptual model is an essential precursor to program Outputs. Similarly, a studies program that aspires to be recognized as first-in-class will commit to consistently improving the quality of program outputs, their relevance, and the effectiveness of how those outputs are communicated. This will appear as a commitment to innovation and program improvement.

For each component of the conceptual model, the committee identified sets of attributes that define a first-in-class program, as well as a set of cross-cutting attributes that underlie all components of the system represented by the model. The attributes are based on insights gained through committee experience and consultation with others during the workshop series. There are multiple ways that an individual program may choose to embody an attribute, so the committee provides examples of how various science programs have done so.

For each attribute, the committee suggested a set of sample evaluation questions that can guide the evaluation of whether or not a program embodies that attribute. Most of the questions in the report are open ended to encourage BOEM to consider not only whether a certain action is being taken or not, but to think about what success would look like. Many of the questions may lend themselves well to the development of metrics.

The combination of the conceptual model, attributes, and evaluation questions are designed to provide BOEM with a framework it can use to evaluate its environmental program and determine whether the program warrants being considered first-in-class. The results of an evaluation will allow BOEM to identify in what areas it can improve the program until it fulfills BOEM’s aspiration. Regular evaluations provide a means for assessing the success of improvement efforts and progress toward first-in-class aspirations.

ATTRIBUTES OF A FIRST-IN-CLASS PROGRAM

Cross-Cutting Attributes

Cross-cutting attributes are those that are important to most, if not all, phases of the conceptual model. They capture an organizational commitment and intentionality to lay the foundation necessary to excel. Without that foundation, a program is not likely to push itself to improve or have the staying power necessary to be recognized as first-in-class.

Cross-Cutting Attribute 1: Articulates Sphere of Influence—Understands and articulates its place in, and necessity to, the overall national policy context, and what policies or decisions it intends to shape or influence.

Description: A federal agency’s ability to understand its responsibility in executing the national policy agenda in the areas touching on its mission creates an internal guiding purpose for the organization, facilitates its ability to define the policies it intends to shape or influence, provides an understanding of the selection of its potential research partners, and helps identify collaboration opportunities. It is the step from which all else follows, including determining what scientific research the agency will undertake, alone or in concert with partners; identifying the key drivers for its scientific research programs; deciding how much of a role innovation (ranging from small to revolutionary) will play, and where, within its programs; assessing the organization’s constraints and potential relative to the outcomes it seeks to achieve; and gauging what effect (impact) its scientific research is having. Without this and a concomitant ability to articulate and operationalize its unique responsibility in and necessity to fulfilling the national policy agenda, an organization risks both internal disorganization stemming from a lack of clear guidance as to its needed contribution to national purpose, and external undervaluation stemming from questions as to the organization’s overall relevance to its responsibilities. Thus, BOEM would determine how to execute the mission of its studies program—to provide information to assess and manage impacts to human, marine, and coastal environments from OCS energy and mineral development—in a manner that best contributes to achieving policy objectives related to energy, climate, and conservation set forth at the national level.

Example: USACE strives to identify early on and continuously throughout its research and development (R&D) process the nature and extent of its intended impact on the national policy agenda. Each R&D cycle is initiated via a campaign plan¹ based on guidance from the organization’s senior leadership and Congress.

___________________

¹ See https://www.usace.army.mil/About/Campaign-Plan.

The development of an accompanying R&D strategy focuses on answering the question of how the organization will have the greatest impact with the resources available.

Sample Evaluation Question:

• To what extent are the stated drivers for the scientific research derived from a clear articulation of the agency’s or program’s role in the national policy agenda?

Cross-Cutting Attribute 2: Disseminates Policy Priorities Internally—Articulates policy priorities as guidance disseminated within the organization to help shape and focus the development of the organization’s scientific research agenda.

Description: Once an organization understands and effectively communicates its role in the national agenda (Cross-Cutting Attribute 1), the next step is for leadership to formally disseminate the program’s priority policy areas and associated top-level goals and objectives in ways that provide clear guidance and focus to those (such as management, staff, and others) participating in the development of the organization’s research agenda. Importantly, priority policy areas that direct research objectives tend to be most effective when they also incorporate a diversity of inputs. Policy guidance can also help define what areas the organization is choosing to emphasize or de-emphasize in any given fiscal year and allow for dialing those areas up or down on the research agenda as new scientific questions and challenges emerge while others have been resolved or are no longer deemed relevant. The process of aligning the organization’s research agenda with its policy priorities can help introduce purpose and clarity into the idea development phase of the organization’s scientific research agenda and thereby also influence the outputs of and impacts from that agenda. In addition, characterizing the time frame for the desired research response to those priorities as short, mid, and long term can enable distinguishing between rapid response versus current and over-the-horizon requirements.

Examples: USGS uses both top-down and bottom-up processes to develop its research agenda by melding guidance from congressional appropriations and overarching direction from DOI and the Office of Management and Budget (OMB) with inputs from scientists within USGS, partner organizations, and external stakeholders. For this process to work effectively, USGS headquarters develops an annual guidance document for a particular mission area to drive the message of key DOI priorities to the science centers and field programs, including an expression of both mission and program goals. This includes potential adjustments to previous focus areas, including areas to increase, reduce, or maintain (from workshop presentation by Mona Khalil, June 22, 2021).

Sample Evaluation Questions:

• How often does leadership generate priority policy areas for its management, staff, partner organizations, and external audiences?
• To what extent does the organization’s research agenda, once developed, reflect the priorities established by its own priority policy area guidance?
• What is the role of diverse users, stakeholders, rights holders, and affected communities internal and external to the agency in providing input on agency priority policy areas, and thus inputs to the research agenda itself?
• What proportion of the research agenda is influenced by policy inputs from these diverse groups?
• How is guidance on priority policy areas used to help identify and look ahead to future challenges that may not currently be at the forefront of the research agenda?

Cross-Cutting Attribute 3: Engages in Systems Thinking and Analysis—Engages in systems thinking and analysis internally and externally to help achieve organizational excellence.

Description: Systems thinking is a way to view, understand, and manage the interdependent components of an organization, including its interface with external organizations within their sphere, to optimize the organizational system (Rossi et al., 1999, 2018; Senge, 2014; Worthen et al., 1997). Systems analysis applies analytical approaches to achieve that optimization by recognizing and adjusting how subsystems contribute to functioning of the larger system (NRC, 1998). Both approaches assume that organizational effectiveness influences the quality of organizational outcomes and allows an organization to draw data-based optimal connections between each phase of its internal activity. In this report those components are Process, Outputs, and Impact, with Innovation featuring in all three phases.

A systems approach distinguishes between different organizational units of a program (such as BOEM’s headquarters and its regions) and its interactions with external entities (such as DOI, peer agencies, and other collaborators) and may also focus on individual phases or subsystems of an organization’s activity. By understanding interdependencies between and within its subsystems, an organization can continually learn and adapt to improve those connections based on observations of how they function. By focusing resources on different components or subsystems in any given review cycle, the organization strengthens its system as a whole and creates a capability for the organization’s leadership and management to evaluate potential trade-offs among system elements and resources in ways that can result in new efficiencies within each subsystem and for the system as a whole.

Understanding and improving an organization’s interdependencies require gathering and learning from relevant data. Data used for decisions include baseline information in order to measure changes and relations over time. As data-based understanding improves, learning (1) becomes “a part of everyday work; (2) results in solving problems at their source; (3) is focused on building and sharing knowledge throughout your organization; and (4) is driven by opportunities to bring about significant, meaningful change and to innovate.” (Notes on Criterion 4.2b(3) from the Baldrige Excellence Framework.) Properly executed, these approaches help the organization identify operational components that need to be adjusted to improve effectiveness and achieve excellence.

Example: The Focused Areas of Regulatory Science (FARS) of the U.S. Food and Drug Administration (FDA) Office of Regulatory Science and Innovation offers an example of an adaptive approach applied in the Process phase of its program. These focus areas represent a shift from the development of structured program priorities to a more agile and dynamic approach to identify areas of investment. The FARS “format is designed to be easy to update to accommodate frequent updates and revisions to align with the rapid pace of scientific advancement as well as evolving priorities and research activities” (workshop presentation by Tina Morrison, June 22, 2021).

Sample Evaluation Questions:

• How is the systems approach defined, structured, and used to understand and adapt your organization’s operations in ways that make it more impactful, efficient, or both?
• How does leadership ensure ongoing communication and improvement within and between the subsystems that constitute the organization?
• How is systems analysis applied to examining the program (i.e., the Process, Outputs, and Impact phases described in this report)?

Cross-Cutting Attribute 4: Partners and Collaborates Effectively—Identifies and engages in scientific partnerships and collaborations with other organizations.

Description: Partnering with other entities is desirable when it enables partners to leverage each other’s resources and address together those areas of critical need that no one partner can address on its own, provided opportunity costs are understood. To avoid draining financial, staff, and in-kind resources when partnering, managers develop a system for identifying, responding to, and selecting potential partners and identify potential costs and benefits before initiating the partnership. Advance written agreements establish partners’ contributions, roles, responsibilities, deadlines, and desired outcomes, as well as regular review of the partnership and the agreement itself. The Subcommittee on Ocean Science and

Technology (SOST) is the lead interagency body for coordination on ocean science and technology and serves as a venue for coordinated strategy development that can ensure an appropriate fit when cooperative partnerships are developed. The National Oceanographic Partnership Program facilitates resource leveraging within and beyond the federal government.

Collaborators are individuals and groups who work directly with studies programs to identify the studies that are needed (i.e., “co-production”) and who may cooperate on the design and execution of those studies (i.e., “participatory research”) (e.g., Ferguson et al., 2014; Jacobs, 2002). Collaborators may include federal, state, local, and tribal partners as described earlier but may also include other agencies; independent scientists and other subject-matter experts; or stakeholders, rights holders, and members of affected communities. A collaborative process for conducting research allows the study to benefit from a broader range of skill sets and knowledge bases while also engendering trust in study outcomes (e.g., Hahn et al., 2006; Henry and Dietz, 2011). Approaches to working with collaborators include joint fact-finding (Susskind et al., 2016, see the example of Rofougaran and Karl, 2005) and the use of decision frameworks that incorporate stakeholder values (e.g., multicriteria decision analysis, Kiker et al., 2009).

Examples: The Canadian Fisheries Research Network (CFRN)² was a partnership and collaboration between government agencies, the fishing industry, and academics (including students) to address research issues pertinent to sustainable fishing and ecosystem-based management (Thompson et al., 2019; presentation by Robert Stephenson, June 22, 2021). The partnership leveraged both the financial and the knowledge-based resources of all participants, and it would not have been possible for the Canadian government to fund the research alone. Partners were also collaborators in the research and were engaged in all parts of the projects, from crafting research proposals to conducting research, to using results. The involvement of the fishing industry in the co-construction of the research plan validated the merit and usefulness to management of the research questions. Co-production during the research studies allowed for activities each partner could not have conducted on their own, including management strategy evaluations; methods for adding social, economic, and institutional aspects; and information and sampling from industry. Added benefits included training a new cohort of researchers for future interdisciplinary work and developing best practices for collaboration and establishing trust.

The SOST Interagency Task Force on Ocean Noise and Marine Life³ is a partnership between the U.S. Navy, BOEM, NOAA, and the Marine Mammal Commission to support research projects of shared interest. Three projects were

___________________

² See http://cfrn-rcrp.ca/CFRN-RCRP.

³ See https://www.navfac.navy.mil/navfac_worldwide/specialty_centers/exwc/products_and_services/ev/lmr/sost.html.

jointly funded under the auspices of the task on the auditory capabilities of mysticete whales. Each project approached the issue from a different perspective and reflected the interests of each partner agency, while still being coordinated to achieve a broader research objective.

Sample Evaluation Questions:

• What criteria are used to decide how and when to solicit or respond to partnership opportunities, or is the process ad hoc?
• How are the costs and benefits to the program in financial, in-kind, and human resources considered when assessing potential partnerships?
• What procedures are in place to foster collaborations (e.g., with scientists, stakeholders, rights holders, and members of affected communities) to benefit from their skills and knowledge, and build trust in study methods and findings?
• How often are partnership and collaboration agreements/arrangements, including outputs, reviewed and provided with written reasons for extending or terminating these partnerships/collaborations?

Cross-Cutting Attribute 5: Fosters Diversity, Equity, and Inclusion—Utilizes an integrative approach to foster an environment of deep diversity, in which a diverse group of perspectives is offered equal participation through an equitable process.

Description: Inclusion of diverse communities inside (e.g., diverse scientists) and outside (e.g., diverse collaborators) a program improves all phases of scientific studies (Hong and Page, 2004; Medin and Lee, 2012). Science achieves objectivity through the collective effort of individuals working within a community of scientists. Individual scientists communicate their ideas and findings to other members of their scientific community where those findings are critically reviewed and are either supported, modified, disputed, or rejected (Oreskes, 2019). This evaluation process works best when the scientific community includes individuals with different points of view, background knowledge, training, and cultural perspectives because diverse communities are more likely to detect potential errors in reasoning and analysis than communities that share common perspectives (Horwitz and Horwitz, 2007; Oreskes, 2019; Phillips and O’Reilly, 1998; Zhan et al., 2015). This will be particularly true in scientific communities that include individuals with different traditions for acquiring knowledge and for answering questions about the natural world. Diversity also provides a foundation for the kind of innovation that is characteristic of first-in-class studies programs. Specifically, the type of diversity that drives excellence in organizational performance is not diversity based on characteristics such as age, gender, and race/ethnicity (sometimes called “surface-level,” “demographic,” “ethnic,” or “bio-demographic” diversity), which may even have a negative effect

on performance and innovation (Phillips and O’Reilly, 1998; Zhan et al., 2015). Rather, diversity in attitudes, beliefs, and values (sometimes called “deep-level,” “cultural,” “attitudinal,” or “task-related” diversity) substantially improves performance and innovation. These two kinds of diversity can be related, but the former does not ensure the latter.

A report by McKinsey & Company (2020) highlights five areas of action to foster diversity, equity, and inclusion: (1) ensure the representation of diverse talent, (2) strengthen leadership accountability and capabilities for inclusion and diversity, (3) enable equality of opportunity through fairness and transparency (including recognizing and reducing biases in the processes used to help ensure that the opportunities are not only equal but also equitable), (4) promote openness and tackle microaggressions, and (5) foster belonging through unequivocal support for diversity in attitudes, beliefs, and values (i.e., “deep diversity”).

An organization that has fully integrated diversity, equity, and inclusion will strive to ensure that its internal processes and external outreach avoid the unwitting introduction of barriers that would preclude equitable and meaningful participation. DOI’s Environmental Justice Vision is “to provide outstanding management of the natural and cultural resources entrusted to us in a manner that is sustainable, equitable, accessible, and inclusive of all populations” (DOI, 2021). The Office of Environmental Policy and Compliance oversees and coordinates DOI’s environmental justice initiatives and implementation. Proactive integration of these resources throughout the research process supports successful achievement of DOI’s environmental justice goals, but more importantly, it helps to support the integrity of the study process and products and facilitates syntheses. Such an organization will also measure how its diversity, equity, inclusion, and environmental justice practices affect the impact of its science program. It will not consider achieving environmental justice outcomes to be a matter of compliance but one of assessing the impacts of the studies it funds on a wide range of communities and groups.

Examples: NOAA’s Diversity and Inclusion Strategic Plan 2020–2024 includes a policy statement that communicates a commitment to fully integrating diversity and inclusion into its business practices and culture (NOAA, 2020a). The strategy includes goals, objectives, tactics, and metrics for supporting a diverse and inclusive workplace and sustaining leadership commitment.

The North Slope Borough Department of Wildlife Management in Alaska provides an example of supporting program excellence through inclusion of diverse perspectives. Robert Suydam, senior wildlife biologist with the Department, described a situation in the 1970s where federal scientists undercounted bowhead whale population numbers, which would have been more accurate with consultation with Indigenous whaling captains. The impact of this deficiency in the science was a management decision—whaling quotas—that could not meet Indigenous community subsistence needs and cultural practices. With the

creation of the Department, Indigenous knowledge and leadership are incorporated into management decisions about sustainable harvest and the identification of other issues of interest. The Department’s activities involve collaboration and co-production with Indigenous community members. Suydam provided insights related to how to maintain an inclusive program, including improving scientists’ abilities in cross-cultural communication, investing time in relationships, working closely with Elders, being respectful and patient, and acknowledging contributions throughout the process (workshop presentation by Robert Suydam, June 24, 2021).

EPA considers the impact of its science and decision making through the lens of environmental justice focusing on the “fair treatment and meaningful involvement of all people … with respect to the development, implementation, and enforcement of environmental laws, regulations, and policies” (EPA, 2021). As part of this policy, it has produced a guidance document (EPA, 2015) promoting consistency by EPA staff as they consider environmental justice during rulemaking actions. The agency has produced additional technical guidance for regulatory analysis that provides the technical underpinnings to fully consider environmental justice during rulemakings.

Sample Evaluation Questions:

• How does the organization characterize and implement distinctions between surface-level diversity and deep diversity?
• Does the organization have a diversity, equity, and inclusion officer?
• What metrics or other criteria are being used to measure whether diversity, equity, and inclusion are being successfully integrated across the organization’s programs and into its scientific research agenda?
• What are specific examples of how diversity, equity, and inclusion have been successfully integrated into the organization?

Attributes That Apply to the Process Phase

The process phase of the committee’s conceptual model encompasses the scientific studies program’s procedures to identify questions that warrant a study, translate those questions into requests for proposals (RFPs), and select those intended for funding or undertaking. A first-in-class, use-inspired scientific studies program identifies its science needs collaboratively; carefully evaluates the questions that capture those science needs; determines to what degree they have already been answered; develops RFPs that accurately capture the scope of those questions; and uses procedures to solicit, review, and select proposals that are inclusive, invite innovation, encourage diversity, and have a high possibility of successfully answering the question.

Process Attribute 1: Clearly Identifies Science Needs—Identifies studies needed to address current and emerging scientific questions collaboratively and articulates those questions clearly.

Description: A first-in-class scientific studies program identifies the studies needed to address current and emerging science needs by working closely with users, stakeholders, rights holders, and affected communities, some of whom may go on to become collaborators. Users are the practitioners and policy makers whose questions are meant to drive the study development process (in the case of BOEM, these are the Division of Environmental Assessment and agency decision makers). Involvement of users in the development of science needs allows the studies to be planned and executed in a manner that will be responsive to the users’ needs as well as the decision context within which study results will be used (Jacobs, 2002). Many external audiences are frequently captured under the term “stakeholders,” but their relationships to BOEM’s activities vary and are distinguished here so that they may be explicitly recognized. Energy developers may be considered stakeholders in BOEM’s decisions about energy development, whereas rights holders (e.g., state, local, or tribal governments) have legal and cultural connections to the marine resources broadly (e.g., whaling traditions and rights). Other affected communities may include other economies dependent on the OCS, such as fishing communities and fleets.

Including all external audiences in identifying and—as needed—designing studies can improve researchers’ and users’ understanding of the impacts of their studies and resulting decisions. Early input also helps to ensure that minority, low-income, and Indigenous or local populations are afforded an opportunity to meaningfully engage with a studies program. Creative approaches may be required to engage all relevant groups meaningfully and fairly in order to meet requirements related to the Administrative Procedures Act and the Federal Advisory Committee Act while providing mechanisms of engagement that consider potential resource (e.g., time, funding, travel, Internet connectivity) limitations as well as differences in languages and cultures.

Every scientific endeavor starts with a clearly articulated question or problem statement. People who do not conduct scientific studies often use vague or ambiguous language to express their science needs or otherwise fail to frame their science needs in ways that facilitate formulation of hypotheses or other scientific questions. This is particularly true for science needs associated with emerging issues, where there may be a feeling of concern but not an urgent need.

A first-in-class scientific studies program works with external audiences to ensure that they communicate their science needs in ways that can be translated into clearly articulated, structured questions to facilitate (1) RFP formulation, (2) development of study designs appropriate to address the science need, and (3) findings that respond to the science need. Frameworks, such as the PICOC Framework (which stands for Population-Intervention-Comparison-Outcome-Context), can

help these external groups communicate their science needs accurately and precisely (see Booth, 2004; Booth et al., 2016; Cooke et al., 2012; Davies, 2011; Denyer and Tranfield, 2009; Richardson et al., 1995). Alternative frameworks are available for quantitative and qualitative questions so they are compatible with hypothesis-driven studies and traditional ways of acquiring knowledge (Booth et al., 2016).

Systematic reviews, data syntheses, and meta-analyses are also reliable methods for identifying questions that warrant study. These methods are particularly useful for understanding the current state of scientific knowledge of a subject, the level of confidence that can be assigned to answers to science needs, and questions that remain unanswered or that have not been answered adequately (Booth et al., 2016; Koricheva et al., 2013). Some of the procedures associated with these kinds of analyses are specifically designed to identify gaps in a body of knowledge and where data conflict. They are powerful tools for identifying targeted questions that warrant study.

Examples: Programs within EPA’s Office of Research and Development (ORD) engage personnel from related EPA programs and regional offices engaged in developing policy and implementing regulations in order to identify solutions-driven research (e.g., ORD’s Air, Climate, and Energy National Research Program supports the Office of Air and Radiation). They also engage with state, tribal, local, and multijurisdictional organizations through workshops and other regular meetings (workshop presentation by Bryan Hubbell, June 22, 2021). This engagement informs the process to develop the Strategic Research Action Plan, which outlines strategic research plans over a 4-year period (e.g., EPA 2020). This engagement is part of a systems approach, starting at the problem formulation stage and continuing through to the delivery of the science.

Crown-Indigenous Relations and Northern Affairs Canada (CIRNAC) is an agency established specifically to facilitate relationships between the government of Canada and First Nations, Inuit, and Métis; support capacity development and self-determination for Indigenous communities; and lead the government of Canada’s work in Northern areas. Activities of the CIRNAC provide well-established models for inclusion of rights holders and affected communities in all phases of a science program. For example, the Northern Contaminants Program’s⁴ health and environmental research is driven by concerns related to human exposure to contaminants through ingestion of animals that compose traditional diets. Indigenous partners receive funding to participate in the management of the program, including providing input on issues of concern, reviewing projects, conducting outreach, and revising program documents with an overall goal of building capacity in these communities to sustain this work. Individual Indigenous hunters participate in collection of samples from subsistence species.

___________________

⁴ See https://science.gc.ca/eic/site/063.nsf/eng/h_7A463DBA.html.

Sample Evaluation Questions:

• To what extent are the science needs of users communicated using structured frameworks (such as the PICOC framework) to ensure that those needs have been clearly articulated?
• To what extent are science questions, in addition to those identified by users, identified collaboratively using workshops; direct involvement of affected communities, industries, and other stakeholders; and potential collaborators?
• To what extent are those user-inspired questions refined by working with stakeholders, rights holders, and affected communities, including potential collaborators?
• How are systematic reviews, data syntheses, and meta-analyses used to identify data gaps that create science needs?
• To what extent are the science needs articulated as research questions or hypotheses that can be addressed by appropriately designed studies?

Process Attribute 2: Appraises and Translates Research Needs—Critically appraises the needs identified by users and ensures that these needs are translated into requests for proposals.

Description: A first-in-class scientific studies program uses subject-matter experts to critically evaluate science needs and questions and uses them to inform a research plan. Subject-matter experts are those whose breadth of knowledge and experience allows them to speak with credibility and authority on that subject. Subject-matter experts may be internal to the program but may also be external experts from government agencies, academia, industry, nonprofits, or local communities. They summarize knowledge on the subject, determine if studies exist that address the proposed questions, and help rank the questions on their urgency for addressing the issues. The subject-matter experts ensure that RFPs clearly invite proposals that meet the science need. Users, stakeholders, rights holders, and affected communities may not be aware that their questions have already been answered or have been answered sufficiently for the regulatory need. Subject-matter experts can evaluate the proposed science questions to determine whether or to what degree science needs have already been addressed, avoiding duplication of effort. This may involve a review of the literature, or more formal methods of systematic review or meta-analyses. While other factors may influence what subset of study topics are ultimately translated into RFPs (e.g., funding limitations or responsiveness to administrative priorities), this process ensures that those that are put forth for funding will be useful.

Example: The International Association of Oil & Gas Producers’ Sound and Marine Life Joint Industry Programme invites input from academic and nonprofit sector advisors to identify science needs (as described in Process Attribute

1). Subject-matter experts from member companies work with these advisors to develop research priorities. Additionally, the program continuously reviews its list of science questions to determine whether they have been answered or if they are still relevant, informing the next period of research planning (workshop presentation by David Hedgeland and Jürgen Weissenberger, June 24, 2021).

The Gulf of Mexico Research Initiative (GoMRI) held workshops with members of the research community throughout its 10-year tenure to refine its R&D topics, which were used by the GoMRI Research Board to develop research themes (Zimmerman et al., 2021). The GoMRI Research Board subsequently developed RFPs based on the science themes determined using community input.

Sample Evaluation Questions:

• How are subject-matter experts identified and selected?
• How does the program determine whether questions have been answered, the strength of those answers, and information gaps that are important to regulatory assessments, decision making, and policy development (e.g., literature review, systematic reviews, data syntheses, or meta-analyses)?
• Are RFPs developed or reviewed by subject-matter experts to ensure that they are presented in language that clearly identifies the science need?

Process Attribute 3: Encourages Diverse Investigators—Encourages and solicits proposals from a broad and diverse range of qualified investigators and ensures proposals are fairly reviewed and evaluated with criteria articulated in the solicitation.

Description: According to the National Science Foundation’s (NSF’s) State of U.S. Science and Engineering 2020,⁵ women accounted for 29% of the science and engineering workforce (with variability among fields) in 2017, while composing 52% of the college-educated national workforce. Underrepresented minorities (Black, Hispanic, American Indian, or Alaska Native) accounted for 13% of the science and engineering workforce and 17% of the college-educated workforce. Comparable data on cultural and attitudinal diversity are harder to find. However, “shallow” and “deep” diversity are related so these data on systemic underrepresentation problems suggest that traditional RFP practices may not access the “deep diversity” that drives excellence in organizational performance (as described in Cross-Cutting Attribute 5). By disseminating RFPs to scientists from underrepresented groups and ensuring that qualified investigators from those groups have equal opportunity to submit proposals as those whose work has been funded in the past, a science program encourages the kind of diversity that improves its organizational performance. Importantly, a more equitable approach to announcing funding opportunities will actively reach out to and recruit from

___________________

⁵ See https://ncses.nsf.gov/pubs/nsb20201.

historically excluded groups that include early-career scientists, women, people with disabilities, and minority investigators, and would increase the number of high-quality proposals. Additionally, clearly articulating the review process and criteria for selection in the RFP will help investigators formulate competitive proposals and ensure equity and fair consideration for all. Finally, these efforts can be paired with practices that eliminate historical barriers in procurement processes to ensure that all qualified proposals receive equitable consideration.

Example: Mike Weise, program manager for the Office of Naval Research’s (ONR’s) Marine Mammal Biology Program described the program’s approach to fostering a diverse and inclusive community of scientists (workshop presentation by Mike Weise, June 24, 2021). While he noted that his program has a community of researchers who have been funded multiple times, he strives to expand this pool to new scientists, including through outreach to and support of women and underrepresented groups. These efforts go beyond supporting research projects by including support for family leave, engagement of early-career scientists, and paid internships for students from Historically Black Colleges and Universities in order to provide equitable and inclusive opportunities for involvement by all communities of researchers.

NSF’s Proposal and Award Policies and Procedures Guide is a publicly accessible document that details the guidance for preparation and submission of a proposal by investigators as well as the procedures followed by NSF staff in awarding, administering, and monitoring grants. Included in this document are the broadly described merit review criteria against which all proposals are reviewed, specifically their intellectual merit and their broader impacts. Individual program solicitations similarly describe the proposal preparation guidelines, NSF procedures for reviewing and selecting proposals, and merit review principles and any program-specific criteria.

Sample Evaluation Questions:

• How much effort is focused on outreach to investigators from diverse communities to ensure those communities are aware of funding opportunities before RFPs are solicited?
• Do RFPs clearly identify the criteria that will be used to review and select proposals?
• How do RFPs encourage inclusion of diverse collaborators as part of the proposed project?
• Are proposals reviewed by subject-matter experts, and how are they selected?
• What are the demographics of investigators funded by BOEM?
• How many successfully funded proposals are from first-time BOEM investigators?

Process Attribute 4: Monitors Study Progress—Monitors the progress of its contractors and grantees and provides feedback through various means and forums.

Description: Once studies are selected and funded, monitoring the progress of awardees allows a funding program to assess whether the study is meeting expectations and avoiding project creep (the expansion of the project beyond its original goal to the detriment of obtaining the original goal). This includes an assessment of the progress toward meeting the goals of the award, the quality of the study, and its scientific merit. If unanticipated problems arise, this provides an opportunity to resolve the problems or rescope the project objectives and/or deliverables. The review can be conducted by peer reviewers or advisory bodies internal or external to the agency.

This time period can be used to assess whether collaborations are occurring as expected and to identify and develop opportunistic collaborations across intramural and extramural entities associated with the general topic or geographic area that had not previously been identified. Collaborative interactions help to build and maintain a community of researchers. Creating a forum for engagement on a particular topic can facilitate interactions across disciplines when multiple projects are approaching a topic from different perspectives. Mechanisms for interactions can include principal investigator meetings or workshops that can occur as independent meetings or affiliated with relevant conferences. Meetings that occur while studies are in progress can also include stakeholders, rights holders, and affected communities and can help develop trust between these groups, policy makers, and researchers. These mechanisms for collaboration can be implemented at the initiation of the studies to identify collaborations early on.

Example: ONR peer reviews projects while they are in progress around their second or third year. The primary purpose is to provide feedback to ONR rather than the investigators, although results are shared with them as well. The review consists of an assessment of the scientific merit, accomplishments, innovations, potential impact, and contributions of individual investigators. ONR also conducts topic reviews through expert panels and workshops about the state of the science (workshop presentation by Mike Weise, June 24, 2021). Similarly, the Navy’s Living Marine Resources (LMR) program conducts annual in-progress reviews with all principal investigators that allow for course correction as well as engagement between investigators. Investigators present their progress, the LMR Advisory Council provides feedback on projects, and investigators are able to learn more about other ongoing work (workshop presentation by Mandy Shoemaker, June 24, 2021).

The Outer Continental Shelf Environmental Assessment Program (OSCEAP) was a program by DOI and NOAA initiated in 1975 that funded research to assess the impacts of oil and gas development on the OCS of Alaska. OCSEAP funded

a diverse array of projects that ranged from physical oceanography to the distribution of marine mammals and seabirds to aquatic chemistry and the effects of oil on organisms. These diverse studies were integrated through workshops that focused on specific issues or questions. For example, when a particular region was being considered for leasing, workshops were been held which reviewed all of the studies that were ongoing or completed relevant to the region across a range of disciplines. Other workshops brought together researchers who worked on a specific topic (e.g., effects of oil on organisms). These workshops allowed information to be synthesized and shared with other researchers and groups (i.e., government agencies or the oil and gas industry), identification of remaining science needs, and assessment of the effectiveness of a program.⁶

Sample Evaluation Questions:

• What mechanisms exist for funded projects to be aware of the relevant research activities beyond their own effort?
• What mechanisms exist for coordination among studies throughout the execution of the study?
• What mechanisms exist for annual progress reports and identification of difficulties encountered?

Attributes That Apply to the Outputs Phase

The outputs phase of the conceptual model covers the products of the studies conducted by a studies program. Specifically, this phase considers how an agency ensures the validity of the reports they fund, how information generated by the studies is presented to a variety of audiences, and how the findings of various reports are considered as a whole. This phase does not consider how the information generated by the studies is used by the agency or by others.

Outputs Attribute 1: Ensures Product Quality—Establishes rigorous procedures that ensure that the completeness, objectivity, and validity of the reports, data, models, and other products generated are routinely applied.

Description: Procedures can ensure that study products are complete, meaning they produce the information that was intended, and that they meet quality standards so that BOEM can have confidence in its use within BOEM and by others. Two aspects of product quality require particular attention: objectivity and validity. “Objectivity” refers to whether the products are accurate, reliable, and unbiased (as described by OMB in 67 FR 8451), for example, that appropriate data and

___________________

⁶ For example, a summary of meetings on the Lower Cook Islands region, Bergin Sea-Gulf of Alaska Geologic Studies, and the topic of Marine Ecosystem Analysis can be found summarized in the May 1978 OCSEAP newsletter. See https://www.arlis.org/docs/vol1/OCSEAP/4542578/4542578-v3.pdf.

supporting information are provided and are coherent, in the sense that the findings and conclusions are supported by information presented or referenced in the text. Validity addresses the appropriateness of the study design and approach to the issue at hand and its relation to the conclusion (Matt and Cook, 2009). This includes consideration of factors such as the methods applied (i.e., study design, study power, control of potentially confounding variables, and sample size), the validation of models, and the quality of the data used. Validity also includes whether results can be applied broadly or are specific to the characteristics of the study.

OMB has established that peer review is an important procedure that can ensure the quality and credibility of published information. OMB’s Final Information Quality Bulletin for Peer Review (70 FR 2664) provides guidance on how to select reviewers and when to make use of peer review. OMB grants agencies “broad discretion to weigh the benefits and costs of using a particular peer review mechanism for a specific information product” but notes that “use of a transparent process, coupled with the selection of qualified and independent peer reviewers, should improve the quality of government science while promoting public confidence in the integrity of the government’s scientific products.” Consistent with these guidelines, the ESP Strategic Framework notes that specific peer review plans are developed for studies identified as “influential” or “highly influential” and that such a review might be conducted by the ESP, BOEM, DOI scientists, or a contracted expert panel (e.g., a committee of the National Academies). Importantly, the review process would include mechanisms to ensure that investigators respond to peer review thoroughly in the development of a final product.

Examples: NOAA Fisheries implements a variety of methods for product review, depending on the management decision involved, ecological and economic risks associated with the decision, or the political implications of the decision. Formal reviews of important documents may be conducted through its Center for Independent Experts or the National Academies. Peer review, either by internal or external experts, is commonly used. Self-review happens as well (workshop presentation by Evan Howell, June 22, 2021). BOEM uses the ESP Program Assessment Tool to evaluate whether studies met their objectives, had a clear use, were conducted on time and in budget, and were subjected to peer review (workshop presentation by BOEM staff, June 22, 2021).

Sample Evaluation Questions:

• What proportion and type of study reports are subject to review by independent subject-matter experts with review questions that address the aspects of objectivity and validity? How are reviewers selected, and what are their breadth and diversity of expertise?
• What procedures are in place to ensure that study deliverables are complete in relation to the intended objectives and whether the questions are answered? How effectively are these procedures applied?

• How does BOEM ensure that comments made by the reviewers are responded to appropriately?
• What guidance does BOEM provide to ensure that all products reach an appropriate standard for credibility and objectivity?

Outputs Attribute 2: Encourages Tailored Products—Encourages or develops an array of outputs and communication strategies for a study tailored to specific uses by broad and various audiences.

Description: The products of a studies program can be of use to a wide range of groups. However, a primary audience will be the users who helped define the scope of studies. A primary purpose of the ESP is to inform assessments of the environmental, social, and economic impacts of the activities BOEM authorizes; through those assessments the ESP informs policy decisions on the development of energy and mineral resources on the OCS. However, the nature of many management-oriented studies is such that their study outcomes may affect multiple communities, industries, and partner agencies; may be of interest to legislators and the general public; and may produce science applicable to studies in a wide range of disciplines. For example, BOEM’s role in offshore wind energy development may result in scrutiny by audiences not previously familiar with BOEM’s studies. Different types of products will have varying degrees of effectiveness in reaching each audience. Coordination and consultation with users of the information and other targeted communities (e.g., rural Alaskan Indigenous communities with limited Internet access) can help identify the types of products and means of communication that work best for them. Additionally, the frequency and timeliness of the products will also influence how they are received and used. Taking the needs of each audience into consideration will allow for the broadest possible benefits to be gained from the investments made in the study (Parkin, 2004).

An agency that funds science extramurally like BOEM can encourage or require the consideration of an appropriate product and communication plan based on potential audiences of the results when making funding decisions. Products could be static (written products including narrative and visuals), interactive (products that allow user selection of information, such as web-based or other electronic media, including searchable summary information) or interpersonal (direct dialogue and exchange of information including peer-to-peer information through formalized networks). Forms of dissemination can also vary, occurring through online portals, social media, or as targeted distributions of digital or printed forms. Consideration may also need to be given to establishing trusting relationships and working across linguistic and cultural differences in order for communication of study results to be effective.

Examples: NOAA Fisheries has contracted with COMPASS,⁷ an organization that specializes in helping scientists share their findings in accessible and understandable ways with the people who need it (workshop presentation by Evan Howell, June 22, 2021). The CFRN produced “product bundles” that include varied ways of disseminating the findings of a study, including peer-reviewed publications, technical documents, conference presentations, posters, blog posts, workshop reports, feature news articles, manuals, videos, and brochures⁸ to ensure that the information is accessible to relevant audiences. This approach may be especially important where industry and other interest groups may have different levels of experience understanding the findings of scientific studies.

Sample Evaluation Questions:

• How are plans for product development and distribution encouraged or developed prior to the study completion?
• What constitutes timely delivery of information generated by the studies?
• What proportion of studies and study findings are communicated to the broader scientific community, to industry, to communities affected by BOEM activities, and to the public?
• What guidance is provided to study scientists on communication and dissemination approaches for various product types/audiences?
• How are appropriate products determined, including emerging forms of information (e.g., storyboards or social media), and how often are these determinations re-evaluated?

Outputs Attribute 3: Synthesizes Scientific Findings—Synthesizes its scientific findings in the context of existing knowledge, and produces outputs for broad distribution.

Description: While some studies may have independent utility, many build on existing work or produce findings that need to be seen in the context of existing practice or other ongoing research. Regular synthesis of scientific work through the production of overview summaries can help make those connections and identify where studies are successfully leveraging other science investments or contributing to the broader knowledge base for use by many. Syntheses may include high-level interpretation of outputs, their relationship to existing knowledge, and connections among findings and outputs of individual studies. Synthesizing technical information from multiple sources and presenting the findings in ways that provide a new level of understanding is a well-established scientific practice (Carpenter et al., 2009). Integrating distinct studies under a common framework

___________________

⁷ See https://www.compassscicomm.org.

⁸ See http://cfrn-rcrp.ca/Public-Products-EN.

can generate novel insights, identify emerging needs, and lead to new approaches to interpreting and using observations or data, as well as presenting a more holistic view of BOEM’s science vision and strategy.

Examples: There are a variety of scales at which such holistic presentation of information can be achieved. GoMRI utilized synthesis workshops⁹ to capture information from a variety of funded studies. These were designed to be working sessions with specific outputs; therefore, participants were identified based on their expertise and intended contribution, rather than the sessions being open to a wide audience. The use of synthesis centers, such as those funded by NSF¹⁰ and USGS,¹¹ allows researchers to come together and work across disparate data sets and studies to find new insights. While this approach may be periodically useful to BOEM and its partners for key issues, it may not be useful as a routine way of bringing together information from a variety of studies.

The NOAA Science Council produces an annual NOAA Science Report that highlights the research accomplishments from NOAA and its academic and industry partners (e.g., NOAA, 2021a). It provides summary information on the NOAA science program as a whole and features particular contributions that are novel or of key interest and potential importance. The document, while including technical information, is written and presented to be accessible for a broad audience. BOEM’s $30 million annual ESP investment is a fraction of NOAA’s R&D budget ($725 million in FY2020), which could mean a more compact document, but the approach of communicating diverse scientific information in a single format to a broad audience could be of value.

Sample Evaluation Questions:

• Can a collective representation of the program’s scientific findings each year be found anywhere? Which audience is it tailored for?
• What types of syntheses of science work could be of value, and are opportunities made available for agency scientists or funded investigators to participate?

Attributes That Apply to the Impact Phase

The Impact phase of the conceptual model encompasses all of the processes a scientific studies program uses to assess the impact or influence of the studies it funds. A first-in-class scientific studies program actively assesses the impact or influence of individual studies, collections of studies, and the studies’ program generally on agency assessments, agency decisions, the state of scientific

___________________

⁹ See https://gulfresearchinitiative.org/gomri-synthesis/workshops-events.

¹⁰ NCEAS: https://www.nceas.ucsb.edu; SeSync: https://www.sesync.org.

¹¹ John Wesley Powell Center for Analysis and Synthesis: https://www.usgs.gov/centers/powell-ctr.

knowledge, and public understanding of issues related to the agency’s activities and decisions. There are a variety of ways of assessing the impact of individual studies and scientific studies programs (Meagher and Martin, 2017; Nutley et al., 2007). This report discusses impacts using a general approach adapted from Penfield et al. (2014) and Scoble et al. (2010). These authors recognize that scientific studies can have an impact because a user reads and is influenced by a study report, because study investigators are asked for advice, or because study outputs have been adopted by a broader community.

Assessments of study impacts will tie directly to how responsive the studies are to the input received from users and stakeholders, rights holders, and affected communities described in Process Attribute 1, as well as the national context and policy drivers described in Cross-Cutting Attributes 1 and 2.

Impact Attribute 1: Informs and Influences Users—Is informative and has influence on the users who requested or inspired the studies.

Description: A first-in-class scientific studies program will assess the impact of its outputs on the users, stakeholders, rights holders, and affected communities who requested the studies or whose science needs inspired the studies. Before new information, findings, and ideas can influence decisions, the information has to spread through an organization and other social systems, then it has to be adopted by individuals who then integrate this new information into their work, decisions, or recommendations (Mascia and Mills, 2018; Rogers, 1962). How quickly study outputs spread through an organization depends on how quickly users adopt and integrate study results, how many users resist adopting them, and the amount of community consensus that forms around study results (Mascia and Mills, 2018; Rogers, 1962). Consequently, it may take time to measure the influence of study findings on a target audience or decision makers; because some decision-making processes can take several years to unfold (e.g., leasing decisions or records of decisions associated with NEPA procedures), the influence of study findings may not be realized for a decade or longer. Because a primary purpose of the ESP is to develop information that supports BOEM’s assessments of the environmental, social, and economic impacts of the activities BOEM authorizes, one measure of study impacts is the degree to which studies inform and influence BOEM NEPA documents, biological assessments, and consultations with federal and state resource agencies. The degree to which studies are cited in regulatory documents, particularly the degree to which they are cited as evidence that supports regulatory arguments, conclusions, findings, and policy positions, would be one measure of whether studies inform BOEM assessment. The degree to which studies are cited as evidence that supports a change in prior positions, arguments, conclusions, findings, and policy positions would be a measure of whether studies influence those assessments. When studies

have served as the foundation for a prediction of environmental outcomes, monitoring actual outcomes can be an indicator of the studies impact.

BOEM’s studies and assessments inform broader agency decisions related to OCS leasing decisions and mitigation requirements for energy and mineral development. Agency decision making and policy making are complex processes. While decisions and policies are informed by scientific studies, they may also be influenced by policy makers’ values and personal experiences, and by how competing sources of information are weighed (Black, 2001; Brownson et al., 2009; Makkar et al., 2016). As a result, assessing the impact of studies on agency decision making and policies will be difficult to measure.

Example: NOAA employs logic models to guide evaluation of whether its research has accomplished its goals (e.g., those articulated in NOAA, 2020b). A logic model is a tool that depicts (usually in flow chart form) the relationship between components of a system, such as the inputs, outputs, and impacts of a science program. Gary Matlock, deputy assistant administrator for science at NOAA, used the stock assessment for Alaska walleye pollock to illustrate how NOAA uses logic models (workshop presentation, June 22, 2021). The logic model tracks the observations and population models made by scientists, which inform the walleye pollock stock assessment. Based on the assessments, NOAA developed a report describing the state of the stock, which informed decisions (e.g., catch limits) of the North Pacific Fishery Management Council.

NOAA Fisheries’ Office of Science and Technology employs national metrics and feedback to evaluate impacts of its science on sustainable harvest of managed species, effective management of protected species, and supporting partners in understanding environmental impacts from planned work or ecosystem changes (workshop presentation by Evan Howell, June 22, 2021). The metrics that apply to these impacts include the status of fish stock (e.g., has overfishing been reduced); endangered species status (e.g., has a species improved or been delisted); and successful technical reviews of projects, process studies, or other work (e.g., successful completion of sample processing, observations, or surveys).

Sample Evaluation Questions:

• What proportion of study findings are presented as evidence that supports arguments or conclusions in environmental documents (as opposed to the proportion of studies that are mentioned or cited parenthetically)?
• What proportion of the studies are designed to influence national or agency policy and/or decisions?
• What proportion of study findings are cited as supporting decisions in decision documents?
• What proportion of study findings are cited as supporting regulatory actions (e.g., Federal Register notices or enforcement actions)?

Impact Attribute 2: Advances the State of Science—Produces products that are used by other investigators and that advance the state of scientific knowledge.

Description: A scientific studies program that aspires to be recognized as first-in-class will produce high-quality studies that are relevant to the broader scientific community as well as to its primary audience. As described in the ESP Strategic Framework, BOEM recognizes that “many of the most important scientific questions and answers on environmental impacts depend on broad-based, long-term research rather than narrowly defined studies,” and as a use “inspired” program, not all studies will be directly tied to immediate uses. An added value of funded studies, with or without immediate uses, will be the impact of the state of knowledge and the development of skilled scientists more broadly. A studies program that addresses current and emerging science-related questions collaboratively; that relies on systematic reviews, data syntheses, and meta-analyses to identify questions that warrant study; and that articulates science needs as research questions or hypotheses (see Process Attribute 1) will be particularly impactful. A studies program that employs such a process can address questions that are relevant to the broader scientific community. As a result, the degree to which other investigators find study methods or findings useful or the degree to which study methods or findings are adopted by the broader scientific community is a measure of the program’s influence or impact.

Unlike Impact Attributes 1 and 3, a variety of methods have been developed to measure the impact of particular studies or groups of studies on the broader scientific community. These impacts are usually measured using bibliometrics, or statistical analyses of books, articles, and other publications that track the impact of authors, particular articles, or journals. Most bibliometrics rely on specific metrics that measure the impact of articles (citation counts), authors (e.g., H-Index or G-Index), and journals (e.g., journal impact factor). Web services, such as scite,¹² go beyond citation counts and identify whether study findings have been supported, mentioned, or contrasted by other investigators. Several studies have shown that open-access articles are cited more and have higher impact scores than non-open-access articles, and this pattern holds across scientific disciplines (Gargouri et al., 2010; Hajjem et al., 2006; Wang et al., 2015). For example, Eysenbach (2006) reported that open-access articles published in the Proceedings of the National Academy of Sciences were twice as likely to be cited as non-open-access articles in the same journal.

Studies of these bibliometrics have found that impact metrics that rely on arithmetic mean values (i.e., the average citation count of a research group compared to all publications) can be biased (e.g., see Bornmann et al., 2008). Bibliometrics that either rely on citing networks, co-citation networks, or cited

___________________

¹² See https://scite.ai.

networks, such as “relative citation ratio” (e.g., see Hutchins et al., 2016, and wizdom.ai) or metrics that do not rely on arithmetic mean value, such as box plots, Lorenz curves, and Gini coefficients (see Bornmann et al., 2008) will be more representative of the impacts of individual articles, authors, or a program on the scientific community. Lorenz curves and Gini coefficients identify the proportion of an author’s or program’s citation counts that are associated with specific articles; lower Gini coefficients mean that a higher proportion of articles have high citation counts. These metrics may vary by discipline; studies on marine mammal impacts may have lower coefficients than sociological studies. These metrics can be applied to individual studies, research portfolios (studies undertaken by individual investigators, groups of researchers, or within a particular subject), and programs.

Example: Several agencies have used bibliometric methods in the past to evaluate federally funded research (e.g., Drew et al., 2016; Hicks et al., 2004; Howell and Yemane, 2006). The National Institutes of Health’s (NIH’s) Office of Portfolio Analysis endorsed the relative citation ratio to quantify the impact and influence of research articles NIH funds as well as other articles. That office maintains the website iCite,¹³ which allows anyone to assess the influence of individual articles or groups of articles using relative citation ratios. NOAA’s Central Library uses bibliometrics to produce research impact reports that include counts of publications with NOAA authorship, network analysis of intramural (within NOAA) collaborations, and counts of extramural and international collaborations with other organizations (Davis et al., 2019). NOAA’s annual science report also includes bibliometrics developed using InCites¹⁴ that include publication counts, publications that are in the top 10% of citations, and H-indexes by core research area (e.g., NOAA, 2021a). The bibliometrics are used to assess the value of NOAA’s publications and to create benchmarks that can be used to compare NOAA research outputs with those of other agencies.

Sample Evaluation Questions:

• What proportion of the research was designed to be influential to the state of science in a particular area, and was it realized as such?
• What proportion of studies are accessed and downloaded from ESPIS?
• What proportion of ESP studies are published in scientific journals, particularly studies that are published as open-access articles?
• What is the trend of BOEM’s relative citation ratio on wizdom.ai?
• Using metrics such as Gini coefficients, what proportion of BOEM studies are responsible for high citation counts?

___________________

¹³ See https://icite.od.nih.gov.

¹⁴ See https://incites.clarivate.com.

Impact Attribute 3: Influences Public Understanding—Influences understanding by the public and engenders acceptance of agency science and decisions.

Description: Putting the best available science at the forefront of federal agency decisions is an opportunity to increase public trust in decision making. In a recent Pew Research Center survey, the proportion of U.S. adults who said they trusted scientists to “act in the best interests of the public” rose from 76% in 2016 to 86% in 2020 (Pew Research Center, 2020). Public trust in government is far lower (Pew Research Center, 2021). A scientific studies program whose primary purpose is to inform assessments of the environmental, social, and economic impacts of the activities a federal agency authorizes and, through those assessments, inform policy decisions on the development of energy and mineral resources on the OCS, can bridge this gap if it is credible and respected by the public, including stakeholders, rights holders, and affected communities.

As noted with the Outputs attributes, engendering understanding and acceptance requires active, consistent, and effective communication of study purposes, study findings, and syntheses. It requires using language that is intended to communicate with an audience (rather than to an audience) and awareness of how information spreads through different communities and cultures before it can be considered credible and reliable (Rogers, 1962). Accomplishing this in turn requires that agencies ensure fair and meaningful engagement and participation of stakeholders, rights holders, and affected communities (including tribal consultations, consultations with historically Black or otherwise underrepresented communities, public comment and listening sessions) throughout all phases, culminating in the Impact phase, and ensure that no community or group will disproportionately bear the burden of environmental impacts resulting from government action. Stakeholders, rights holders, and other communities affected by an agency’s decision making will be particularly attentive and most likely to provide public comments to agency decisions. Adhering to the principles of environmental justice is an important element in building a productive relationship with stakeholders, rights holders, and affected communities and trust from the broader public.

Alternative metrics (“altmetric”) use citation counts, news sites and news aggregators, blog posts, social media mentions, Wikipedia citations, and citation manager bookmarks to measure the impact of articles and authors (see altmetric.com and the Hunt Library’s EAGLEsearch).

Example: FDA considers the impact of the outputs of its Centers of Excellence in Regulatory Science and Innovation on its stakeholders (FDA, 2018). Specific metrics for these impacts include adoption/adaptation of findings by stakeholders or use in advocacy, technology transfer to stakeholders, improvements in consumer understanding, and adoption for use into medical practice. These impacts

in turn support development or change in guidelines, regulations, policy, and other measures that inform regulatory decision making.

Sample Evaluation Questions:

• What is the trend of non-citation-based mentions (e.g., press or social media), such as reported by altmetric.com?
• What proportion of study findings are adopted or adapted by public advocacy groups or other groups or in public comments on agency decisions or other activities?
• Does the study program explicitly include a stakeholder outreach element?
• How has the studies program considered impacts to minority, low-income, and Indigenous or local populations?

Attributes That Apply to Innovation

To be first-in-class, it is incumbent upon an agency to foster the growth of basic and applied marine scientific research by including scientific innovation as a cornerstone of its research programs. Innovation in a government agency means improving on existing solutions or developing new approaches to policy. It can also mean building and improving upon solutions that already exist. Risk-tolerant initiatives supporting cutting-edge research can produce a range of innovative, environmentally responsible energy projects and policies by doing the following:

• Stimulating technological innovation
• Providing the R&D needed to support changing and emerging regulatory/policy functions
• Fostering and encouraging participation by a wide and inclusive range of groups representative of diverse cultures, values, and attitudes

Innovation is supported by collaboration, cooperation, and equity among a diverse group of industry, academic, and government partners to create a vibrant scientific ecosystem with a number of attributes.

Building an organizational ecosystem that fosters innovation in response to new scientific issues, management demands, and emerging technologies is a challenge to all government agencies. However, an agency’s nimbleness in response to changing conditions is one of the defining attributes of a first-in-class program.

Innovation Attribute 1: Seeks Opportunities for Innovation—Embraces a risk-tolerant research agenda by explicitly identifying innovation as part of strategic planning.

Description: There have been numerous initiatives promoting innovative research that are tolerant of risk in the federal government. In 2021, the House Science Committee, in considering strategic challenges faced by the U.S. research enterprise, called for a “bold and more inclusive model for innovation in the 21st century.”¹⁵ In testimony, Dr. Farnam Jahanian, president of Carnegie Mellon University, argued that “to choose between increased support for curiosity-driven research versus strategic mission-driven initiatives in emerging technologies ... is a false choice. Scientific exploration occurs along a dynamic continuum.” As a general rule, it is very difficult to promote innovation without accepting risk. Despite massive federal investments, innovation remains a frustrating pursuit. The difficulty with sustaining innovation may be in part because, while agencies regularly define their mission-oriented strategies, they rarely define their innovation efforts within their strategic plan (i.e., an innovation strategy). An agency that develops a conscious strategy supporting innovation and embraces a tolerance for risk promotes advances in its science and improves the quality of its scientific advice.

Example: USACE has an explicit goal to “foster innovative technology development and implementation within USACE” (workshop presentation by Todd Bridges, June 22, 2021). The agency considers innovation to be a “fundamental business practice” that is consistent with the Army Innovation Strategy. USACE convened an “Innovation Summit” in September 2019 to hear from its partners and stakeholders how innovation can be fostered in the USACE, resulting in the publication of the USACE’s “Technology Innovation Strategy” in August 2020. Planning for R&D builds on the goals of the technology innovation strategy; for example, the ERDC 2020–2030 Strategy by USACE’s Engineering Research & Development Center connects to the innovation strategy through finding or creating new technologies and capabilities; building solutions to tactical, operational, or strategic requirements from these innovations; and consulting with users throughout the development process in order to deliver useful solutions (ERDC, 2020).

Sample Evaluation Questions:

• Is an overarching policy statement supporting innovation a key component of the strategic plan?
• How often is the innovation strategy updated through reviews of the science portfolio to identify emerging science needs?
• What proportion of the scientific program is regularly evaluated to consider whether innovations in methods, questions, or policies are necessary?

___________________

¹⁵ See opening statement by Chairwoman Eddie Bernice Johnson at https://science.house.gov/hearings/reimagining-our-innovation-future.

• To what extent are a diverse and inclusive range of groups involved in the identification of emerging needs?
• To what extent does the process for defining the research agenda allow for research questions that challenge traditionally held views?

Innovation Attribute 2: Adapts to New Challenges—Supports a science program that improves understanding of the changing conditions and responds nimbly to new issues and challenges.

Description: Natural resource agencies now confront the challenge of making decisions when faced with an environment that is changing in novel and unpredictable ways. As an example, the northwest Atlantic and Arctic Oceans are experiencing some of the most rapid and intense climate-related changes in water temperature with associated impacts on circulation, ice extent, and marine species occurrence and behavior. The Gulf of Mexico is affected by the increasing frequency and strength of summer storms and hurricanes. These patterns are projected to continue or become more severe in the foreseeable future. In these circumstances, historical environmental data and models will be insufficient to support assessments of future environmental consequences and future management decisions; however, institutional memory and knowledge coupled with the information contained in legacy datasets will still be crucial for addressing future problems.

At the same time, these agencies also need to respond to evolving priorities (e.g., an emerging focus on critical minerals for BOEM) and policy changes that address those priorities. This will require scientific studies programs that can balance a need to produce information that supports contemporary management decisions with a need to produce information that anticipates and responds to future management questions and decisions.

To be responsive to future needs, agencies will have to think strategically and invest in developing new study designs, sampling methodologies, analytical methods, system modeling, and forecasting procedures that improve understanding of changing environments and policy responses. As NSF noted in the Strategic Plan for Fiscal Years (FY) 2018–2022, “There is growing consensus that some of the most intractable problems in the scientific, technological, and social arenas require perspectives and approaches from multiple disciplines” (NSF, 2018). Strategic thinking and innovating new study approaches will require stronger communication and collaboration between interdisciplinary groups made up of individuals with diverse interests, ideas, and experience (Bridle et al., 2013) and who are encouraged to think outside the box (NSF, 2018).

Example: Support for innovative approaches is at the forefront of the Strategic Action Plans in EPA ORD’s six strategic areas. For example, the Air and Energy Strategic Action Plan, 2019–2022 highlights the need for innovative thinking to

address growing complexity of problems from emerging pollutants, wildfires, a diversifying energy portfolio, changing land uses, evolving transportation systems, and climate change (EPA, 2020). Due to past investments, the Air, Climate, and Energy Research Program is well poised to address issues emerging now. These investments include development of low-cost air quality sensor technology, energy system models, and electronic health records, all of which support current concerns related to energy system transitions, wildland fires, and disproportionate impacts in communities with environmental justice concerns (workshop presentation by Bryan Hubbell, June 22, 2021).

NOAA’s Climate and Fisheries Initiative provides a nimble approach to understanding rapid changes in the marine environment and thus providing the scientific information needed to manage marine resources. The initiative is a cross-NOAA effort to build an operational ocean modeling and decision support system that NOAA’s (and other organizations’) resource managers can use “to reduce impacts, increase resilience, and help adapt to changing ocean conditions … today, next year, and for decades to come” (NOAA, 2021b). The initiative involves ongoing research and observations to continuously validate and improve the modeling and decision support as knowledge is gained and conditions continue to change in the future.

Sample Evaluation Questions:

• How does the agency use scenario planning and management strategy evaluations to anticipate future decision making and the scientific information those decisions will require?
• How does the agency ensure that it balances a need to fund studies that support current management needs with a need to anticipate and respond to emerging issues and challenges?
• Does the agency engage in public–private partnerships, partnerships with academic institutions, international collaboration, or others to anticipate emerging issues and challenges that will influence agency decision making?
• Does the agency convene interdisciplinary groups representing diverse expertise, interests, and experiences to identify emerging science needs, challenges, technologies, and study methodologies (e.g., Next Generation Science Standards)?

Innovation Attribute 3: Implements an Innovation Strategy—Implements a system to move an innovation strategy forward, including resources, skills, time, and space dedicated to supporting innovation.

Description: Innovation improvement efforts can easily become a grab bag of best practices, and there is nothing wrong with any of those practices per se. An agency’s capacity for successful innovation relies on having processes and

structures in place (an “innovation system”) that the agency follows to identify and pursue the necessary activities to allow it to support strategic innovation. Implementing such a system increases the potential for successful innovation, and as a result improves the capability of the agency to provide the best scientific information available to its users, partners, stakeholders, rights holders, and affected communities.

Example: Innovation is a key component of FDA’s Office of Regulatory Science and Innovation, an office of the chief scientist that provides strategic leadership to foster excellence and innovation throughout the agency (workshop presentation by Tina Morrison, June 22, 2021). One focused effort is the “Chief Scientist’s Challenge Grants,” which provides intramural funding for innovative, high-risk, collaborative efforts in FDA’s priority science areas. These grants leverage creative ideas throughout the agency to solve complex problems. Proposed projects undergo review by internal or external reviewers in order to select which projects will be funded.

An innovation system is embedded in the NOAA Research and Development Vision Areas: 2020–2026 (NOAA, 2020b). This report provides a characterization of the priorities and guidance for NOAA’s R&D activities for the future and integrates innovation objectives (e.g., lead innovations in environmental sensors, unmanned systems, data processing, use of artificial intelligence, or cloud computing) into the overall R&D vision. Uses include the following:

• Provide direction and guide investments for NOAA and NOAA-funded R&D priority research areas,
• Set concrete targets with which to measure progress, and
• Create a common understanding of the purpose of NOAA’s R&D for NOAA’s leadership, workforce, partners, constituents, and Congress.

Sample Evaluation Questions:

• Does the organization have a concerted push toward innovation, and if so, is this articulated in ways that help create an internally cohesive research program and a sense of purpose for the organization?
• What proportion of funding opportunities (both intra- and extramural) are provided to support and award the research needed for emerging science?
• How much does the agency invest in innovation, including expanding and updating the knowledge, skills, and abilities of its staff?
• How does the agency encourage and recognize staff innovation through its performance management system?
• What procedures are in place to remove barriers to interdisciplinary communication and collaboration?

A WAY FORWARD

This report outlines a framework and suggested approach for the evaluation of a use-inspired science program (in this case, BOEM’s ESP and its connection to the broader BOEM environmental program). This framework could be used to guide both self-evaluations and evaluations that involve external reviewers. However, the committee considers a self-evaluation an important first step for implementation of the framework that will include establishing evaluation procedures and envisioning desired program goals, as well as providing an initial assessment of the program. Once goals and procedures are established, evaluations can be periodically conducted and external evaluation introduced. External evaluation allows for broader feedback from subject-matter experts and can build on the self-evaluation.

Recommendation: BOEM should develop procedures and conduct regular evaluations to assess whether and how well its environmental program meets the attributes of a first-in-class program and to identify areas for improvement. In order to implement evaluation and improvement efforts, BOEM will need to do the following:

• BOEM should develop ways to measure its success related to each attribute. Responses to the evaluation questions suggested in this report will enable BOEM to assess how well an attribute is being met. By design, these questions usually cannot be answered with a “yes” or “no” to prompt consideration not only of whether something is being achieved but whether it is being achieved at a sufficient level. BOEM may find it valuable to identify additional evaluation questions and other tools of relevance, or make adjustments to tailor the evaluation approach to the agency’s mission. BOEM will need to develop tools, such as metrics, that it considers appropriate to progress toward first-in-class status. These should be shared within and outside the organization to foster transparency and ensure accountability. Finally, BOEM may wish to consider prioritizing the desired attributes and implementing those that are of highest priority to the agency or easiest to accomplish—or both—first.
• BOEM should implement improvement actions based on the outcome of its evaluations. Once BOEM has developed measurement tools, such as evaluation questions or metrics and realistic targets, BOEM can conduct an evaluation and use its results to determine which attributes need additional consideration by the agency. To be a first-in-class program, BOEM will need to develop a strategy to continuously improve its environmental program to achieve the targets established for attributes that are not being met or not being met well. This may require prioritizing and/or staging implementation of improvement activities. Such prioritization or staging may be based on the expected value of

meeting a specific attribute or the near-term feasibility of implementing improvements.

• BOEM should involve external advisors in the development of evaluation and improvement initiatives. While BOEM should develop its own evaluation procedures and goals, input and oversight from outside advisors would be beneficial for developing the evaluation strategy and for conducting periodic external evaluations. External involvement allows for independent, objective input and accountability and provides increased capacity. For example, the development of ways to measure success or the prioritization of improvement efforts would benefit from consultation with others. BOEM already receives feedback on its activities by various means, including short-term efforts like the Evaluating Connections study or longer-term relationships such as that with COSA. COSA provides an opportunity for informal feedback from its membership and serves as a forum for meetings on targeted topics with participation from headquarters and the regions. Models may also be taken from other agencies, such as those examples shared at the committee’s workshop series described throughout the report. Lastly, the Baldrige Performance Excellence Framework may offer high-level guidance regarding criteria BOEM may want to incorporate into its evaluation.
• BOEM should institutionalize evaluation and improvement processes and procedures. Evaluation and improvement should be a regular, repeated occurrence within an agency. Responsibility for these efforts should be distributed across the agency staff (i.e., in each division of the OEP and in the regions). Success of evaluation efforts relies on careful tracking of BOEM’s procedures and outputs by staff across the agency so that evaluations can be informed by robust data. Improvement efforts similarly will need to be implemented broadly and tracked. Therefore, BOEM leadership support and buy-in are critical, and they should lead development of the strategy for rolling out evaluation and improvement efforts and foster an agency culture that values continuous improvement and investment to achieve these improvements. A supportive agency culture will not only be important for broad-based support of these efforts, but many improvements (e.g., commitments to diversity, equity, inclusion, and justice) will themselves likely require culture shifts. Staff participation in evaluation and improvement efforts should be embedded in their annual performance plans and evaluations, and care should be taken to incentivize staff to make positive changes to agency programs.

Presentations from the committee’s workshop series illustrated that evaluation occurs throughout federal agencies and other science programs under varying approaches. For example, NOAA Fisheries is using a tiered series of reviews where evaluations of individual science centers and program areas were followed

by a “review of reviews” that bundled outcomes and recommendations and will culminate in an enterprise-wide review of all NOAA Fisheries science in 2021–2022. NSF uses a systems approach to evaluate various aspects of its research enterprise; committees of visitors review the processes, outputs, and impacts of individual funding programs while directorate-level advisory councils provide recommendations and oversight for the research portfolio.

A variety of tools is available to support evaluations as described in the examples throughout this report. Evaluation and improvement efforts can be informed by information collected through surveys of users or partners; feedback from a wide range of advisors and reviewers (not just within the organization); annual review cycles to improve quality of staff and program products; and a common practice of longer-term (5- to 10-year) lookbacks focused on improvement of science, staff, and impacts. Subsequent improvement actions will vary immensely and may result in changes in organizational structures, processes, and priorities.

Appendixes

1. Statement of Task
2. Committee on the Assessment and Advancement of Science in the Bureau of Ocean Energy Management’s Environmental Studies Program
3. Workshop Series Agenda
4. Acknowledgments
5. References

This page intentionally left blank.