Under a mandate from the U.S. Congress, the Advanced Research Projects Agency-Energy (ARPA-E) asked the National Academy of Sciences to conduct an assessment of the progress the agency has made toward achieving its congressionally mandated mission and goals. This report includes both an operational assessment of the agency’s funding programs and a technical assessment of its awards, to the extent possible. The ad hoc committee convened to conduct this assessment relied on quantitative and qualitative analyses of data on ARPA-E’s creation of technology-focused funding programs, its decision-making processes for granting awards, its management of projects and awardees, the patenting and publishing activities of awardees, and further investments made in awardee projects following ARPA-E funding.
There are clear indicators that ARPA-E is making progress toward achieving its statutory mission and goals, and it cannot reasonably be expected to have completely fulfilled those goals given so few years of operation and the size of its budget. Importantly, especially at this early stage, the committee found no signs that ARPA-E is failing, or on a path to failing, to deliver on its mission and goals. From its complete set of 18 findings, the committee developed 14 recommendations, which are listed in full at the end of this summary. This summary presents an overview of the study and highlights the 5 findings and 5 recommendations the committee believes are most important. The complete list of the committee’s findings and recommendations is presented in Boxes S-1 and S-2, respectively, at the end of this summary.
SCOPE OF THE ASSESSMENT
The first part of the committee’s task was to assess the progress ARPA-E has made toward achieving its statutory mission and goals during the first 6 years of its operation and whether it is on a trajectory to achieve them. Congress established the agency with a mission “to overcome the long-term and high-risk technological barriers in the development of energy technologies,” and specific goals to
- enhance the economic and energy security of the United States through the development of energy technologies that result in—
- reductions of imports of energy from foreign sources;
- reductions of energy-related emissions, including greenhouse gases; and
- improvement in the energy efficiency of all economic sectors; and
- ensure that the United States maintains a technological lead in developing and deploying advanced energy technologies.”1
The second part of the committee’s task was to conduct both an operational assessment and a retrospective and prospective technical assessment in the context of the agency’s statutorily defined means of achieving its goals “through energy technology projects by—
- identifying and promoting revolutionary advances in fundamental and applied sciences;
- translating scientific discoveries and cutting-edge inventions into technological innovations; and
- accelerating transformational technological advances in areas that industry by itself is not likely to undertake because of technical and financial uncertainty.”2
The committee’s operational assessment considers how ARPA-E is organized, how it selects projects to support, how it partners with performers to manage those projects, how it actively manages projects, and what nontechnical support it provides to projects. The technical assessment outlines how ARPA-E, through its project selection and management, has made progress toward producing commercial products with the potential to transform the energy sector. The committee also considered the value to ARPA-E of developing a framework, processes, and specific systems for the collection of data that will be valuable for continuous improvement of operational processes and can serve as the basis for future self- or external technical and impact assessments.
The scope of this study did not include providing a comprehensive benefit-cost analysis or other review of ARPA-E’s value, such as a comparison with other possible uses of federal funding. Such an analysis is infeasible with currently available data. The study scope did include consideration of what
1 42 U.S.C. 149 § 16538(c)(1).
2 42 U.S.C. 149 § 16538(c)(2).
lessons learned from the operation of ARPA-E may apply to other DOE programs, as well as factors that Congress should take into account in determining the agency’s future.
STUDY METHODOLOGY AND LIMITATIONS
The committee’s findings and recommendations are based on both quantitative and qualitative data, including agency data, publicly available data, observations at agency events, presentations by personnel from ARPA-E, DOE, and the Defense Advanced Research Projects Agency (DARPA), case studies of completed awards, consultations with current and former ARPA-E personnel, and consultations with individuals from other programs and offices at DOE. The development of new or transformative energy technologies from initial discovery to broad market deployment typically takes several decades. ARPA-E provides support for projects very early in this process, typically around first translation from scientific discovery to engineering—focusing on ideas at technical readiness levels 2 to 4—with the possibility of leading to a marketable product. Most ARPA-E awards last for about 3 years, much shorter than the decades required to commercialize energy technologies. Unsurprisingly, few data were available for this study regarding ARPA-E’s impact on energy technologies or the sector as a whole. Still, 6 years of operation provides data demonstrating the intermediate impacts of ARPA-E’s activities. The committee developed its findings and recommendations by analyzing these available data.
DEFINING ORGANIZATIONAL FEATURES OF ARPA-E
Within the Department of Energy (DOE), ARPA-E can be distinguished by its culture, methods, and focused mission and goals. Through the course of its deliberations and analyses of the evidence gathered for this assessment, the committee found that ARPA-E benefits from three defining organizational features:
- The director provides technical and leadership skills that enable and sustain a culture of empowerment.
- ARPA-E’s program directors are empowered with the authority, responsibility, and ability to make program- and project-related decisions.
- Active project management is important to ARPA-E.
Collectively, these three features have the potential to contribute to ARPA-E’s ability to achieve its intended mission and goals. The absence of these features would not guarantee failure, and their presence does not ensure success.
However, these features are important to creating a culture that can enable success.
Closely linked to the defining organizational features set forth above, the committee developed 18 findings based on the available data and evidence. While all of these findings are necessary for understanding the progress ARPA-E has made toward fulfilling its mission and goals, five stood out to the committee as especially important.
Finding 3-7:3 ARPA-E selects projects to fund through a multifaceted process that entails evaluating each project’s potential to contribute to the achievement of the agency’s goals should it be successful.
Quantitative evidence demonstrates that ARPA-E has instituted a system focused on finding and funding ideas with a high potential for impact on achievement of the agency’s goals. The evidence also shows that this system involves a numbers of stages, and that at each stage ARPA-E uses a multifactor process to make decisions regarding applications. This process emphasizes technical comments from internal and external reviewers; applicants’ responses to those comments; and a holistic assessment of funding recommendations that considers the technical content of the applications, the potential for impact on achieving agency goals should projects be successful, and nontechnical factors. The process addresses such important issues as portfolio balance, both across technical categories and within the funded program, with an eye to ensuring sufficiently varied approaches. This process is distinguishable from those of some other agencies that make funding decisions based principally or solely on numerical reviewer scores, often utilizing a strict cutoff that does not allow for discretion on the part of the agency or program directors. Strong and consistent evidence indicates that projects selected through ARPA-E’s process have the potential to yield measurable outcomes at least as good as, if not better than, those of projects that would have been selected had less discretion been allowed.
Finding 3-8: ARPA-E program directors have wide authority to develop new focused technology programs that are potentially transformative.
3 The committee’s findings and recommendations are numbered according to the chapter and ordering where they appear.
The committee consistently found both qualitative and quantitative evidence that ARPA-E’s program directors are empowered with wide authority to carry out their responsibilities, including those outlined in the agency’s authorizing statute. Program directors create new technologically focused programs through a process that encourages novel ideas with the potential to identify and promote revolutionary advances or translate discoveries into technological innovations. This process involves collaboration and critical review with the agency director, other program directors, and the wider research community. It also encourages the pursuit of ideas overlooked or ignored by other funders, as well as truly novel ideas.
Finding 3-9: ARPA-E program directors actively manage projects through technical research guidance and feedback, regular and frequent assessments of progress made toward stated technical milestones, and revision of milestones in response to new findings and research discoveries.
Throughout a program’s life cycle—from review of applications, through award negotiations, to completion of individual projects—program directors engage in active project management. They work closely with performers to create milestones, which can be modified in accordance with what the team learns through the course of its research. Program directors regularly engage with performers to discuss a project’s technical approach and collaborate to revise it based on results to date. They work with performers to modify their research approaches, when appropriate, in response to data obtained through the course of research. Such changes can range from minor modifications of protocols, techniques, or project milestones to more significant project restructuring or changes in direction. Program directors even can, and do, recommend personnel changes and work with performers to identify and recruit qualified personnel or subcontractors. They also recommend that the agency terminate funding early when projects repeatedly fail to meet their milestones and appear unlikely to do so in the future.
Collectively, findings 3-7, 3-8, and 3-9 suggest that ARPA-E has thus far maintained its independence from such constituencies as groups seeking funding. Implementation of several of the recommendations presented below, particularly Recommendations 3-1, 4-3, and 4-8, would help ensure that ARPA-E maintains its independence.
Finding 4-2: The projects ARPA-E has funded support its statutory mission and goals.
Finding 4-3: While 6 years is not long enough to produce observable evidence of widespread deployment of funded technologies, there are clear indications that ARPA-E is making progress toward its statutory mission and goals.
While the full market impacts of the technologies that ARPA-E has funded undoubtedly will not be seen for years, some intermediate outcomes are evident now. Roughly half have published results of their research in peer-reviewed journals, and about 13 percent have obtained patents. One quarter of the supported project teams or technologies have received follow-on funding for continued work. All of these are positive indicators for technologies on a trajectory toward commercialized products. In fact, several are either already commercially available or poised to enter the commercial market.
Two important observations can be derived from these facts. First, there is an inherent tension to be managed by ARPA-E between having a short-term impact on a technology within the 3-year funding timeframe while producing transformational technologies. Second, after 6 years of operation, there exist only about 3 years of completed projects to serve as evidence of progress toward ARPA-E’s mission and goals. These two observations speak simultaneously to the need to consider ARPA-E’s impact in context and over a duration that is well aligned with the agency’s mission and the reality of the market dynamics of energy technologies, and to the need to gather, systematically, more and better data that can be used to discern and monitor mechanisms that may lead to a better understanding of how a technology’s full impact is achieved over time.
Reviewing the findings presented above, together with all the findings and evidence gathered and presented for this report, it is evident that assessing ARPA-E at this time is a difficult task. ARPA-E’s was expressly created to achieve long-term environmental, security, and competitiveness goals. It was structured to fund and manage research and development (R&D) undertaken by entities other than the agency rather than undertaking its own R&D activities. Because the agency is tasked with seeking out transformational technological advances, it has necessarily utilized novel operational benchmarks to try to accomplish its goals.
Any assessment of the agency at this time will encounter a well-known problem in R&D management: since sufficient time has not passed for outcomes to have become evident, an assessment cannot draw strong conclusions unless the enterprise is in an extreme situation, such as doing very badly. The findings make clear that ARPA-E is not in an extreme situation. The agency is not failing and is not in need of reform. In fact, attempts to reform the agency—such as applying pressure for ARPA-E to show short-term successes rather than focusing on its long-term mission and goals—would pose a significant risk of harming its efforts and chances of achieving its mission and goals.
Nonetheless, the committee is confident that the data obtained and analyzed for this assessment indicate that ARPA-E has grown from a concept into a functioning organization and has made demonstrable progress toward achieving its statutory mandates. Moreover, the committee hopes that this report will provide useful guidance to ARPA-E as it continually assesses its data collection procedures with an eye toward improving operations and supporting future self- and independent evaluations of the agency.
On the basis of its findings, the committee formulated 14 recommendations intended to help ARPA-E continue to strengthen and build upon its early success. Five of these 14 recommendations stand out as key to positioning the agency for success to fulfill its mission and goals, and are presented here with their supporting findings.
Recommendation 3-1: ARPA-E should preserve its distinctive and flexible management approach that empowers program directors and stresses active project management.
Findings 3-1, 3-2, and 3-3 show how ARPA-E has internalized the principles of an innovative culture, dynamic leadership, and program director autonomy in its organizational structure. Specifically, it is evident that ARPA-E’s program directors have been empowered to take risks in project selection in line with the agency’s mission; have been given discretion that enables ARPA-E to fund relatively risky projects, with no clear indication that average project performance in the short term is reduced; and continuously engage in active management of ongoing projects, as reflected in the altering of project milestones, budgets, and timelines. These findings highlight the important role of the program directors in supporting ARPA-E’s vitality and in enabling the agency to execute its mission and goals.
The committee recommends that ARPA-E strive to preserve this management approach that gives its program directors wide authority to develop new focused technology programs with potential to be transformative and enables them to manage projects actively through technical research guidance and feedback, regular and frequent assessments of progress toward stated milestones, and revision of milestones in response to new findings and research discoveries. This management approach is a defining organizational feature that can contribute to the agency’s ability to achieve its statutory mission and goals, and helps to distinguish ARPA-E from other public funding initiatives for energy R&D.
Recommendation 4-8: The ARPA-E director and program directors should develop and implement a framework for measuring and assessing the agency’s impact in achieving its mission and goals.
As described in Finding 4-9, ARPA-E is not yet able to assess the full extent to which it has achieved its statutory mission and goals. The agency has in place an extensive data gathering and recordkeeping system at the project level with which to track and monitor internal metrics and facilitate active
program management. It has a less extensive system for collecting, tracking, and reporting publicly available high-level innovation metrics such as publications; funding from other sources; and intellectual property information, including disclosures and patents over time. Even if these traditional innovation metrics were available through a more systematic approach to their collection, they could not enable a robust, quantifiable assessment of whether and how ARPA-E’s activities have contributed to achieving its statutory mission and goals. Consequently, neither the agency nor any other assessor can at present perform such an assessment. ARPA-E is, however, in a good position to develop a framework for prospectively mapping project-level data from program creation, through project selection and management, to mission success and achievement of goals.
The development and implementation of such a framework would be very valuable and important for ARPA-E to undertake as soon as practicable, providing the agency with greater ability to demonstrate its value and impact. It is critical that ARPA-E not delay implementation. The longer the agency waits, the more difficult it will be to implement such a framework and the less valuable it will be, and it will become more difficult if not impossible to assess program impacts in a way that allows for meaningful reform in response. The agency could link data from its robust internal database of project-level metrics to program-level goals, including indicators of commercial and noncommercial outcomes over the short and long terms; connect those goals to standard, observable innovation metrics; and then translate those metrics into progress toward achieving the agency-level mission and goals. Such a framework would need to include a system for tracking performers postfunding for at least 10 years, and very likely longer, to capture technologies that are transferred in arms-length transactions along with other ways of observing technology deployment.
At the agency level, ARPA-E already is known for a willingness to assess its structure and operations and experiment with changes aimed at improving both operations and outcomes. Designing and implementing such a framework could place the agency at the forefront of self-evaluation, with the ultimate aim of improving the outcomes of its work. To develop and implement this framework in a way that would best serve the agency, ARPA-E’s director and staff would need to be empowered with the autonomy to do so based on their direct experience with running the agency and managing projects.
Recommendation 3-3: ARPA-E should reconceptualize its “tech-to-market” program to account for the wide variation in support needed across programs and performers with respect to prospective funding, commercialization, and deployment pathways.
Finding 3-4 describes how ARPA-E views its “tech-to-market” (T2M) activities as an ongoing experiment, and the challenge of developing such a
program may be greater than originally thought. Incumbent energy technologies have long usable life spans. Adequately verifying and validating new energy technologies usually takes decades, and large amounts of capital are required relative to what is necessary in other technology sectors. Still more time is required to develop technologies into commercial products. The roughly 3-year timeframe of an ARPA-E project is too short to expect a technology to move from concept to market.
The value added by ARPA-E’s T2M activities varies by project and performer. Some performers consulted during this study, in particular those with established product development and marketing capacity, have not found ARPA-E’s current approach for T2M helpful. Other performers, such as academic teams, have found value in the agency’s T2M guidance. Given that the agency continually strives to evolve and improve its approach to T2M, the committee encourages further evolution of that approach while cautioning against overexpansion. For example, ARPA-E should consider making full T2M plans optional—encouraging development of these plans by performers most likely to benefit, such as academics—but requiring performers to describe potential product applications if they can prove technological feasibility. It also could provide information or research to performers on critical nontechnical factors that could impact market adoption of future products, such as regulatory risk and other, common risks other than business-market risks.
Recommendation 4-3: ARPA-E should continue to use processes designed to identify and support unexplored opportunities that hold promise for resulting in transformational technological advances.
Finding 4-4 describes the importance for ARPA-E of seeking high-risk, potentially transformative technologies and overlooked, “off-roadmap” opportunities pursued by neither private firms nor other funding agencies, including other programs and offices within DOE, as a way to position itself to accomplish its mission. ARPA-E’s underlying organizational features include encouraging its program directors to seek potentially high-impact projects and recognizing that many of its projects will produce only valuable knowledge, including knowledge of research pathways that should not be pursued further, instead of commercialized products.
Maintaining this focus will be one of the greatest challenges for ARPA-E in the future. It is not guaranteed that ARPA-E will be able to maintain a culture of pursuing high-risk but potentially transformative technologies and research pathways characterized as novel or significantly underexplored as the energy technology landscape evolves. ARPA-E leadership and the secretary of energy should actively work to sustain this culture. ARPA-E should continue to balance its overall portfolio between technologies that appear to have the potential to be transformative and other valuable opportunities that are being ignored.
Recommendation 3-5: The secretary of energy should ensure that other offices and programs within DOE continue to explore and adopt elements of ARPA-E’s practices that can improve the department’s operations.
Findings 3-1, 3-2, and 3-3 provide details on ARPA-E’s program creation and project management. Finding 3-6 supports the positive influence those practices have had on other offices within DOE. To cite a direct example, the Office of Energy Efficiency and Renewable Energy has incorporated several elements of ARPA-E’s approach into the management of its programs, including use of a workshop to define a program, use of concept papers to screen funding applicants, and early termination of underperforming projects. While some elements of ARPA-E’s approach may be difficult to scale or translate to other programs and offices, there is great benefit in exploring their adaptability and suitability. Of particularly high value would be finding suitable ways to incorporate such key features as term-limited program managers, use of constructive engagement among program directors to sharpen the focus of programs, the degree of operational freedom accorded to program managers, and the risk-taking orientation of programs. Other DOE offices have expressed interest in adopting a number of these features. The secretary of energy should encourage and empower those offices to explore and adopt appropriate practices.
ARPA-E has the ability to make significant contributions to energy R&D that likely would not take place absent the agency’s activities. The committee believes that implementation of its recommendations would benefit ARPA-E, and the nation, as the agency continues to evolve and improve its operations in service of its mission and goals. The committee also believes that these recommendations should be helpful to Congress as it considers ARPA-E’s future.