1

Introduction

When the U.S. Congress authorized creation of the Advanced Research Projects Agency-Energy (ARPA-E) within the U.S. Department of Energy (DOE), it mandated that the secretary of energy ask the National Academy of Sciences to assess how well the agency was achieving its statutory mission and goals. ARPA-E’s authorizing statute states that its mission is “to overcome long-term and high-risk technology barriers in the development of energy technologies,”¹ and its goals are

1. to enhance the economic and energy security of the United States through the development of energy technologies that result in—
   1. reductions of imports of energy from foreign sources;
   2. reductions of energy-related emissions, including greenhouse gases; and
   3. improvement in the energy efficiency of all economic sectors; and
2. to ensure that the United States maintains a technological lead in developing and deploying advanced energy technologies.”²

ARPA-E began operating in 2009, and as requested by the Congress, the National Academy of Sciences convened an ad hoc study committee to undertake this assessment starting in 2015, after 6 years of agency operations. The purpose of this study was to provide the information requested by the Congress as input to its consideration of the future of ARPA-E, to summarize

___________________

¹ 42 U.S.C. § 16538(b). This subsection is labeled “ESTABLISHMENT,” but its second clause appears to state Congress’s intended mission for ARPA-E.

² 42 U.S.C. § 16538(c).

lessons learned that may apply to the operations of other DOE programs,³ and to provide ARPA-E leadership with information it can utilize as the agency continues to evolve and seeks to improve its efficacy and impact.

STATEMENT OF TASK

ARPA-E was originally conceived in a recommendation from a previous study of the National Academies (NAS et al., 2007). The statutory language mandating this assessment requested a review of the agency’s methods, its success in fostering transformational innovation, and the adequacy of its resources. Those topics are reflected in the statement of task for this study (see Box 1-1).

___________________

³ 42 U.S.C. § 165381(l).

STUDY SCOPE

To operationalize the statement of task for this study, the ad hoc committee posed three broad questions that this report is intended to answer:

• What qualitative and quantitative standards are appropriate for assessing ARPA-E’s performance?
• To what extent are ARPA-E’s organization, operational methods, and personnel practices positioning it to accomplish its mission and goals as established by the Congress?
• What impacts has ARPA-E had on its awardees and on the nation’s energy technology innovation system?

In attempting to answer these questions, the committee had to take several caveats into account:

• ARPA-E’s performance has been affected by external factors, such as the length of time that elapsed while awaiting Senate confirmation of both its first and second directors.
• The 6 years of ARPA-E operations provides insufficient data to enable a full assessment of potentially transformational research whose impact likely will not be fully observable until years or decades in the future.
• A number of important terms in the statement of task—such as “cutting-edge inventions,” “revolutionary advances,” “qualified key personnel,” and “transformational technological advances”—are intrinsically subjective, difficult to define, and even more difficult to measure.

STUDY APPROACH

ARPA-E’s authorizing legislation is not specific as to how or to what standards the agency’s operations and performance were to be measured. Accordingly, the committee elected to apply metrics traditionally used in the study of innovation in attempting to measure intermediate outcomes of the work completed under projects funded by ARPA-E.

The committee approached its task by utilizing both quantitative and qualitative methods. To carry out the assessment, it formed four subgroups with a number of committee members and National Academies staff participating in more than one subgroup. Some data gathering and processing was performed by independent consultants. The Internal Operations Qualitative Data Team collected and analyzed qualitative data, including information gathered during interviews with current and former ARPA-E personnel and DOE officials and discussions with stakeholders at agency events, such as program kickoffs and

interim meetings. The Case Study Team conducted three types of case studies (a focused program, a portfolio of energy storage projects, and individual projects). The Internal Operations Quantitative Data Team accessed internal, proprietary ARPA-E data and reported aggregated findings. Finally, the External Quantitative Data Team obtained data from external sources, such as the U.S. Patent and Trademark Office’s (USPTO) patent database and publications. (See Appendix C for more details on the data and methods used in this report.) Using the statement of task as the point of departure, the subgroups considered the evidence that became available through their research activities. Their deliberations on this evidence led to provisional findings that were summarized and presented to the full committee, which discussed, debated, and compared them. This process went through a number of iterations, leading to final findings and recommendations based on those findings.

The first ARPA-E awards were made in October 2009 for projects of 3 years’ duration, meaning that any external impacts from these first-generation projects after ARPA-E funding had ended would have begun to manifest only during the 3 years preceding the start of this study. By contrast, the development of new energy technologies usually requires decades of effort, and decades also must usually elapse before it becomes evident that innovations are truly transformative. It is likely, therefore, that any external impact of more recently initiated projects, which represent the bulk of ARPA-E’s portfolio, will not be observable for some years. Consequently, the committee’s findings are best regarded as early assessments based on the evidence available over a relatively short timeframe. While more data and more time for analysis may result in more understanding, the committee is confident that the findings and recommendations presented in this report are more than sufficient for use by the Congress in determining ARPA-E’s future, and will be of value to ARPA-E as it continues to evolve and improve its operations.

ORGANIZATION OF THE REPORT

Chapter 2 of this report presents an overview of ARPA-E’s genesis, structure, and operations; its congressional mandate and funding; and the results of previous agency audits and oversight initiatives. Chapter 3 provides details of the committee’s operational assessment of ARPA-E’s performance as an organization, its operational methods, and its personnel practices. Chapter 4 presents the committee’s technical assessment of the extent to which the projects ARPA-E has funded have contributed to the agency’s achievement of its statutory mission and goals, to the extent this can be ascertained from the relatively brief experience base accumulated since the agency began its operations. The report also includes six appendixes: Appendix A provides biographical information on the committee members; Appendix B provides a disclosure of conflict of interest of a committee member; Appendix C

summarizes the various methodologies used in this assessment; Appendix D contains the full text of the 10 case studies developed as input for this assessment; Appendix E provides a glossary of acronyms and abbreviations used in this report; Appendix F contains the committee’s formal request for data made to ARPA-E and the agency’s official response; and Appendix G includes the white papers prepared by external consultants detailing their data analyses.