An Assessment of
ARPA-E
Committee on Evaluation of the Advanced Research Projects Agency-Energy
(ARPA-E)
Board on Science, Technology, and Economic Policy
Policy and Global Affairs
Board on Energy and Environmental Systems
Division on Engineering and Physical Sciences
Pradeep K. Khosla and Paul T. Beaton, Editors
A Consensus Study Report of
THE NATIONAL ACADEMIES PRESS
Washington, DC
www.nap.edu
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, N.W. Washington, DC 20001
This activity was supported by Contract No. DE-PI0000010, Order Number DE-DT0008147, with the U.S. Department of Energy, and Grant No. G-2014-13809 from the Alfred P. Sloan Foundation. Any opinions, findings, conclusions, or recommendations expressed in this publication do not necessarily reflect the views of any organization or agency that provided support for the project.
ISBN-13: 978-0-309-45945-7
ISBN-10: 0-309-45945-1
Digital Object Identifier: https://doi.org/10.17226/24778
Additional copies of this publication are available for sale from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.
Copyright 2017 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
Suggested citation: National Academies of Sciences, Engineering, and Medicine. 2017. An Assessment of ARPA-E. Washington, DC: The National Academies Press. doi: https://doi.org/10.17226/24778.
The National Academy of Sciences was established in 1863 by an Act of Congress, signed by President Lincoln, as a private, nongovernmental institution to advise the nation on issues related to science and technology. Members are elected by their peers for outstanding contributions to research. Dr. Marcia McNutt is president.
The National Academy of Engineering was established in 1964 under the charter of the National Academy of Sciences to bring the practices of engineering to advising the nation. Members are elected by their peers for extraordinary contributions to engineering. Dr. C. D. Mote, Jr., is president.
The National Academy of Medicine (formerly the Institute of Medicine) was established in 1970 under the charter of the National Academy of Sciences to advise the nation on medical and health issues. Members are elected by their peers for distinguished contributions to medicine and health. Dr. Victor J. Dzau is president.
The three Academies work together as the National Academies of Sciences, Engineering, and Medicine to provide independent, objective analysis and advice to the nation and conduct other activities to solve complex problems and inform public policy decisions. The National Academies also encourage education and research, recognize outstanding contributions to knowledge, and increase public understanding in matters of science, engineering, and medicine.
Learn more about the National Academies of Sciences, Engineering, and Medicine at www.nationalacademies.org.
Consensus Study Reports published by the National Academies of Sciences, Engineering, and Medicine document the evidence-based consensus on the study’s statement of task by an authoring committee of experts. Reports typically include findings, conclusions, and recommendations based on information gathered by the committee and the committee’s deliberations. Each report has been subjected to a rigorous and independent peer-review process and it represents the position of the National Academies on the statement of task.
Proceedings published by the National Academies of Sciences, Engineering, and Medicine chronicle the presentations and discussions at a workshop, symposium, or other event convened by the National Academies. The statements and opinions contained in proceedings are those of the participants and are not endorsed by other participants, the planning committee, or the National Academies.
For information about other products and activities of the National Academies, please visit www.nationalacademies.org/about/whatwedo.
COMMITTEE ON EVALUATION OF THE ADVANCED RESEARCH PROJECTS AGENCY-ENERGY (ARPA-E)
PRADEEP K. KHOSLA (NAE), Chancellor, University of California, San Diego, Chair
MAXINE SAVITZ (NAE), General Manager (ret.), Honeywell, Inc. (Vice Chair for Technical Evaluation)
PIERRE AZOULAY, International Programs Professor of Management, Massachusetts Institute of Technology
TERRY BOSTON (NAE), Director, GridLiance GP, LLC and Grid Protection Alliance
ERICA R. H. FUCHS, Professor, Engineering and Public Policy, Carnegie Mellon University
SUPRATIK GUHA (NAE), Director, Center for Nanoscale Materials, Argonne National Laboratory; Professor, Institute for Molecular Engineering, University of Chicago
MARK JONES, Executive External Strategy and Communication Fellow, The Dow Chemical Company
ERIC LANDREE, Associate Research Department Director, Engineering and Applied Sciences Department, and Senior Engineer, RAND Corporation
GILBERT E. METCALF, Professor of Economics, Tufts University
JOHN PLASTER, Managing Director, Global Power and Utilities Group, and Head of Alternative Energy, Barclays Capital
LOUIS SCHICK, Founding Partner and CTO, NewWorld Capital Group
CHARLES SHANK (NAS/NAE), Senior Fellow, Howard Hughes Medical Institute, and Professor Emeritus of Chemistry, Physics, Electrical Engineering, and Computer Sciences, University of California, Berkeley
STEPHANIE S. SHIPP, Deputy Director and Research Professor, Social and Decision Analytics Laboratory (SDAL), Biocomplexity Institute of Virginia Tech
SCOTT STERN, David Sarnoff Professor of Management, Massachusetts Institute of Technology
JOHN WALL (NAE), Vice President and CTO (ret.), Cummins, Inc.
JAY WHITACRE, Professor, Engineering and Public Policy and Materials Science and Engineering, Carnegie Mellon University (Member 5/8/2015–9/20/2016)
VALERIE WILLIAMS, Senior Program Evaluator, University Corporation for Atmospheric Research
PEIDONG YANG (NAS), S. K. and Angela Chan Distinguished Chair Professor in Energy and Professor of Chemistry, University of California, Berkeley
___________________
NOTE: See Appendix B, Disclosure of Conflict of Interest.
PROJECT STAFF
BOARD ON SCIENCE, TECHNOLOGY, AND ECONOMIC POLICY
POLICY AND GLOBAL AFFAIRS
PAUL BEATON, Study Director
GAIL COHEN, Board Director
JEFF ALSTOTT, Christine Mirzayan Science and Technology Policy Graduate Fellow
DAVID ALLEN AMMERMAN, Financial Officer (until June 2017)
DAVID DIERKSHEIDE, Program Officer
CHRISTOPHER J. JONES, Christine Mirzayan Science and Technology Policy Graduate Fellow
FREDERIC LESTINA, Senior Program Assistant
ERIK SAARI, Senior Program Assistant
BOARD ON ENERGY AND ENVIRONMENTAL SYSTEMS
DIVISION ON ENGINEERING AND PHYSICAL SCIENCES
K. JOHN HOLMES, Study Co-Director
ELIZABETH EULLER, Senior Program Assistant
BOARD ON SCIENCE, TECHNOLOGY, AND ECONOMIC POLICY
For the National Academies of Sciences, Engineering, and Medicine, this project was overseen by the Board on Science, Technology, and Economic Policy (STEP), a standing board established by the National Academies of Sciences and Engineering and the Institute of Medicine in 1991. The mandate of the STEP Board is to advise federal, state, and local governments and inform the public about economic and related public policies to promote the creation, diffusion, and application of new scientific and technical knowledge to enhance the productivity and competitiveness of the U.S. economy and foster economic prosperity for all Americans. The STEP Board and its committees marshal research and the expertise of scholars, industrial managers, investors, and former public officials in a wide range of policy areas that affect the speed and direction of scientific and technological changes and their contributions to the growth of the U.S. and global economies. Results are communicated through reports, conferences, workshops, briefings, and electronic media subject to the procedures of the National Academies to ensure their authoritativeness, independence, and objectivity. The members of the STEP Board and staff are listed below:
RICHARD K. LESTER, Massachusetts Institute of Technology, Chair
JEFF BINGAMAN, Former U.S. Senator, New Mexico
ELLEN R. DULBERGER, Dulberger Enterprises, LLC
ALAN M. GARBER (NAM), Harvard University
RALPH E. GOMORY (NAS/NAE), New York University
MICHAEL GREENSTONE, The University of Chicago
ARATI PRABHAKAR (NAE)
LUIS M. PROENZA, University of Akron
KATHRYN L. SHAW, Stanford University
Staff
GAIL COHEN, Board Director
PAUL BEATON, Senior Program Officer and Study Director
DAVID DIERKSHEIDE, Program Officer
FREDERIC LESTINA, Senior Program Assistant
ERIK SAARI, Senior Program Assistant
SUJAI SHIVAKUMAR, Senior Program Officer
BOARD ON ENERGY AND ENVIRONMENTAL SYSTEMS
The Board on Energy and Environmental Systems (BEES) is a unit of the Division on Engineering and Physical Sciences (DEPS) of the National Academies of Sciences, Engineering, and Medicine. Since 1975, the Board (formerly the Energy Engineering Board [EEB]) has conducted a diverse program of studies and related activities (workshops, symposia, etc.) to produce authoritative, independent recommendations about the science and technology aspects of public policy questions in energy, the environment, national security, and defense.
JARED COHON (NAE), Carnegie Mellon University, Chair
DAVID ALLEN (NAE), University of Texas, Austin
TERRY BOSTON (NAE), GridLiance GP, LLC and Grid Protection Alliance
WILLIAM BRINKMAN (NAS), Princeton University, Princeton, New Jersey
EMILY A. CARTER (NAS/NAE), Princeton University, Princeton, New Jersey
BARBARA KATES-GARNICK, Tufts University, Boston, Massachusetts
JOANN MILLIKEN, Independent Consultant, Alexandria, Virginia
MARGO OGE, Office of Transportation and Air Quality, Environmental Protection Agency, McLean, Virginia
JACKALYNE PFANNENSTIEL,∗ Independent Consultant, Piedmont, California
MICHAEL RAMAGE (NAE), ExxonMobil Research and Engineering Company (retired), New York City
DOROTHY ROBYN, Consultant, Washington, D.C.
GARY ROGERS, Roush Industries, Livonia, Michigan
KELLY SIMS-GALLAGHER, The Fletcher School, Tufts University, Medford, Massachusetts
MARK THIEMENS (NAS), University of California, San Diego
JOHN WALL (NAE), Cummins, Inc. (retired), Belvedere, California
ROBERT WEISENMILLER, California Energy Commission, Sacramento, California
Staff
K. JOHN HOLMES, Acting Director/Scholar
JAMES ZUCCHETTO, Senior Scientist
DANA CAINES, Financial Associate
LANITA JONES, Administrative Coordinator
MARTIN OFFUTT, Senior Program Officer
JANKI PATEL, Program Assistant
___________________
∗ Deceased on April 26, 2017.
Preface
In 2005, the National Research Council (NRC) report Rising Above the Gathering Storm recommended a new way for the federal government to spur technological breakthroughs in the energy sector. It recommended the creation of a new agency, the Advanced Research Projects Agency-Energy, or ARPA-E, as an adaptation of the Defense Advanced Research Projects Agency (DARPA) model—widely considered a successful experiment that has funded out-of-the-box, transformative research and engineering that made possible the Internet, GPS, and stealth aircraft. This new agency was envisioned as a means of tackling the nation’s energy challenges in way that could translate basic research into technological breakthroughs while also addressing economic, environmental, and security issues. It was unclear ex ante whether such an adaptation would work, whether an ARPA-E engineered from the ground up to be lean, agile, and independent would be effective at catalyzing the transformation of energy technologies as DARPA had been with other technologies. Accordingly, Rising Above the Gathering Storm also recommended that ARPA-E be independently reviewed after some years of operation. When Congress authorized ARPA-E in the 2007 America COMPETES Act, it followed the recommendations both to create the agency and to request an early assessment following 6 years of operation to examine the agency’s progress toward achieving its statutory mission and goals. This report documents the results of that assessment.
Now 7 years into its operations, ARPA-E is demonstrably built on the DARPA model while differing in certain respects. People form the core of the agency. Program directors are hired for limited terms; highly empowered to act outside of the box when designing new programs; expected to search for, identify, and support high-risk but potentially high-impact projects; and substantially involved in managing the research and technical aspects of funded projects. ARPA-E is characterized by institutional independence and a flat organizational structure, and it can quickly initiate and terminate projects based on performance. The agency has tailored the DARPA model to include in-house contracting so that funding decisions can be made much more quickly relative to most government funding agencies. It also has developed a continuously
evolving system intended to aid project teams in preparing for the eventual development of their technologies into commercial products.
The assessment documented in this report is based on currently available data and analytical methods, and should be useful to ARPA-E as it continues to evolve and improve its operations and programs. In addition, the report provides valuable information for Congress as input to its decisions regarding ARPA-E’s future, and for scholars and stakeholders within the energy technology innovation system who study or interact with ARPA-E and other participants in that system. Independent reviews can provide reliable, external indicators of the performance of government programs or agencies. A number of the analyses conducted for this assessment take advantage of analytical methods developed only recently. As increases in computing power and the unlimited ingenuity of the human mind continue to expand the frontiers of analytical methodology and thus the value of assessments, this report can serve as a vision of what is possible, as well as a glimpse of how much more would be possible with additional data, time, and other resources.
Since 1991, the NRC, under the auspices of the Board on Science, Technology, and Economic Policy (STEP), has undertaken a program of activities designed to improve policy makers’ understanding of the interconnections among science, technology, and economic policy and their importance for the American economy and its international competitive position. STEP’s activities have corresponded with increased policy recognition of the importance of knowledge and technology to economic growth. New economic growth theory emphasizes the role of technology creation, which is believed to be characterized by significant growth externalities. Likewise, under the auspices of the Board on Energy and Environmental Systems (BEES), the NRC has undertaken a program of studies and other activities to provide independent advice to the executive and legislative branches of government and the private sector on issues in energy and environmental technology and related public policy. BEES directs expert attention to issues surrounding energy supply and demand technologies and systems, including resource extraction through mining and drilling; energy conversion, distribution and delivery, and efficiency of use; environmental consequences of energy-related activities; environmental systems and controls in areas associated with the production, energy conversion, transmission, and use of fuels; and related issues in national security and defense.
A central focus of NRC analysis has been the importance of energy innovation to the growth of the U.S. economy and to the reduction of negative environmental, public health, and other consequences of energy-related activities. Many performance gains in energy technologies remain to be achieved, such as the capture of carbon from the use of fossil fuels, advanced nuclear power from fission and fusion, renewable fuels for electricity generation and for vehicles, and increasingly efficient use of energy. Yet undertaking the efforts required to produce the innovations needed to transform the performance of the energy sector so as to mitigate the risks from energy production,
conversion, and consumption may be the greatest challenge humanity has ever faced. It is a worldwide challenge demanding tremendous effort and leadership. Throughout history, the United States has consistently demonstrated that its greatest resource is its people and their talent for innovation and leadership. There has never been a greater need or opportunity for American leadership than that posed by the challenge of achieving dramatic innovations in energy technology, a challenge that is the subject of this report.
ACKNOWLEDGMENTS
On behalf of the National Academies of Sciences, Engineering, and Medicine, the committee expresses its appreciation for and recognition of the insights, information, experiences, and perspectives provided by the many participants in workshops and roundtables held while this study. We would particularly like to recognize Anna Goldstein, Tom Howell, and Michael Kearney for their invaluable research and technical assistance in the preparation of this report. We also thank Frederic Lestina, Erik Saari, Alisa Decatur, and Rona Briere for their assistance in preparing this report for publication.
Acknowledgment of Reviewers
This Consensus Study Report was reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise. The purpose of this independent review is to provide candid and critical comments that will assist the National Academies of Sciences, Engineering, and Medicine in making each published report as sound as possible and to ensure that it meets the institutional standards for quality, objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process.
We wish to thank the following individuals for their review of this report: Gary Anderson, National Institute of Standards and Technology; Ewa Bardasz, Zual Associates in Lubrication LLC; Harvey Blanch, University of California, Riverside; William Brinkman, Princeton University (retired); Douglas Cameron, First Green Partners; Maryann Feldman, University of North Carolina; Shane Greenstein, Harvard University; Douglas Kirkpatrick, InnerProduct Partners; Gerald D. Mahan, Pennsylvania State University; Ramana Nanda, Harvard University; and David Popp, Syracuse University.
Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations of this report nor did they see the final draft before its release. The review of this report was overseen by Robert Frosch, Harvard University (retired), and Michael Ladisch, Purdue University. They were responsible for making certain that an independent examination of this report was carried out in
accordance with the standards of the National Academies and that all review comments were carefully considered. Responsibility for the final content rests entirely with the authoring committee and the National Academies.
Pradeep Khosla | Paul Beaton |
Committee Chair | Study Director |
Contents
3 ARPA-E’s INTERNAL OPERATIONS: CULTURE, PEOPLE, AND PROCESSES
Defining Organizational Features of ARPA-E
ARPA-E’s Culture, People, and Processes: A Qualitative and Quantitative Analysis
Summary of Findings and Recommendations
ARPA-E Technical Objectives: Transformational Technologies and White Space
External Metrics Used to Assess Outcomes
Evidence of Impact from Case Studies
Transforming Energy Industry Attitudes
Additional Metrics That May Be Used for Technology Assessments
Summary of Findings and Recommendations
A COMMITTEE MEMBER BIOGRAPHIES
B DISCLOSURE OF CONFLICT OF INTEREST
C METHODS USED TO ASSESS ARPA-E
D CASE STUDIES USED TO ASSESS ARPA-E’S OPERATIONS AND POTENTIAL TO ACHIEVE ENERGY IMPACTS
Boxes, Tables, and Figures
BOXES
S-1 Complete List of All Findings
S-2 Complete List of All Recommendations
2-1 Questions That Must Be Answered for New Projects
3-1 Creating the Counterfactual Project Selection System
4-1 Description of Metrics for Technology Assessment of ARPA-E Used in This Report
4-2 Summary of External Metrics Analysis
C-1 Case Study Questions for ARPA-E Performers
TABLES
2-2 Example of Focused Program Technical Targets from the MOSAIC Program Announced in 2014
2-3 ARPA-E Programs as of October 2016
3-1 Transition Matrix for Overall Status
C-2 Methods and Data Sources for Questions under Statement of Task Charge 1
C-3 Methods and Data Sources for Statement of Task Charge 6
C-4 Methods and Data Sources for Statement of Task Charge 9
C-5 Consultations Conducted for ARPA-E Assessment
C-6 Categorization of Questions Used for Individual Consultations
C-7 ARPA-E Events Attended by Committee Members
D-1 Projects Funded under the SWITCHES Program
D-2 Funding for Electricity Storage Projects, 2009−2014
D-3 Impact Analysis of the 63 Electricity Storage Projects Funded by ARPA-E from 2009 to 2014
D-4 ARPA-E Individual Project Case Studies
FIGURES
2-2 Organizational chart of ARPA-E in 2016
2-3 ARPA-E’s depiction of how it has built itself on a foundation modeled after DARPA
2-4 Schematic of ARPA-E’s “white space” strategy
3-1 Career pathways of ARPA-E program directors as of May 2017
3-2 Mean overall scores of applications
3-3 Percent of concept paper applications that result in selection for award negotiation
3-4 Selection of full applications by focused and OPEN programs
3-7 Frequency of milestone creations and deletions
3-8 Frequency of program director changes across projects
3-9 Technology readiness level of projects funded by ARPA-E
4-1 Plot of performance per cost from 1910 to 2010 for common information storage technologies
4-4 Focus of ARPA-E electricity storage projects funded 2009-2014
4-5 Degree of novelty and prior support for ARPA-E electricity storage projects
4-6 Funding history for 1366 Technologies
4-8 Funding history for Smart Wires
C-1 Relationship between statement of task charges and themes that arose during consultations
C-2 Statement of task charges and topics covered by case studies
D-3 Electricity storage projects by status, 2009–2014
D-5 Focus of the ARPA-E electricity storage projects funded 2009- 2014
D-6 Performer collaboration types of electricity storage projects funded by ARPA-E, 2009–2014
D-7 Degree of novelty and prior support for ARPA-E electricity storage projects
D-8 Funding history for 1366 Technologies
D-9 1366 Technologies’ growth as a company
D-11 Schematic of 24M Li-ion battery cell versus conventional Li-ion cell
D-12 Funding history for SLIPS
D-14 Funding history for Smart Wires
D-15 Funding history for Agrivida
D-16 Funding history for Ceres
D-18 Funding history for Stanford University radiative coolers