The statement of task for this study (Box 1-1 in Chapter 1) directs the committee to conduct a retrospective assessment and technical evaluation of ARPA-E, including the most significant accomplishments and impacts of the ARPA-E program and any unique features of the program that may have contributed to these accomplishments. In undertaking this assessment, the committee considered the stated goals of ARPA-E, how those goals might be measured, and the impacts of the agency’s projects and programs. There are clear indicators that ARPA-E is making progress toward achieving its statutory mission and goals, and it cannot reasonably be expected to have completely fulfilled those goals given so few years of operation and the size of its budget.
With respect to ARPA-E’s stated goals, this chapter begins by explaining two terms that are key to those goals—transformational technologies and white space. The chapter then describes the external metrics used to assess impacts of ARPA-E programs and projects. Next, the chapter presents the results of the committee’s technology assessment: evidence of the impact of ARPA-E programs and projects based on three key external metrics—scientific publications, patents, and market engagement; evidence of impact from the case studies conducted to support the technology assessments (see Appendix D); and the agency’s impact on transforming energy industry attitudes, creating new communities of researchers, and improving public awareness of its achievements. As with its approach to the operational assessment reported in Chapter 3, the committee used multiple methods for this technology assessment, which are described in detail in Appendix C. The chapter ends with a summary of the findings resulting from the committee’s technology assessment and its recommendations for moving forward.
ARPA-E TECHNICAL OBJECTIVES: TRANSFORMATIONAL TECHNOLOGIES AND WHITE SPACE
Chapter 1 describes how ARPA-E was created to turn revolutionary advances and out-of-the-box ideas into transformational energy technologies to reduce energy imports and emissions, improve efficiency, and ensure that the United States maintains a technological lead in advanced energy technologies. The agency also describes itself as looking to fund projects that fill “white space”—energy technologies not being fully addressed by the private sector or other federal research programs (ARPA-E, 2013b; Rohlfing, 2015). For this study, transformational technologies and white space were defined as described below.
The congressional authorization of ARPA-E includes a mandate that the agency accelerate transformational technological advances but provides no guidance on the meaning of the term. ARPA-E describes its own mission as to “catalyze and accelerate the creation of transformational energy technologies” through high-risk, potentially high-reward investments, and often uses the term “transformational” in its program descriptions (ARPA-E, 2014c).
ARPA-E does not offer a specific, concise definition of the word transformational, but it does state that it seeks to fund research with the potential to identify new technology learning curves with potential to lead to technologies with performance/cost ratios significantly greater than incumbent technologies (ARPA-E, 2013d). For any technology, energy or otherwise, whether or not one of these curves will produce truly transformational technologies, in most cases, can be recognized only in retrospect,1 after the specific technology has matured
1 There are exceptions in which the potential for technologies to be transformational was recognized in advance. The first p-n junction solar cell, discovered circa 1954, was hailed at that time by The New York Times (1954, April 26), which states that the silicon solar cell “may mark the beginning of a new era, leading eventually to the realization of one of mankind’s most cherished dreams—the harnessing of the almost limitless energy of the sun for the uses of civilization.” A second example is the switch from bipolar junction technology (BJT) to metal-oxide-semiconductor field-effect transistors (MOSFETs) for high-end computing. Today all servers are MOSFET based, whereas prior to 1993, every powerful mainframe was BJT based. The IBM CEO at the time, Lou Gerstner, bet on the projected progress in MOSFETs predicted by Moore’s Law for this transformation to put IBM ahead of its competitors, such as Hitachi. A third example is the search for blue light emitters. Researchers in the early 1990s knew that an efficient semiconductor light emitter or laser in the blue would be transformational. GaN turned out to be that material, and it revolutionized lighting. In all instances, though, several decades passed before the technologies’ full impact could be observed, and such impact has been seen only in retrospect.
enough to disrupt its market by being more widely adopted than incumbent technologies, or by creating an entirely new market. Often the specific details—cost, performance, application, size and scope of market, specific impact on society—look different from what was imagined when the technology is in its early development stages.
Figure 4-1 provides illustrative examples of the unpredictability of technology disruption in terms of the performance/price ratio over time for common information storage technologies. In the mid-1980s, looking at the curve for magnetic tape since 1950, it would not have been immediately clear that its performance to price ratio would overtake the incumbent technology, printed material. Following several decades of slow improvement, the gains appeared to have leveled off before overtaking printed material. By contrast, around the year 1970 magnetic disk technology appeared to be on a fast trajectory to disrupt, having already outperformed magnetic tape on a trend that suggested it would soon overtake printed material, too. For the next 15 years, though, magnetic disk technology made only modest gains and still lagged behind printed materials. Both technologies, however, did eventually outperform printed material by the mid-1990s, after over 40 years of development. Punch cards and optical disks tell different stories of unpredictability. When first introduced, punch cards provided much less storage per dollar than printed materials. They showed some improvement over time, but never caught up to the incumbent technology or the newer technologies that entered the market. Optical disks started at a high performance to price ratio which quickly rose.
Overall, all of the technologies appeared to have disruptive potential when introduced. But only optical disks outperformed printed material when first introduced. For the rest, whether or not they would prove to be transformational only became evident in retrospect after 40 years or longer.
This helps in understanding that ARPA-E’s core philosophy of supporting transformational research is properly viewed as supporting projects that have the potential to be transformational. The project or program that holds the real potential to be transformational must be structured to seek forks in the technology roadmap, such as the multiple technologies shown in Figure 4-1, and must challenge the conventional understanding of what is possible, practical, or profitable in impactful applications. Such a project also can shake up industrial or political establishments. One way ARPA-E assesses the potential for a program or project to be transformational is to evaluate it against a version of the Defense Advanced Research Projects Agency’s (DARPA) Heilmeier questions (Rohlfing, 2016).
For this assessment, then, the committee developed a set of questions to use in determining whether a program or project is potentially transformational:
- Does the program/project hold potential to create “forks” in the technology roadmap that could change the conventional understanding of what is possible, practical, and profitable in applications with large potential impact?
- Does it reach for impacts that may be very far off but that may eventually result in faster learning curves and potentially disruptive entitlements?
- Does it bridge gaps in technological development that might otherwise prevent a particular research pathway from being advanced?
- Does it challenge a community to tackle barriers seen as too risky for rate of return–based investment decision metrics?
- Does it empower a community to challenge conventional wisdom regarding feasibility?
- Does it shake up “group think” and political or industrial establishments?
- Does it set aggressive targets that, if attained, could result in high impact?
- Does it create a stakeholder or technical constituency?
- Does it revisit a failed approach in the context of related supporting advances or changes in economic or other external factors?
- Does it fuse disparate technologies in novel ways?
In developing and reviewing proposed programs, ARPA-E personnel make it clear that the agency seeks to fill white space as it develops and reviews its proposed programs (e.g., Rohlfing, 2015). Funding of white space is intended to address a perceived gap or opportunity in the energy technology landscape. As discussed in Chapter 2, searching for white space for ARPA-E entails pursuing energy technology innovation with two distinct but related objectives: the search for technological approaches that are truly novel or greatly underexplored, and the search to fill gaps left in other research or funding programs. Figure 4-2 shows how the Full-spectrum Optimized Conversion and Utilization of Sunlight (FOCUS) program addresses white space.
While the distinction is not explicitly described by ARPA-E, the committee notes that funding white space technologies may or may not necessarily be the same as funding transformational technologies. This distinction is apparent in reviewing the breadth of ARPA-E’s programs. Clearly, some of ARPA-E’s programs—such as the Strategies for Wide bandgap, Inexpensive Transistors for Controlling High Efficiency Systems (SWITCHES) program for developing advanced semiconductor materials or the Batteries for Electrical Energy Storage in Transportation (BEEST) program for developing dramatically better batteries for plug-in electric vehicles—are aimed at potentially transformational energy technologies. But other programs—such as the Advanced Management and Protection of Energy storage Devices (AMPED) program for improving battery management controls and sensors and Generating Realistic Information for the Development of Distribution and Transmission Algorithms (GRID DATA) program for developing open-access electricity system models, datasets, and data repositories—address a more specific technological challenge. These latter programs are about funding ideas in white space areas that are perceived to be overlooked by other parts of DOE, other agencies in the federal government, and private companies. Such projects may not necessarily be pursuing technologies with high likelihood of being truly transformational but rather filling a critical technological need that could help enable the development of such technologies.
Importance of Balancing Expectations
Given that truly transformational technologies, whether in the energy or some other sector, take many years or decades before becoming apparent and that ARPA-E has existed for only 6 years, the committee would not expect any ARPA-E programs to have had any transformational impacts on the energy sector yet. As discussed in subsequent sections, however, discernable intermediate outcomes provide indications of the potential for success. It is often impossible to gauge what will prove to be transformational; tests or breakthroughs that garner big headlines can end up being underwhelming in the long run, whereas small, incremental tweaks can turn out to enable major shifts in technologies or processes. More fundamentally, it is unclear whether all projects within the ARPA-E portfolio should be targeted to transformational technologies. Rather, keeping in mind the distinction drawn above, funding projects that fill a white space may also produce results consistent with the agency’s objectives and help enable transformational energy technologies.
EXTERNAL METRICS USED TO ASSESS OUTCOMES
The statement of task for this study directs the committee to examine processes, deliverables, and metrics used to assess the short- and long-term success of ARPA-E programs. In her presentation to the committee, then
director of DARPA, Arati Prabhakar, offered a cautionary note that there are “no viable metrics for judging success” for an agency seeking to bring about transformational innovations (Prabhakar, 2015). However, federal funding of energy technologies receives more scrutiny than defense-related funding, and any technical evaluation of ARPA-E must rely to some extent on metrics. While recognizing that all metrics are imperfect, then, the committee used the quantitative and qualitative metrics described in Box 4-1 in conducting its technology assessment of ARPA-E.
point that others pick up and take over. Alternatively, such follow-on funding might be to fund DARPA-like scale demonstrations. This could, however, require decades and multiple performers, depending on the broader context in both the innovation ecosystem and the marketplace. For example, DARPA has been funding alternative materials to silicon complementary metal-oxide-semiconductors (Si-CMOS) for decades because such alternative materials have relevance in certain military contexts, but it is only in the last few years that Si-CMOS’s remaining runway has become sufficiently short to garner industry attention and funding (Khan et al., 2014).
- New firm development: In addition to follow-on funding, the formation of a new start-up company following support by ARPA-E is an important metric of market impact. Like with patents and publications, not every project should have firm formation as its immediate goal. Some projects may lead directly to formation of new firms while others will only plant the seeds for firms to form around the technologies supported well past the project timeline.
NOTE: In addition to these metrics, the technical and other milestones created and adjusted during a project are discussed in Chapter 3.
The success of individual programs and projects cannot be determined by a single metric—different projects by definition have different goals and thus different outcomes. Nor, given the wide diversity of project objectives, are there metrics that are appropriate for all projects. And in many cases, outcomes will be difficult to measure. ARPA-E funds different organization types and different technology areas that require different assessment metrics. Funding academic institutions may result in more publications, whereas funding small companies may lead to more patents. Large companies may be more interested in using project results internally; in such cases, outcomes as measured by external metrics may be less substantial, but internal impacts on the company may be greater in the form of providing validity or encouraging a new direction. Further, different technologies will differ in their feasibility and scale and in the funds required for prototyping and demonstration.
The committee was interested in developing systematic metrics for the social, economic, and environmental impacts of ARPA-E projects, but given the relatively short timeframe being assessed and the small number of projects that have achieved downstream market engagement, the committee did not believe there would be measurable evidence with which to conduct assessment against the program’s long-term objectives. As noted in Chapter 2, the goals for ARPA-E include reducing imports of fossil fuel, reducing energy-related emissions, and ensuring that the United States maintains a technological lead in
the development and deployment of advanced energy technologies. Nonetheless, ARPA-E has completed more than 200 projects, and these projects provide a record of early accomplishments. Accordingly, the following sections review some of the intermediate impacts of ARPA-E projects and programs according to key external metrics described in Box 4-1 and the case studies conducted to support this assessment, keeping in mind that the passage of decades may be required for the ultimate successes of the work funded by ARPA-E to become manifest. A longer discussion of other metrics that could be used to undertake a retrospective assessment of ARPA-E’s technology is included at the end of this chapter.
EVIDENCE OF OUTCOMES FROM ANALYSIS OF KEY EXTERNAL METRICS: SCIENTIFIC PUBLICATIONS, PATENTS, AND FOLLOW-ON FUNDING AND NEW FIRM FOUNDATION
As discussed above, the committee’s central task was to assess the impact of projects funded by ARPA-E given the relatively short timeframe that has elapsed since the agency began operating. ARPA-E provided the committee with aggregate information on some intermediate outcomes of the work it has funded. As of midyear 2015, 581 journal articles and 74 patents had acknowledged ARPA-E funding (Williams, 2015b). At the 2016 ARPA-E Energy Innovation Summit, the agency announced that 45 projects had secured more than $1.25 billion in private-sector follow-on funding, 36 projects had led to the formation of new companies, and 60 projects had partnered with other government agencies for further development (ARPA-E, 2016b). The agency has also begun to compile and publish examples of intermediate outcomes at the project and portfolio levels (ARPA-E, 2016c). In particular, the agency has developed project impact sheets that include the intellectual property and publications from selected projects in grid storage and operations, power electronics, transportation, clean energy, and efficiency. The same report also describes the results from the portfolio of projects and programs in stationary energy storage.
To derive further insight, the committee worked with an external consultant to undertake in-depth analyses based on the metrics described in Box 4-1. Data from ARPA-E was augmented as described in Appendices B and F. This analysis gave the committee insight into both the incidence of observable outputs from ARPA-E projects and the sources of variation in these outputs across programs and types of organizations funded.
The committee directed the consultant to undertake two interrelated analyses summarized in Goldstein (2016). First, Goldstein analyzed the scientific publication and patenting outputs of ARPA-E projects relative to research projects funded by other U.S. Department of Energy (DOE) agencies, most notably the Offices of Science and Energy Efficiency and Renewable
Energy (EERE) (Goldstein, 2016).2 This analysis leveraged the fact that ARPA-E is one of many DOE entities that fund research, development, demonstration, and deployment for energy-related technologies. Among these funding entities, ARPA-E has a unique mission—to explore high-risk, potentially high-reward ideas within the energy landscape—and unique operational capabilities for executing this mission. Despite these unique characteristics, however, there are agencies within DOE that support early-stage basic and applied research with which ARPA-E can be compared. Specifically, research conducted within the Offices of Science and EERE bears the greatest resemblance to that funded by ARPA-E. Second, Goldstein examined variation among ARPA-E projects and considered the role of organization type (e.g., university versus start-up versus established firm) and project characteristics (e.g., the program with which the project is associated) in explaining that variation. For this latter analysis, it was possible not only to look at publications and patents but also to leverage ARPA-E’s own public data reporting on the external market engagement of its projects.
While the committee recognizes that these offices have different missions both from each other and from ARPA-E, these analyses help demonstrate ARPA-E’s productivity to date and how the agency can contribute to DOE’s overall mission to develop and deploy innovative energy technologies. They show that ARPA-E has produced intermediate impacts that demonstrate progress toward accomplishing its ambitious goals.
Goldstein (2016) performed a detailed analysis of the impact of ARPA-E relative to that of the Offices of Science and EERE using a dataset of all awards offered by the three agencies. This dataset links award-level specifications for recipient type, amount of funding, and project length to publicly available outcomes credited to each award. The sample includes data on patents, patent quality, publications, and publication quality, which, while not completely informative, are appropriate measures for innovative output for purposes of this comparison. For publications (and patents in the following section), the author evaluated both the likelihood of producing at least one output and the incidence rate for producing several outputs. The analysis controlled for award amount, project length, awardee type, and fiscal year in which the award was made, all of which can impact the rate of publishing and patenting. A number of important observations with respect to publication output emerged from this analysis.
First, ARPA-E projects were more likely to publish and to do so with high frequency relative to other DOE offices. Recipients of 44 percent of ARPA-E awards published at least once, compared with the Office of Science and EERE at 27 percent and 18 percent of awards, respectively. In fact, ARPA-E awardees
2 As discussed in Appendix C, Goldstein excluded national laboratory-led projects.
were three times as likely to publish as EERE awardees. Importantly, this latter finding is robust to the inclusion of control factors, such as the amount of funding awarded and project duration. After controlling for these observable differences, however, ARPA-E awardees had the same rate of publishing as Office of Science awardees. ARPA-E awardees received significantly more funding than Office of Science awardees on average, and their projects were slightly longer in duration, which led to a higher percentage of ARPA-E projects that published.
In addition to this baseline result, Goldstein (2016) found significant differences in the nature of these publication outputs. Specifically, ARPA-E awards resulted in more publishing in top journals3 relative to EERE awards and in more energy journals relative to Office of Science awards. (ARPA-E awards published in top journals at the same rate as Office of Science awards and received an equivalent number of citations.)
Of course, ARPA-E’s objectives go beyond the publication of scientific papers, and many projects of a potentially transformative nature may never engage in the scientific publication process as part of their market and social impact. Further, research shows that the probability that a paper is cited peaks years after its publication, that the median lag from publication to patent citation is nearly a decade, and that other non-peer-reviewed literature is used nearly as often as formal publications in patent citations (Anderson and Breitzman, 2017; Popp, 2016a, b). The channels from knowledge production to invention are diverse. Nonetheless, these bibliometrics do offer one window into the scientific and invention outputs of ARPA-E projects that buttress the case for the agency’s effectiveness in selecting and managing projects that result in scientific achievement. Intuitively, it is understandable that ARPA-E awards, which tend to support earlier-stage research, would produce more publications than EERE awards. However, the results of comparing the publication output of ARPA-E and Office of Science awards suggest that ARPA-E’s portfolio, while targeting more applied technologies that could directly affect the energy landscape, is simultaneously able to expand the boundaries of science.
As with scientific publications, patents are an imperfect but informative indicator of the technological output of federally funded research. As with scientific publications, it is possible to compare the patenting activities of projects funded by ARPA-E with those of projects funded by the Office of Science and EERE, once again controlling for such factors as the size and
3 For this analysis, a top journal refers to one of the 40 journals from the Thompson Reuters Energy and Fuels category with the highest number of citations for the time period 2005–2015.
duration of the research grant and how much time had elapsed since the beginning of the project period.
During the time period investigated, ARPA-E’s portfolio of projects resulted in more patents per project than the portfolios of either the Office of Science or EERE. The 13 percent of ARPA-E awards that have resulted in at least one patent compares with 2 percent for Office of Science projects and 5 percent for EERE projects. Moreover, the odds of an ARPA-E awardee being granted at least one patent are three times higher than those for an EERE awardee.
Looking beyond the likelihood of patenting, individual ARPA-E projects also have a higher incidence ratio for patenting than either the Office of Science or EERE over the time period analyzed, suggesting that ARPA-E awardees produce patents in greater numbers than awardees of both of these other offices. Together, the increased likelihood of patenting and the higher incidence rate ratio result in a superior patenting efficiency relative to funds allocated: ARPA-E projects returned a patent for each $8.2 million awarded, while the equivalent figures for the Office of Science and EERE were $18.1 million and $28.4 million, respectively.
Goldstein (2016) measured patent quality with two indicators: the first captured whether the patent was cited by other patents and the second the number of claims made on a specific patent. With regard to the first of these indicators, ARPA-E projects had a greater likelihood than EERE projects of producing patents that would be cited. With regard to the second, ARPA-E projects produced patents with more claims relative to the projects of both the Office of Science and EERE. These measures suggest that on average, the patent portfolio of ARPA-E projects is of higher quality than those of the other two officers.
These results reflect favorably on ARPA-E’s portfolio. Intuitively, as an applied research program, ARPA-E would be expected to fund projects leading to patents more frequently than the Office of Science, which focuses on basic scientific research. However, the relative patenting performance of ARPA-E and EERE supports ARPA-E’s value proposition. Effectively, ARPA-E’s research portfolio yields a higher return with respect to inventive metrics.
Publications, Patents, Follow-on Funding, and Firm Formation among ARPA-E Projects
Goldstein (2016) also considered factors that could explain the variance in outcomes among ARPA-E projects themselves. These factors include funding amount, project length, organization type, program, and team composition. The author evaluated this award-level variance with respect both to patents and publications and to additional metrics related to market engagement. Specifically, in addition to patents and publications, Goldstein leveraged ARPA-E’s publication of a series of market engagement metrics focused on three types
of activity—follow-on public funding, follow-on private funding, and new firm formation.
Overall, there is evidence that market engagement metrics are particularly sensitive to organization type. On the one hand, each of the 20 companies that formed following ARPA-E funding originated in a university. On the other hand, 64 percent of awards that received follow-on private capital were start-up companies, and 77 percent of the awards that received follow-on public funding were in the private sector.
With regard to publications, university projects were more productive than projects of the other organization types. This is intuitively a result of the incentives to publish within academia. With regard to patents, however, the data are unclear. At this stage, patents remain a lagging indicator for ARPA-E’s success. The mean time to patent for ARPA-E projects is 4 years, but as discussed earlier, 4 years has not elapsed since the inception of most ARPA-E projects. Given this lag, the committee recognizes that published patent applications could be used as an alternative metric. However, it considers issued patents to be the better long-term metric.
The data suggest that established firms, that is, private-sector companies that were founded prior to 5 years before receipt of an ARPA-E award, were less productive than the other organization types. Specifically, established firms produced fewer patents and publications and received less private or public follow-on funding. However, it is unlikely that these output metrics adequately capture success within a large company. Incentives to publish at an established firm are low relative to those at universities, where publications lead to career advancement, or at start-up firms, where publications serve as technology validation in the market for capital. Established firms also have a broader tool set for protecting intellectual property, such as trade secrets, a resource that can reduce the incentive to patent. Moreover, these firms may not pursue patents on technologies they do not intend to commercialize. Finally, established firms typically finance commercialization without the support of venture capital. While established firms may not perform as well as other organization types with respect to these metrics, then, more research is needed to identify an appropriate set of outcome metrics for evaluating ARPA-E projects of established companies.
Team composition appears to have an effect on project outcomes. Applicants to ARPA-E are able to partner with other organizations and divide tasks within the statement of work. Overall, these partnerships appear to have a positive effect, significantly increasing the likelihood that an award will produce a publication or patent or receive follow-on funding. The mechanism for this advantage is unclear. It may be that combining the strengths of multiple organizations contributes to a project’s technical progress. But it may also be that partnerships serve as applicant validation, such that technically superior projects are able to attract partners.
Significant variation in project outcomes can be seen across ARPA-E programs. More than 80 percent of projects within the Innovative Materials and
Processes for Advanced Carbon Capture Technologies (IMPACCT) and Electrofuels programs have resulted in a publication. Most programs have yet to produce a patent, but 60 percent of projects in the Batteries for Electrical Energy Storage in Transportation (BEEST) program and more than one-third of projects in the OPEN 2009 program have produced patents. The latter figure is particularly striking, as the OPEN programs fund a large number of projects across different technology areas. Projects within two programs—BEEST and Grid-scale Rampable Intermittent Dispatchable Storage (GRIDS)—emerge as highly productive with respect to accelerating commercialization of the funded technology through company formation or follow-on funding. Finally, 100 percent of projects within IMPACCT and GRIDS have received some validation of success through either a patent, a publication, or some form of market engagement.
Summary of External Metrics Analysis
ARPA-E projects produce patents more than projects of either the Office of Science or EERE, with a corresponding reduction in the overall cost per patent. ARPA-E projects also publish more than those of EERE and on par with those of the Office of Science. Relative to EERE, ARPA-E’s focus on emerging technology results in an increase in inventive capacity. Relative to the Office of Science, ARPA-E’s focus on technology with an applied orientation results in an increase in inventive capacity without diminishing its ability to produce high-quality scientific research. Box 4-2 provides a summary of these observations, which suggest that among these three organizations, ARPA-E is serving a unique role in terms of funding projects that are focused on the energy system, hold both scientific and technological potential, and are associated with early-stage indicators of practical impact. While the committee emphasizes the caveats discussed earlier—that these organizations have different missions and that the metrics used in this analysis represent only intermediate outcomes—these results demonstrate ARPA-E’s productivity and the contribution the agency makes to the DOE mission.
EVIDENCE OF IMPACT FROM CASE STUDIES
Case studies are one of the methods used by the committee to understand, describe, and then, leveraging the technical expertise of its members, evaluate ARPA-E’s operations and impacts (Appendix D contains the full write-ups of the case studies). The descriptions of the selected projects and programs in this section and Appendix D focus on the role played by ARPA-E in their management and the committee’s assessment of the potential transformational nature of the respective technologies, the impact on the field, and the
expectations for market adoption. An important caveat to this discussion is that these case studies are not intended to be representative of all ARPA-E programs and projects. Together, however, the quantitative analysis of outcomes presented in the previous section and the case studies presented here provide evidence of the current operations, outcomes, and potential long-term impacts of ARPA-E programs and projects on energy sector technologies.
Three types of case studies were conducted: the first focuses on one program, Strategies for Wide-bandgap, Inexpensive Transistors for Controlling High-Efficiency Systems (SWITCHES); the second examines a portfolio of electricity storage (primarily electric battery) projects; and the third includes 10 individual projects grouped into three categories—successful, cancelled, and other. The first two types allowed the committee to understand the implementation and impacts of a broad set of projects within both a single program (SWITCHES) and a broad set of programs that funded electricity
storage projects. In both instances, the committee aimed to select cases with a broad set of potential impacts. The case studies of individual projects considered their technical potential as well as the role played by ARPA-E.
An important consideration when attempting to assess the success of the programs and projects described in the case studies was the issue noted throughout this report: the state of maturity of ARPA-E. Because most projects are funded for approximately 3 years, there was at most 3 years of data available on completed projects for use in this assessment. Therefore, the focus was on evaluating ARPA-E’s operations and providing a technical assessment of whether the selected programs and projects dealt with potentially transformational technologies and/or filled white spaces not being pursued by other public and private entities because of technical and financial uncertainty.
Overview of One Program (SWITCHES)
The committee undertook a case study of a single focused program to understand the implementation of and impacts from a broad set of projects. The committee selected the SWITCHES program because it sees the development of wide-bandgap materials for semiconductors as a potentially transformational development in power electronics. The description of the SWITCHES program in Appendix D highlights many of the hallmarks of ARPA-E, including program director autonomy; investments in high-risk technology; and plans for ensuring commercialization pathways, including the need to keep power electronics manufacturing in the United States.
Power electronics are ubiquitous, and enormous effort is expended on making electronic switching work better. Doing so means developing devices that are more efficient, longer-lasting, lower-cost, and useful in new applications. Improvements in power electronics, for example, can make the grid more efficient and reliable and improve the integration of renewables. There may be other applications as well, including for electric vehicles and lighting. The SWITCHES program solicited proposals for investigation of less well-understood materials with the potential to be tens or hundreds of times better for some vital applications, but not being pursued because the risks of failure were too high and times to market were too long to make such projects competitive for typical industrial research or even much government research funding. Specifically, the SWITCHES program has funded projects pursuing transformational advances in wide-bandgap (WBG) materials, device fabrication, and device architectures to enable the development of new types of semiconductors.
Program Objectives and Impacts
Within the SWITCHES program, ARPA-E recognized the substantial potential for WBG semiconductor materials to yield energy savings and new designs for these applications relative to existing technologies. The goal of this program is to enable the development of high-voltage (approximately 200–2,000 V), high-current power semiconductor devices and circuits that, upon ultimately reaching scale, could offer affordable breakthrough performance in terms of higher efficiencies, higher switching frequencies (and therefore smaller packages), and higher-temperature operation (ARPA-E, 2013d). Currently, industrial power electronics circuits are overwhelmingly (>95 percent) built with transistors made out of silicon. Silicon power transistors, however, become increasingly inefficient beyond operating voltages of 400–500 V. There is demand today for power transistors that operate in the approximately 650 V range for circuits that can be plugged into the wall. A second set of needs is in the 1.2–1.7 kV range for automotive applications. The adoption of WBG semiconductors is a move toward higher-frequency circuits, which would allow decreased component sizes and increased efficiencies. Figure 4-3 shows the improvements (lower losses, higher breakdown voltages, and higher-power applications) afforded by various WBG semiconductors over conventional silicon. The left panel of the figure shows the potential for WBG semiconductors to be used in high-power applications with high breakdown
voltage. The right panel shows materials with better power device performance (high breakdown voltage and low loss) toward the upper-right corner.
The objective of this program is not only to reduce the barriers to ubiquitous deployment of low-loss WBG power semiconductors, but also to develop approaches that will bring the costs of these devices to functional cost parity with silicon transistors while offering better performance. The program encompasses four major tasks: (1) creating crack-free device substrates using an economically promising method, (2) growing a doped semiconductor layer on these substrates with necessary electronic properties, (3) demonstrating working electronic devices with increasing levels of voltage capability using these new materials, and (4) completing a technology-to-market (T2M) plan. ARPA-E also considers the SWITCHES program a complement to the New Generation Power Electronics Innovation Institute, which is working to help create and manufacture WBG semiconductor-based power electronics in the United States (ARPA-E, 2014e). Within the program, ARPA-E has funded 14 projects: 2 on diamond, 1 on silicon carbide (SiC), and 11 on gallium nitride (GaN). Roughly half of these projects have focused on transistor technology and design and roughly half on the synthesis of better substrate materials on which to build the transistor technology. This is a good mix that underscores the importance of materials issues in the technology.
The SWITCHES program clearly aspires to have a transformational impact on power semiconductor devices. The design space changes if the boundaries on switching frequency, voltage, and temperature can be significantly moved. However, the risk associated with launching a new materials set for power electronics is high. The default choice is always silicon because it is cheap and has established supply chains. The risk is that the WBG devices could require a decade from invention to profit, let alone broad application, and the tooling is enormously costly—thus the motivation for ARPA-E to develop the SWITCHES program.
The committee’s assessment is that the overall project selection within the SWITCHES program has been good in terms of mixing materials, devices, and university and industry skills. However, the lack of participation of most of the major industry players (with the possible exception of Triquint) is significant. It represents a lack of suitable “catchers,” companies that are able to manufacturer the technology should it be successful, and thus a risk profile that may be somewhat extreme and the possible need for a refined technology-to-market (T2M) strategy specific to semiconductor technology.
In terms of ARPA-E’s role, the performers for these projects demonstrate that Tim Heidel, program director for SWITCHES, and his team have productively and efficiently scoped, shaped, and supported their projects. In general, the committee observed that the program director and core staff are
highly competent, dedicated, and responsive. Each project is subject to 10 or more “touches” annually by the program director and the staff front office team, including quarterly reviews, annual reviews, teleconferences, site visits, and other events. “Pivots,” or changes in milestones, are rigorously documented and tracked. Further, ARPA-E has not hesitated to cut projects when doing so made sense—2 of the 14 SWITCHES projects were cancelled within 2 years. These actions serve as evidence of the robust and active program management and oversight policies ingrained in ARPA-E’s culture.
Overall, the move to low-loss WBG power semiconductor devices would enhance the efficiency of electric motors through improvements to variable-frequency drives; substantially reduce the weight and additional cost of power electronic systems for plug-in electric vehicles; and reduce the cost, weight, volume, and losses from wind and solar electric power inverters. There may be other potential applications as well, although this outcome is not preordained. Engineers do not wait for innovations from the science world before designing things. The corollary is that once scientific discoveries are brought into the design space, engineers will find applications not fully conceived of before. Thus, ARPA-E’s efforts in the SWITCHES program are consistent with its goal of bringing potential transformational technologies beyond discovery to the demonstration phase.
Overview of One Portfolio (Electricity Storage)
The committee decided to focus on electricity storage for its portfolio analysis because these technologies constitute the largest fraction—10–15 percent—of ARPA-E’s funded projects, making this one of the largest classes of technologies funded by the agency over the past 7 years. The committee performed an overarching high-level assessment of this portfolio to examine, at least for this field, the extent of ARPA-E’s impact and whether any broad trends within this group of projects may reflect the strengths and weaknesses of the approaches used by the agency. However, the degree to which this collection of projects can be thought of as a true portfolio is unknown—specifically, when selecting projects for an award, how much consideration is given to prior energy storage funding decisions and technical content in the application when weighing the relative programmatic merits of new proposals, especially for those projects awarded in the agency’s open calls.
The specific purpose of this exercise was to examine numerically all of the projects addressing electric energy storage that ARPA-E funded from its inception through June 2015. The goals were to
- provide analysis to show how a single class of technologies—energy storage—functioned within ARPA-E;
- search for patterns or trends;
- examine whether the narrative ARPA-E portrays is reflected in reality;
- compare data for this group of projects against the quantitative analyses applied to all ARPA-E projects; and
- provide the other committee subgroups with additional data.
Note that for this analysis, the committee examined only projects focused on storage of electricity. The committee did not consider projects focused on thermal storage or on improvements to battery management systems, sensors, or testing.
ARPA-E has funded its electricity storage projects through three focused programs (BEEST, GRIDS, and Robust Affordable Next Generation Energy storage systems [RANGE]), as well as through its OPEN 2009, OPEN 2012, and OPEN 2015 solicitations. The projects in the GRIDS program, which began in 2010, focused on developing technologies that can store renewable energy for use at any location on the grid at an investment cost below $100 per kilowatt hour (kWh). The majority of these projects are electric battery projects, with a few looking at compressed air and flywheel storage approaches. The projects in the BEEST program, which began in 2011, focused on developing a variety of rechargeable battery technologies for plug-in electric vehicles that would meet or beat the price and performance of gasoline-powered cars. The program considered radical improvement of current lithium-ion technologies and new designs using other battery chemistries that incorporate magnesium, sodium, zinc, lithium-sulfur, and lithium-air designs. The projects in the RANGE program, which began in 2013, continued efforts on rechargeable battery technologies for plug-in electric vehicles, focusing on battery designs that would enhance safety, maximize the overall energy stored in a vehicle, and minimize manufacturing costs. The OPEN 2009, OPEN 2012, and OPEN 2015 solicitations were designed to fund transformational breakthroughs across the entire spectrum of energy technologies, including stationary and transportation electricity storage projects. Although a number of storage projects were funded as a result of the OPEN 2015 call, these are very new projects that are not included in the analysis provided below.
Overview of Electricity Storage Portfolio
The committee assessed the electricity storage technology portfolio as part of its overall assessment of ARPA-E’s impact and to determine whether any broad trends within these projects might reflect the strengths and weaknesses of the agency’s approaches. The vast majority of these projects focused on electric battery technologies. Additionally, the committee did not consider programs or
projects designed to explore improvements to battery management systems, sensors, or testing. Even with the analysis limited to energy, the important unknown remains of the extent to which ARPA-E considers prior electricity storage funding decisions and technical content in weighing the relative merits of new programs or projects focused on storage, particularly projects awarded through the OPEN solicitations. Thus, it is not possible to know the extent to which this collection of programs and projects can be thought of as a true “portfolio.”
ARPA-E has funded 63 electricity storage projects; 5 have been cancelled, 30 are still active, and 28 have been completed. ARPA-E funds were distributed approximately equally among the five programs examined for this analysis that sponsored electricity storage technology—RANGE, BEEST, GRIDS, and the 2009 and 2012 OPEN calls—although the programs that funded more individual projects, in particular RANGE, had lower per-project funding. The typical storage project lasted 2.7 years and consumed roughly $980,000 per year of ARPA-E funding, although both funding levels and durations varied significantly. For example, the longest project stretched over 6 years, and one project received more than $10 million in funds.
Focus Area and Project Type
From a technology maturity perspective, the electricity storage projects were divided between proof-of-concept (36 projects) and prototyping (27 projects), and between stationary (25 projects) and transportation (21 projects) applications. A healthy fraction of the projects (17 projects) were “crossover” in nature in that the technology, if successful, was aimed at both the stationary and transportation markets. Importantly, no project focused on scaling the production level of materials or devices, which intuitively makes sense given ARPA-E’s emphasis on early-stage R&D and device proof-of-concept.
The committee reviewed the project descriptions and classified the projects based on their technical focus (Figure 4-4). A number of projects focused on a single technical outcome, such as safety or durability. Five projects had a dual focus on durability and performance. More than half of the projects had a shared focus on cost and performance, while four had a shared focus on cost and durability, and six had cost as their singular focus.
Funding of White Space and Project Storyline
As discussed previously, ARPA-E has been mandated to seek out and fund transformational ideas and to find technical white space in which no significant work has been done. In about half of the projects, there is clear evidence of such white space being addressed (Figure 4-5). Further, while there is evidence of significant prior work in the public domain for 50 of the 63 projects, in 25 of
these the prior work was done at the institution of the project’s principal investigator. These findings show that ARPA-E uses its funding to move early-stage innovations to the demonstration stage.
The committee also looked at the storyline of the projects, meaning the combination of the type of project lead and type of project. The majority of the projects (47 of 63) fall into the category of either a small-company principal investigator further developing a novel technology initiated by that individual or a university professor principal investigator gaining support for a new idea. Additionally, 6 awards went to large companies exploring new concepts.
Metrics of Outputs and Outcomes of Electricity Storage Projects and Comparison with Overall ARPA-E Projects
The committee compiled the metrics of publications, patents, follow-on funding, and commercialized products for ARPA-E’s electricity storage projects. These projects had generated a total of 115 peer-reviewed papers and 22 patents as of January 2016. Among the papers, 37 were published in high-impact journals, and 5 were highly cited. A total of 20 projects (approximately 30 percent) received follow-on funding from public or private entities or both. As of the end of 2016, four companies had a mass-manufactured product on the market that had been developed during or after receipt of ARPA-E funding (although in these cases, it is not altogether clear that the product on the market was the same as that which was developed with ARPA-E support).
The committee also looked at how the outcomes from the electricity storage portfolio compared with those of the entire collection of projects funded by ARPA-E. Company formation has resulted from 7.5 percent of the 475 projects overall, compared with 9.5 percent of the electricity storage projects (6 of 63); the corresponding percentages for follow-on funding are 9.5 percent and 19 percent. Thus it appears that the electricity storage portfolio has funding and technical development outcomes similar to or better than those of ARPA-E projects overall.
The committee observed that ARPA-E has funded a wide array of projects related to electricity storage, and that these projects generally do not overlap significantly with projects funded by other sources. Demonstration-style projects require prior work as a base, so it is no surprise that for just over half of the projects, there is evidence of prior similar work having been conducted at either the principal investigator’s or some other institution (Figure 4-5). Combined with programs not included in this analysis that focus on thermal storage technologies (High Energy Advanced Thermal Storage [HEATS]), technologies for improving battery management and sensor systems (Advanced Management and Protection of Energy storage Devices [AMPED]), and approaches for testing and evaluating battery storage systems (Cycling Hardware to Analyze and Ready Grid-scale Electricity Storage [CHARGES]), ARPA-E has invested a significant fraction of its resources in improving electricity storage systems.
This is a worthy investment given the critical need for improved electricity storage in both the transportation and electricity sectors. The committee considers much of the electricity storage portfolio to have a medium or high degree of technical risk, meaning that its projects have goals that go beyond current technical capabilities and are in many cases difficult to achieve. Further, the committee notes that the technical background of the program directors has been appropriate for most projects, with most having had significant entrepreneurial experience.
Given the high-risk/high-payoff approach both DARPA and ARPA-E take in creating programs and funding projects, the degree of follow-on support resulting from company founding appears to be reasonable for these projects, a significant portion of which found support after the ARPA-E funding ended. On the other hand, it can also be said that the majority of companies involved in ARPA-E electricity storage projects that have actual electricity storage product offerings on the market today were funded by ARPA-E after being founded, and in most cases had a well-defined technical path/plan forward. In these cases, the most common scenario is that ARPA-E funded demonstration projects or product improvements. These types of cases tend to make up the bulk of the projects ARPA-E currently counts as successes, which is not surprising given that it takes 7 to 10 years to mature a new concept that would not have existed substantially prior to being funded. Thus it is quite possible that more of those projects that have received follow-on funding will yield product offerings in the next 5 years.
Individual projects for case studies can be chosen using three approaches—convenience, purposive, and probability (Morra and Friedlander, n.d.). The committee used a purposive approach for these case studies to ensure that particular phenomena would be represented (Balbach, 1999). The purposive criterion was to select an approximately equal number of projects funded through open and focused solicitations that represented a mix of organizational types, including academia, national laboratories, and companies of varying sizes.
Applying this criterion, the Case Study Team reached out to several performers (recipients of ARPA-E awards) with the goal of obtaining a specific number of case studies across three broad categories of projects defined by their current status (see Table 4-1):
- “Successes” are projects that the committee or the agency considers likely to have or that have had market success in the energy sector and that have received follow-on private-sector funding.
TABLE 4-1 ARPA-E Individual Project Case Studies
|Company Name||Short Project Title||OPEN/Focused||Company Type|
|Successes—projects that the committee or the agency considers likely to have or that have had market success in the energy sector and that have received follow-on funding|
|1366 Technologies||Cost-Effective Silicon Wafers for Solar Cells||OPEN 2009||Small Co.|
|Foro Energy||Laser-Mechanical Drilling for Geothermal Energy||OPEN 2009||Small Co.|
|24M||Semi-Solid Flowable Battery Electrodes||BEEST||Small Co.|
|Harvard University||Slippery Liquid-Infused Porous Surfaces (SLIPS)||OPEN 2012||Univ.|
|Smart Wires||Distributed Power Flow Control||GENI||Small Co.|
|Cancelled—projects that were ended before the original end date because they were not meeting their goals and appeared likely not to do so eventually|
|General Electric||Nanostructured Scalable Thick-Film Magnetics||ADEPT||Large Co.|
|Other—projects that were completed and that to date have resulted in little or no direct energy market success, but still advanced the state of knowledge|
|Agrivida||Engineering Enzymes in Energy Crops||OPEN 2009||Small Co.|
|Ceres, Inc.||Improving Biomass Yields||OPEN 2009||Small Co.|
|HRL Laboratories||Low-Cost Gallium Nitride Vertical Transistor||SWITCHES||Small Co.|
|Stanford University||Radiative Coolers for Rooftops and Cars||OPEN 2012||Univ.|
NOTES: The full case studies are in Appendix D. ADEPT = Agile Delivery of Electrical Power Technology; BEEST = Batteries for Electrical Energy Storage in Transportation; GENI = Green Electricity Network Integration; SWITCHES = Strategies for Wide-bandgap, Inexpensive Transistors for Controlling High-Efficiency Systems.
- “Cancelled” are projects that were ended before the original end date because they were not meeting their goals and appeared likely not to do so eventually.
- “Other” are projects that were completed and that to date have resulted in little or no direct energy market success, but still advanced the state of knowledge.
ARPA-E performers (principal investigators) for these individual case studies were asked about the state of their research before applying to ARPA-E, their reason for responding to the agency’s Funding Opportunity Announcement (FOA), their interactions with their program director and T2M team, the tracking of progress against the metrics established for their project, what they accomplished during the project, and other observations about their project that they wished to share. The case studies also include an assessment of the technology developed during the project and its potential to have an impact in the energy sector.
Short summaries of 3 of the 5 projects that the committee categorized as successful are presented below. (The complete case studies for all 10 projects reviewed are presented in Appendix D.) It should be noted at the outset that, although these projects are categorized as successful in that they show potential for success, none of them have as yet been transformational in the energy sector—as would be expected given the extended period of time in the market required for transformational technologies to become apparent.
Cost-Effective Silicon Wafers for Solar Cells
1366 Technologies is developing a potentially disruptive solar technology, a process aimed at reducing the cost of solar wafer manufacturing by 50 percent by 2020. Silicon solar wafers are the building blocks of solar cells and panels, and reducing their costs has implications for solar’s adoption rate. Instead of growing expensive crystals and cutting them into thin fragile wafers, 1366 Technologies casts the wafers directly from molten silicon and can make shapes that are more durable and use less material. The efficiency of these cells compares favorably with that of today’s advanced technologies, so 1366’s wafers could seamlessly replace wafers currently used in the market while at the same time greatly reducing their costs. Since most solar panels are made outside of the United States, moreover, 1366’s technology has the potential to dramatically increase domestic production, thus increasing the nation’s energy security (ARPA-E, 2009).
1366 Technologies was selected in the OPEN 2009 solicitation and is the only funded project to develop novel silicon production technologies. At the time of the solicitation, it was a young company in the midst of a funding round and growth (Figure 4-6). ARPA-E funding represents a small fraction of the
total capital raised by the company to date, but represented approximately 20 percent of the total raised at the time of the grant. ARPA-E provided 1366 with funding that allowed the company to investigate the basic science that helped lead to a technology capable of being commercialized. This technology ultimately replaced the leading technology options that had received venture capital funding. This project is an example of a case in which a focus on the technology and the freedom to pursue basic science made a promising innovation ready for the market. 1366 Technologies also benefited from the positive press associated with videos and the company’s presence at the ARPA-E Energy Innovation Summits.
Slippery Liquid-Infused Porous Surfaces (SLIPS)
Harvard University developed various slippery surface technologies intended to enable materials and coatings to achieve extreme energy savings in many industrial settings. The idea for SLIPS was inspired by the slippery surface of the carnivorous pitcher plant, which uses liquid and a mechanical trap to catch insects. By copying the plant’s systems, the Harvard team developed a porous material that could hold liquid similarly to the way a sponge holds water. ARPA-E guided the team to investigate commercial applications for the
technology and select those with the greatest promise. After a 6-month exploratory effort, the team, together with ARPA-E, chose to develop a SLIPS coating for refrigeration coils that would reduce defrost cycles by enabling faster shedding of frost and water from the surface of the coils. The team tested large-scale coated coils at LG and other local coil-producing companies to demonstrate significant energy savings with SLIPS-coated coils. After 2 years, sufficient proof-of-concept had been established to prompt the launch of a startup company, SLIPS Technologies, Inc. (STI), to broadly commercialize the technology. STI was launched in October 2014 with venture capital financing led by the venture capital arm of BASF (Figure 4-7). Following this spinoff, ARPA-E extended funding for a third year to enable exploration of the use of SLIPS coatings to prevent marine fouling on ship hulls so as to reduce drag and improve fuel efficiency in various marine applications. Early results demonstrate the technology’s potential to address unmet needs in that market with a nontoxic alternative to current antifouling coatings (ARPA-E, 2012b).
ARPA-E’s flexibility and ability to adjust project goals and metrics helped the team achieve meaningful technology goals. The principal investigator highlighted ARPA-E’s interest in identifying the right technology and markets for the innovation and its insistence on a rigorous due diligence process to
identify such markets. Focusing on early market adoption made funding available to continue the research in more complex areas. This was a successful case of ARPA-E helping a principal investigator’s team explore commercially relevant applications for an innovative material, establish proof-of-concept, and then transition the effort to a commercialization path via a start-up company.
Distributed Power Flow Control
Modernizing the electrical grid requires the ability to integrate renewable energy sources into the grid and increase controllability to obtain more throughput. ARPA-E funding helped enable Smart Wires to develop a solution for controlling power flow within the electric grid to better manage transmission capacity. Smart Wires’ devices clamp onto existing transmission lines and control the flow of power within the lines. The principles behind the devices are modularity with rapid deployment and ease of redeployment on an increasingly unpredictable grid that is being influenced by the growing penetration of distributed energy resources and other technologies. The modularity of the devices is unique and offers several advantages over traditional power flow control solutions, which typically require long lead times, are highly complex, demand significant capital investment, and ultimately represent single points of failure (ARPA-E, 2011a).
ARPA-E’s role in the Smart Wires project was to enable the start-up to build prototype devices and deploy and test them in an operating environment (Figure 4-8). The project represented an instance of very late-stage development, with the technology already in existence and ready for field testing. ARPA-E’s financial and advisory contributions were invaluable to Smart Wires’ success, helping to derisk the technology for follow-on investors and making it easier for the company to raise capital going forward. Smart Wires’ management attributes much of the company’s success to ARPA-E’s engagement. Since receiving the funding from ARPA-E, Smart Wires has undertaken several rounds of successful fundraising, including $30.8 million in 2015 to bring its PowerLine Guardian product to commercial production.
Summary of Evidence from Case Studies
The case studies evaluated by the committee help demonstrate how the agency is fulfilling its mandate to fund transformational technologies or fill white space opportunities. They provide some early indicators of success and demonstrate how the principles behind ARPA-E play out in the management of programs, portfolios of projects, and individual projects. The descriptions in this chapter and Appendix D are for an array of projects, from those supporting the earliest development of a potential technology from a scientific principle
(radiative cooling) to technologies that are emerging into commercial products (e.g., cost-effective silicon wafers for solar cells, SLIPS, Smart Wires).
The case studies also illustrate how ARPA-E shapes projects, from providing strategic funding to offering support and guidance for changes in both technological and commercialization pathways. While the summary of the case studies in this chapter focuses on three successful projects, the case study write-ups in Appendix D illustrate other elements of ARPA-E’s project selection and management approach. For instance, the Ceres, Inc. project provides an example of a supported firm that moved away from energy markets. Given ARPA-E’s goal of spanning boundaries, such outcomes appear more likely for an ARPA-E project than for other DOE programs. Nonetheless, the case also shows how the project manager was able to adapt. In addition, the write-ups in Appendix D emphasize some of the potential issues associated with the high-touch approach of ARPA-E’s program directors, including the potential loss of interest in a project when the program director changes and the burdens placed on performers by the number of interactions with the program directors. These findings help support the notion that ARPA-E is implementing the vision Congress had when it authorized the agency. While the committee emphasizes the caveat expressed earlier—that these case studies are not representative of all projects and programs funded by the agency—these findings help demonstrate
ARPA-E’s productivity and how its unique operations can contribute to its success.
It should be noted that the committee heard presentations from current and former ARPA-E directors and program directors about other projects and programs, including others that could qualify as successes (e.g., Branz, 2015; Gur, 2015; Heidel and Gould, 2015; Majumdar, 2015; McGrath and Umstattd, 2015; Schiff and Zahlar, 2015; Williams, 2015b). Time and resources did not allow the committee to undertake a more exhaustive review.
TRANSFORMING ENERGY INDUSTRY ATTITUDES
ARPA-E projects develop technologies for applications in the energy sector, a complex, extensively regulated sector that is capital-intensive and risk-adverse and adopts innovations over long time horizons. ARPA-E is “developing technologies to alter the status quo of the existing energy economy, where much of the end product is a uniform commodity (liquid fuel or delivered electricity)....ARPA-E technologies must enter a crowded market and ultimately compete on price with the legacy fossil and nuclear sectors” (Mehra, 2013). ARPA-E thus operates in a very different competitive landscape from that of DARPA, which has developed completely new technologies in areas in which no prior capability whatsoever existed, such as unmanned aerial vehicles and global positioning technology (Mehra, 2013).
The conservatism of the established energy base provides a critical motivation for the substantial proportion of ARPA-E projects that involve pilot-scale prototyping, demonstration, and testing aimed at persuading risk-averse entities to consider new technological approaches (NAS et al., 2009). Electric utilities, for example, bulk purchasers of energy technologies from the private sector, have an interest in providing the lowest possible rate for their ratepayers, place a high premium on reliability, and tend to earn a low rate of return on investments. Thus, these entities take a conservative approach to the adoption of new technologies (NASEM, 2016). Similarly, other energy sector companies tend not to adopt next-generation energy technologies unless there is a high rate of return on investment and some degree of assurance that the technology will perform reliably. So despite the promise of ARPA-E technologies, the agency still has the additional challenge of navigating these deployment barriers (Mehra, 2013). Finally, it should be noted that the commercial viability of some ARPA-E projects will depend, at least in part, on policies that incorporate environmental costs or impact the relative costs of renewable energy compared with fossil fuel energy.
In such an environment, ARPA-E demonstration projects that result in attitudinal changes in the energy industry may be transformational. Concentrating the agency’s programs and projects in white space research areas and on technologies that have high risk and the potential for high reward has a
number of benefits apart from avoiding duplication of existing research. It challenges established attitudes and entrenched constituencies and may alter pathways, barriers, and risks in key technology areas, and it builds community knowledge and expertise in new areas as well. Most of the project case studies examined for this study involved prototyping, data gathering in operational environments, scale-up, and the development of pathways to commercialization rather than research into entirely new scientific concepts. As noted in the full Smart Wires write-up in Appendix D, for example, the program director predicted that utility executives’ realization that they can modulate power flows on command would have transformational effects on their decision making, ultimately leading to far more efficient, lower-cost transmission.
CREATING NEW COMMUNITIES OF RESEARCHERS
The statement of task for this study directed the committee to assess the success of ARPA-E’s focused technology programs in spurring the formation of new communities of researchers in specific fields. ARPA-E’s Strategic Vision 2013 states that the agency’s focused programs are “developed through engagements with diverse science and technology communities, including some that may have not traditionally been involved in the topic area...” (ARPA-E, 2013c). Project applicants are encouraged to form interdisciplinary teams covering broad ranges of technology. The FOA for the FOCUS program, for example, stated that applicants “may benefit from formation of interdisciplinary teams with expertise in more than one of the following areas: non-imaging optics, advanced optics and photonics, mechanical engineering….” (ARPA-E, 2014d).
The committee finds that it is too soon to assess whether ARPA-E programs will result in the creation of permanent new communities of researchers. The agency is actively seeking to convene diverse constituencies: ARPA-E’s workshops, kickoff sessions, and annual program reviews are all institutional mechanisms not only for developing focused programs but also for building communities. These activities could evolve into new, lasting research communities if cultivated over the long term. In his presentation to the committee, former ARPA-E director Arun Majumdar (2015) indicated that the agency does not initiate a program without convening a workshop that not only helps avoid duplication but also “brings people together from different disciplines, and magic happens.” Majumdar recalled that with respect to power semiconductors, “dots had to be connected. The soft magnet guys have to talk to the wideband semiconductor guys.” Though it may be too early to assess whether ARPA-E programs result in the creation of permanent new research communities, this may be a topic for ARPA-E to assess in a future evaluation.
The DARPA Model for Orchestrating Communities of Scientists
The desire of ARPA-E to foster new communities of researchers in energy follows DARPA’s lead in creating such communities. Fuchs (2010) provides a comprehensive study of the operations of DARPA’s Microsystems Technology Office between 1992 and 2008 that highlights the key role played by that agency’s program managers in convening and managing social networks to achieve technological objectives.
According to Fuchs (2010), key to understanding DARPA’s role in influencing technology directions is understanding the role of the program manager. The program manager is not just someone who “opens windows” for researchers to bring funding ideas (Block, 2008) or merely acts as a “boundary spanner” or “broker” who connects different communities (Ansell, 2000; Block, 2008), but rather a person who plays a more deliberate role in changing the shape of the network of scientists so as to identify and influence new directions for technology development. Processes used by DARPA program managers for seeding and encouraging new technology trajectories so as to “prevent technological surprise” include
- bringing star scientists together to brainstorm new ideas, whether informally (just three star scientists called together by a program manager) or more formally through such mechanisms as the Information Science and Technology working groups and the Defense Science Board (DSB);
- seeding disparate researchers around common themes;
- encouraging early knowledge sharing among these star researchers through program reviews in which they present their results to each other;
- providing external funding agencies and industry with third-party validation for new technology directions; and
- not sustaining the technology (e.g., ensuring that industry or other funding agencies eventually take over).
Fuchs (2010) concludes that the DARPA program managers become a central node to which information from the larger research community flows because each is a former member of the research community who has risen in status and holds the promise of providing funding. In this role, they do not give way to the invisible hand of markets, nor do they step in with top-down bureaucracy to “pick technology winners.” Instead, they are in constant contact with the research community, bringing people together to brainstorm new directions, understanding emerging themes, matching these themes to military needs, “betting on the right people,” connecting disconnected communities, positioning competing technologies against one another, and maintaining the
birds-eye system-integrating perspective that is critical to integrating disparate activities across the nation’s innovation ecosystem.
Finally, throughout these activities—whether bringing together members of research communities that may not normally interact or funding an entire suite of technologies necessary to achieve an integrated outcome—DARPA program managers contribute a system-level perspective to organizing national R&D. DARPA has been able to cultivate such communities over the long term by continuing to fund activities in focused areas until the technology and the community can be sustained outside the walls of the agency. Such accomplishments as the early development of the Internet, Global Positioning Systems, and autonomous vehicles were not the result of a single successful project or program, but of the cultivation of a community across a pyramid of interdependent technologies (Fuchs, 2010) over an extended period of time.
Examples from ARPA-E
The committee focused on two programs in which ARPA-E is bringing together communities of researchers that have tended not to interact in the past. The committee was unable to identify instances in which ARPA-E programs are attempting to play a longer-term role in their evolution into new, lasting research communities.
Advanced Management and Protection of Energy storage Devices (AMPED)
ARPA-E’s AMPED program is an example of a case in which the agency has created a community with longer-term potential. The program was designed to foster advanced sensing, control, and power management systems so as to change conventional attitudes toward battery management. Ilan Gur, a program director who participated in AMPED, said that most advanced battery research had focused on new chemistries, so the AMPED program was devised to use novel modeling, sensing, and control technologies to yield lighter, cheaper batteries, producing nearly as great an impact as new chemistries (Gur, 2015).
Gur (2015), for example, recalled for the committee that ARPA-E talked to people outside the field of battery research in developing information about acoustic monitoring techniques usually associated with geological research but with possible application in monitoring battery performance. The agency kicked off the AMPED program with a 2-day meeting whose participants included not only the project performers, but also representatives from academia, industry, government, and the national laboratories (ARPA-E, 2013a). In March 2015, ARPA-E convened a 3-day annual program review meeting for AMPED whose attendees included project teams and key stakeholders in advanced battery management from academia, industry, and government (ARPA-E, 2015c). At the meeting, participants had opportunities to engage in discussions with the AMPED teams on topics that included how to position these new battery
management systems for strategic investors (ARPA-E, 2015c). Gur (2015) concluded that ARPA-E catalyzed new research in the area of battery management systems that generated excitement in the private sector.
Full-spectrum Optimized Conversion and Utilization of Sunlight (FOCUS)
ARPA-E’s FOCUS program is an example of how the agency has convened different research communities that previously existed in separate spheres. The program was aimed at developing new hybrid solar energy converters and hybrid energy storage systems that would accept both heat and electricity from variable solar sources. A subsidiary objective of the program was to form a diverse research community of experts in the concentrating solar power (CSP) and solar photovoltaics (PV) fields that would innovate together through this program (ARPA-E, 2014d). These include CSP mechanical engineers, PV semiconductor materials and device scientists, optics/photonics experts, chemists, low-cost manufacturing experts, and system integrators (ARPA-E, 2014d). CSP and PV are such fundamentally different technologies that Howard Branz, a program director who participated in the program, recalled for the committee that prior to FOCUS, the “PV and solar thermal communities were totally separate, didn’t talk to each other” (Branz, 2015).
ARPA-E Energy Innovation Summit and Kickoff Meetings
In testimony before the committee, several ARPA-E representatives mentioned the agency’s annual Energy Innovation Summit in the context of community building. Additionally, the committee attended the kickoff meetings of several projects. The committee observed that these events provide venues for communication, but likely do not themselves indicate creation of a community. There is little evidence that a community would be sustained in the absence of funding, and the events are focused on funded projects, not the creation of new directions. The committee found no evidence of communities spontaneously forming without the direction and influence of ARPA-E program directors, an observation that further reinforces the pivotal role of the program directors within ARPA-E.
IMPROVING PUBLIC AWARENESS OF ARPA-E ACHIEVEMENTS
While the committee noted significant outcomes emerging from projects funded by ARPA-E, these achievements may not be sufficiently appreciated outside of energy research communities within companies, government agencies, and academic institutions. While many exhibitors at the ARPA-E Energy Innovation Summit and participants in program meetings attended by the committee appeared to have the capability to discuss the technical aspects of
their projects, the compelling nature of those projects and how they could be transformational often was unclear. The committee is concerned that the inability to deliver compelling messaging consistently in the language of the general public limits ARPA-E’s ability to describe its programs and projects to the broader set of policy makers and the public at a time when energy issues often are at the forefront of public debates. The National Science Foundation has long recognized education as a key element of its mission, enabling it to share excitement and inspire the next generation of breakthrough science performers. Such a function may be even more relevant for ARPA-E as the customers of an innovation funded by the agency often are business leaders who lack deep science or engineering knowledge.
It is already difficult to demand that performers simultaneously be experts in their own technical fields, adept in business, and skilled in both project management and government relations while communicating work that crosses traditional boundaries of technical training. The committee notes that ARPA-E has used videos and webinars through the ARPA-E University portion of its website (http://arpa-e.energy.gov/?q=arpa-e-site-page/arpa-e-university) as part of its training efforts, and this might be one avenue for facilitating such communication training for performers. However, expecting these same people to be skilled at explaining technology to a general audience—for example, relating narrow innovations in plasma physics to daily life—is unreasonable. Engaging communications experts would be a valuable way to improve awareness of ARPA-E’s achievements and dispel myths about what is essential to U.S. economic and energy security. Such experts could assist in developing the compelling and instructive graphics needed to show the potential role of specific programs in producing transformational breakthroughs. Their efforts could clarify that diverse metrics are necessary to understand the success of individual projects and programs and help dispel the idea that there may be a single metric suitable for evaluating all projects and programs.
ADDITIONAL METRICS THAT MAY BE USED FOR TECHNOLOGY ASSESSMENTS
As discussed earlier in this chapter, an important part of this study was developing systematic metrics for the social, economic, and environmental impacts of ARPA-E projects, but given the relatively short timeframe since ARPA-E commenced operations, there was little measurable evidence to use for assessing performance against the agency’s long-term objectives. Moreover, there are a variety of important outcomes that are difficult to assess including:
- funding critical science that is essential to achieving energy transformation and/or discovering new directions;
- demonstrating the viability of existing scientific theory that is critical to energy transformation;
- demonstrating that particular technological pathways should not be pursued; and
- funding individual components of the energy system that may be underfunded and could hold back advancement throughout the system because of interdependencies with other components, including funding white space technologies, funding white space where other agencies or companies are not working, funding a portfolio of projects to ensure compatibility across the energy system, and funding activities for an existing technology to drive down its costs or the cost-to-performance ratio and enable its deployment to be transformational.
Other metrics that should be considered for a longer term assessment include metrics of commercialization, technological or environmental impacts, lessons learned, and the creation of novel collaborations or new research communities. One metric of success is a product based on the technology that generates revenue. Another metric of commercialization could be revenue from licensing patents if the firm chooses not to manufacture the product itself. (Arora, 1995; Cannady, 2013; Chan, 2014). Given the early stage of development of most projects and their 3-year duration, commercialization would not be an appropriate metric for many projects while they are supported by ARPA-E. Nonetheless, commercial success is one important objective for the technologies supported by ARPA-E and necessary for any technology to transform the marketplace. However, long lags exist between basic and translational research support and commercial outcomes (Popp, 2015).
ARPA-E should also keep track of the lessons learned about what works and what does not work. Knowledge about particular technological pathways that should not be pursued is as important for the advancement of science and technology and successful technologies. These results are often more difficult to publish, if published at all. Finally, the success of ARPA-E’s focused technology programs at spurring formation of new communities of researchers in specific fields is another metric to consider in assessing the agency. Studies could be done to evaluate whether research projects funded by ARPA-E are more likely to involve novel (e.g., first-time) collaborators relative to projects funded by other energy agencies or industry (Colatat, 2015). When metrics are not available, case studies can provide qualitative information on project and program outcomes.
ARPA-E has in place an extensive data gathering and recordkeeping system at the project level that can track and monitor internal metrics and facilitate active program management. It has a less extensive system that can collect, track, and report publicly available high-level innovation metrics such as publications, funding from other sources, and intellectual property information,
including disclosures and patents. Moreover, these traditional innovation metrics are poorly suited to evaluating ARPA-E’s activities. What really is needed is a framework that maps the linkages of technical goals of each program through intermediate outcomes, such as traditional innovation metrics, to the agency’s statutory mission, means, and goals.
Developing and implementing such a framework for impact evaluation would be very valuable and important for ARPA-E to undertake as soon as practicable, providing the agency greater ability to assess and demonstrate its value and impact. Critically, ARPA-E should not delay implementation. The longer the agency waits, the more difficult it will be to implement such a framework and the less valuable it will be. If too much time is lost, it will become more difficult if not impossible to assess program impacts in a way that allows for meaningful responses to what is learned from utilizing the framework—i.e., by making tweaks to systems or revisions intended to maximize impact (Davis et al., 2008; Link et al., 2012).
The agency could link data from its robust internal database of project-level metrics to program-level goals, including indicators of commercial and noncommercial outcomes over the short and long terms; connect those goals to observable innovation metrics; and then translate those metrics into progress toward achieving the agency-level mission and goals. Such a framework would need to include a system for tracking performers postfunding for at least 10 years, and very likely longer, to capture technologies that are transferred in arms-length transactions along with other ways of observing technology deployment. As of this writing, the agency already has taken steps supportive of creating such an impact assessment framework, such as creating and filling in September 2016 a new staff position of Senior Technical Advisor for Impact and Assessment.
SUMMARY OF FINDINGS AND RECOMMENDATIONS
Finding 4-1: ARPA-E has funded research that no other funder was supporting at the time. The results of some of these projects have prompted follow-on funding for various technologies, which are now beginning to enter the commercial market.
As a result of ARPA-E’s activities, companies have been founded, patents applied for and issued, and papers written about work performed in projects funded by the agency. Dead ends also have been identified, which, if properly documented, would provide important reference data for later researchers. In general, the agency was not solely responsible for the development of these
technologies, as most of them originated before ARPA-E funding was forthcoming. In a number of cases, however, ARPA-E provided crucial early-stage funding and frequent technical and market analysis expertise that enabled a company or idea to pass at least partially through the “valley of death,”4 as ARPA-E was established to do. It is too early to determine conclusively whether any technology supported by ARPA-E will be truly transformational; this assessment can be made only retrospectively, after the passage of a decade or more.
Finding 4-2: The projects ARPA-E has funded support its statutory mission and goals.
Measuring and assessing the impact of any one project or program on ARPA-E’s statutory mission and goals is very difficult. Having a longer time series of data would be helpful to this end, but would not eliminate the inherent challenges since Congress’s goals for the agency are highly diffuse and affected by disparate factors. At this early stage, however, it is still possible to gain some understanding of whether ARPA-E’s activities have contributed to supporting the agency’s mission and goals. Currently available qualitative and quantitative evidence suggests that ARPA-E has funded projects that support its statutory mission and goals. Several supported technologies have received follow-on private funding and appear to be on a trajectory toward commercialization of products—with at least one already in the marketplace—that will impact the energy sector and the environment. The committee’s review of programs and selected projects indicated that the agency created some funding programs for technology areas that the private sector or other federal agencies would have been unlikely to pursue. One example is the program on wide-bandgap, high-voltage solid state electronics. ARPA-E also has funded projects in areas that clearly overlapped with areas supported by the private sector and other funding agencies. In some of those cases, ARPA-E was a key accelerator.
Finding 4-3: While 6 years is not long enough to produce observable evidence of widespread deployment of funded technologies, there are clear indications that ARPA-E is making progress toward its statutory mission and goals.
ARPA-E has provided crucial early-stage funding for some performers. One quarter of these projects have received follow-on funding for continued work on technologies that now are poised to enter the commercial market. Many funded projects have produced publications in peer-reviewed journals, and some
4 As noted in “10 Ways for Startups to Survive the Valley of Death,” the term “valley of death” is common in the start-up world, referring to the difficulty of covering the negative cash flow in the early stages of a start-up, before its new product or service is bringing in revenue from real customers (Zwilling, 2013).
have received patents. However, most transformative energy technologies require many years, often several decades, to go from nascent research to first marketable product. ARPA-E projects are funded for roughly 3 years, a far shorter time.
Two important observations can be derived from these facts. First, there is an inherent tension between the pressure to demonstrate success by bringing a product to market quickly and funding of early-stage technologies with the potential to be transformative in the face of the many years of work nearly always required to realize a technology’s full impact. Second, after 6 years of operation, there exist fewer than 4 years of projects to serve as evidence of progress toward ARPA-E’s mission and goals. These two observations speak both to the need to consider ARPA-E’s value in context and over a duration that is well aligned with the agency’s mission, and to the need for more and better data for fully understanding that value. Nonetheless, the committee is confident that the evidence obtained and analyzed for this study indicates that ARPA-E has grown from a concept into a functioning organization and has made demonstrable progress toward achieving its mandates.
Recommendation 4-1: Policy makers should recognize that there is limited evidence to date on transformational impacts emerging from ARPA-E, given the short time since ARPA-E began.
The agency had been in operation for only 6 years when this study was undertaken. By comparison, as noted above, truly transformational technologies emerge over much longer time periods. The committee emphasizes the most fundamental tension to be managed by ARPA-E between having a short-term impact on a technology within the 3-year funding timeframe while producing transformational technologies.
ARPA-E’s was expressly created to achieve long-term environmental, security, and competitiveness goals. It was structured to fund and manage R&D undertaken by entities other than the agency rather than undertaking its own R&D activities. Because the agency is tasked with seeking out transformational technological advances, it has necessarily utilized novel operational benchmarks to try to accomplish its goals. Together these findings mean that any assessment of the agency will encounter a well-known problem in R&D management: since sufficient time has not passed for outcomes to have become evident, an assessment cannot draw strong conclusions unless the enterprise is in an extreme situation, such as doing very badly. These same findings also make clear that ARPA-E is not in an extreme situation. The agency is not failing and is not in need of reform. In fact, attempts to reform the agency—such as pressure to show short-term successes rather than focusing on its long-term mission and goals—would pose a significant risk of harming its efforts and chances of achieving its mission and goals.
ARPA-E may eventually be assessable by the impacts on energy and environment of the technologies and products it has enabled, much as the Defense Advanced Research Projects Agency (DARPA) is judged. To gather data on impacts will require patience, however, as many years still must pass before the full impacts of ARPA-E’s investments will be known. ARPA-E projects develop technologies for applications in the energy sector, a sector that is generally capital-intensive, risk-adverse, and adopts innovations over long time horizons.
Despite these limitations on an early assessment, currently available qualitative and quantitative data suggest that ARPA-E is pointed in a positive direction—it has funded projects that support its statutory mission and goals—and there are signals that indicate potential for future success. A review of programs and selected projects suggests that the agency has identified technology areas that neither the private sector nor the federal government would likely have pursued alone, funded projects when no other entity would, and provided crucial early-stage funding for some projects. One example is the program on wide-bandgap, high-voltage solid state electronics. In other cases, ARPA-E has accelerated the funding of projects in areas also supported by other public funding and private investors.
Maintaining Focus on Funding High-Risk Technologies and Innovations
Finding 4-4: One of ARPA-E’s strengths is its focus on funding high-risk, potentially transformative technologies and overlooked, “off-roadmap” opportunities pursued by neither private firms nor other funding agencies, including other programs and offices within DOE.
This finding describes the importance for ARPA-E of seeking high-risk, potentially transformative technologies and “off-roadmap” opportunities that have been overlooked or ignored by both private firms and other offices and programs at DOE or other funding agencies. This makes ARPA-E’s focus on potentially high-impact innovations in both cases vital to accomplishing the agency’s mission. ARPA-E’s underlying organizational features—including encouraging its program directors to seek potentially high-impact projects and recognizing that many of its projects will produce only valuable knowledge, including knowledge of research pathways that should not be further pursued, but not commercialized products—is distinctive within DOE.
Maintaining this focus will be one of the greatest challenges for ARPA-E in the future. It is not guaranteed that ARPA-E will be able to maintain a culture of pursuing high-risk but potentially transformative technologies and research pathways characterized as novel or significantly underexplored as the energy technology landscape evolves. ARPA-E leadership and the secretary of energy should actively work to sustain this culture. ARPA-E should continue to balance
its overall portfolio between innovations that appear to have the potential to be transformative and other important opportunities that are being ignored despite their potential for impact.
Recommendation 4-2: The director of ARPA-E should continue to promote and maintain a high-risk culture within the agency. Means to this end include periodic reassessment to ensure that the principles that drive support for high-risk projects are being maintained.
Recommendation 4-3: ARPA-E should continue to use processes designed to identify and support unexplored opportunities that hold promise for resulting in transformational technological advances.
When conducting future evaluations, it is important for ARPA-E and policy makers to remember that the agency is tasked with supporting “high-risk/high-reward” research. This implies a skewed distribution of research outcomes: many projects and even programs may not produce tangible outcomes measured against the agency’s overall goals and means, but the overall return of the whole portfolio may be positive because of a few highly successful projects.
Clarification of ARPA-E’s Mission
Finding 4-5: Some of the language used by ARPA-E creates an impractical expectation and mission that are not necessarily in the agency’s original authorizing statute.
Problematic expectations expressed to the committee included that ARPA-E should be the only federal agency funding a technology and that all projects should be transformational.
Recommendation 4-4: ARPA-E should be careful not to misinterpret or extend its interpretation of its original authorizing statute, whose careful language is appropriate to the agency’s mission and goals.
For example, ARPA-E should not be measured by or consider its mission to be serving as the only federal agency funding a technology or funding only transformational technologies. The agency may best be served by a balanced portfolio of projects aimed at attaining both large and small successes, not only “home runs.” Funding white space can be orthogonal to funding transformational technologies, including “filling in” where other agencies are not working. And driving down costs (or the cost-to-performance ratio) for an
existing technology to enable its adoption could be transformational to the energy system.
ARPA-E’s success also should not be measured based on whether a funded project has reached the market. Equally important outcomes include (1) funding critical science that is essential to achieving energy transformation and/or opening up new directions, (2) demonstrating the physical viability of existing scientific theory that is critical to energy transformation, (3) demonstrating that particular technological pathways should not be pursued, and (4) funding individual components of the energy system that may be underfunded and, for example, could hold back advancement throughout the system because of interdependencies.
High-Touch Management and Reporting Requirements
Finding 4-6: The high-touch nature of project management at ARPA-E is a hallmark of the agency and has been praised by performers. That said, quarterly reporting in terms of required written documentation is currently challenging, depending on the technical context. Given that quarterly written reports are offset in time with site visits from program directors and their teams, a project performer may end up having 8–10 direct interactions with ARPA-E per year.
Recommendation 4-5: ARPA-E should consider streamlining the quarterly reporting process so it consists of presentations to the program directors and their teams, with only the fourth-quarter/annual report providing full written details. Doing so would allow the agency to maintain close contact with performers while relieving some of the burdens on them.
Documenting Lessons Learned
Finding 4-7: Through its projects and programs, ARPA-E is accumulating not only technical knowledge of what is working and has promise, but also potentially very useful information on what does not work that can be an important addition to ARPA-E documentation.
As a condition of funding, the agency may wish to require that each performer document what does not as well as what does work. Further, ARPA-E should explore publishing an annual, easily referenced lessons learned document focused on what performers learned does not work, for the benefit of the
scientific community, to aid in the review of existing programs, and to serve as a reference for future programs. In this last case, ARPA-E may need to consider producing both an internal document with confidential information available only to its program directors and a sanitized public document with no performer-confidential information.
Recommendation 4-6: ARPA-E program directors should compile a document or other repository of lessons learned on all projects, including both positive and negative outcomes.
Improving Awareness of ARPA-E’s Role among the Public and Policy Makers
Finding 4-8: ARPA-E is in many cases successfully enhancing the economic and energy security of the United States by funding transformational activities, white space, and feasibility studies to open up new technological directions and evaluate the technical merit of potential directions. However, ARPA-E is doing a poor job of creating awareness of these very real successes while at the same time holding itself to a metric of success that is not aligned with its authorizing statute or fundamentally essential to the energy and economic security of the United States (see Finding 4-5).
The process of developing an energy technology is a complex story involving many actors and events. Successful energy technologies often have interesting stories with long pathways from basic scientific discoveries and engineering innovations to mainstream products. Those pathways are often complex, convoluted, and unique. Succinct language and visual forms of communication would be quite valuable in making the stories and processes accessible to a range of audiences, especially non-technical ones. For the SWITCHES program, for example, it would be helpful to have a one-page explanation, written for general audiences, relating why this program matters and its potential transformational impact should it prove successful. One way ARPA-E could increase and improve its communication for non-technical audiences would be to engage experts in communication of popular science who could train staff or perhaps help produce materials. Regardless of the story being told, ARPA-E should ensure that key audiences such as Congress, the secretary of energy, and other members of the administration clearly understand the value of ARPA-E’s activities.
Recommendation 4-7: ARPA-E should increase and improve its communication for non-technical audiences, including the impact of its activities, the diversity of appropriate metrics to judge the success of individual projects and programs, and the fact that no single metric is appropriate for this purpose.
Improving ARPA-E’s Measurement and Assessment of Its Impact
Finding 4-9: ARPA-E is not yet able to assess the full extent to which it has achieved its statutory mission and goals. However, it is in a good position to develop a framework for prospectively mapping data on project selection and management to mission success and goal achievement.
ARPA-E’s active project management processes include an extensive data gathering and recordkeeping system to monitor project status through internal metrics such as progress toward milestone, but does not extensively collect, track, and report publicly available high-level innovation metrics such as publications; funding from other sources; and intellectual property information, including disclosures and patents. The agency is in a good position, though, to create a mapping framework that links the technical goals of each program through intermediate outcomes, such as traditional innovation metrics, to the agency’s mission, means, and goals.
The agency could link data from its robust internal database of project-level metrics to program-level goals, including indicators of commercial and noncommercial outcomes over the short and long terms; connect those goals to standard, observable innovation metrics; and then translate those metrics into progress toward achieving the agency-level mission and goals. Such a framework would need to include a system for tracking performers for at least 10 years following the end of their funding from ARPA-E, and very likely longer, to capture technologies that are transferred in arms-length transactions along with other ways of observing technology deployment.
It is critical to keep in mind that ARPA-E itself is, and will continue to be, the entity best positioned to establish this framework, what to measure, and how to incorporate the information learned thereby into agency operations and decision making. The agency should not be bound by legislative mandate that specifies methodology or inclusions for any future impact assessments. ARPA-E needs to be allowed to remain nimble so it can respond quickly to new information. ARPA-E already is known for its internal reflection and efforts to improve its operations and outcomes. Designing and implementing such a framework would make the agency an exemplar of public agency self-evaluation and evidence-based improvement. ARPA-E should be emboldened and
empowered to develop and implement such a framework in a way that best serves its mission and goals.
Recommendation 4-8: The ARPA-E director and program directors should develop and implement a framework for measuring and assessing the agency’s impact in achieving its mission and goals.