The National Academies Press

Currently Skimming:

Pages 47-90

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.

From page 47... ... An Assessment of ARPA-E 3 ARPA-E's Internal Operations: Culture, People, and Processes In accordance with its statement of task (Box 1-1 in Chapter 1) , the committee undertook an operational assessment to "appraise the appropriateness and effectiveness of the agency's structure to position it to achieve its mission and goals." This chapter describes and assesses ARPA-E's internal operations, addressing the following specific charges in the statement of task: • Evaluate ARPA-E's methods and procedures to develop and evaluate its portfolio of activities; • Examine appropriateness and effectiveness of ARPA-E programs to provide awardees with non-technical assistance such as practical financial, business, and marketing skills; • Assess ARPA-E's recruiting and hiring procedures to attract and retain qualified key personnel; • Examine the process, deliverables, and metrics used to assess the short and long term success of ARPA-E programs; • Assess ARPA-E's coordination with other Federal agencies and alignment with long-term DOE objectives; • Provide guidance for strengthening the agency's structure, operations, and procedures; • Evaluate, to the extent possible and appropriate, ARPA-E's success at implementing successful practices and ideas utilized by the Defense Advanced Research Projects Agency (DARPA) Read the entire page →
From page 48... ... An Assessment of ARPA-E 48 AN ASSESSMENT OF ARPA-E DEFINING ORGANIZATIONAL FEATURES OF ARPA-E Through the course of its deliberations and analyses of the evidence gathered for this assessment, the committee found that ARPA-E benefits from three defining organizational features: • The director exercises technical and leadership skills that enable a culture of empowerment to be sustained. • ARPA-E's program directors are empowered with the authority, responsibility, and ability to make program- and project-related decisions. Read the entire page →
From page 49... ... An Assessment of ARPA-E ARPA-E'S INTERNAL OPERATIONS 49 Innovation Summit to identify and observe key stakeholder interactions. Quantitative methods included examination of descriptive statistics and regression analysis of data related to the program directors' roles, responsibilities, and activities. Read the entire page →
From page 50... ... An Assessment of ARPA-E 50 AN ASSESSMENT OF ARPA-E that relies on program director discernment and discretion rather than strict adherence to a particular bureaucratic process; to fund projects that may appear riskier than others but are deemed to hold greater potential for impact if successful; and to engage actively in ongoing projects by altering project milestones, budgets, and timelines when necessary. This quantitative analysis was supplemented by a qualitative analysis drawn from discussions with the founding ARPA-E director; current and former program directors; leaders of other DOE offices and programs, including those at the national laboratories; presentations at the study committee's meetings by ARPA-E leadership, DARPA leadership, and other outside experts; 2 and case studies prepared by the committee (Appendix D) Read the entire page →
From page 51... ... An Assessment of ARPA-E ARPA-E'S INTERNAL OPERATIONS 51 A critically important difference between ARPA-E and other DOE offices is its statutory mandate to identify and promote ideas with the promise of being revolutionary, and to support ones that industry is unlikely to support alone. In other words, ARPA-E was created to tolerate risk and foster activities that would lead to transformative technology innovations and to do so by being structured and operating in ways that are innovative for a DOE program. Read the entire page →
From page 52... ... An Assessment of ARPA-E 52 AN ASSESSMENT OF ARPA-E decisions made under uncertainty" (Lyons et al., 2007, p. Read the entire page →
From page 53... ... An Assessment of ARPA-E ARPA-E'S INTERNAL OPERATIONS 53 pitched is for no one to challenge it. Program directors also expect that the agency director will engage with them, providing critique and feedback regarding programs and projects, and do so constructively and in a way that program directors do not fear personal attack or reprisal. Read the entire page →
From page 54... ... An Assessment of ARPA-E 54 AN ASSESSMENT OF ARPA-E milestones. Those data were provided by ARPA-E and anonymized and aggregated to protect sensitive information. Read the entire page →
From page 55... ... An Assessment of ARPA-E ARPA-E'S INTERNAL OPERATIONS 55 attracting talented personnel who can achieve an organization's mission and goals (Szczepańska-Woszczyna, 2015) Read the entire page →
From page 56... ... An Assessment of ARPA-E 56 AN ASSESSMENT OF ARPA-E technology field and have the ability to perform the research (Bonvillian and van Atta, 2011) Read the entire page →
From page 57... ... An Assessment of ARPA-E ARPA-E'S INTERNAL OPERATIONS 57 Recruiting Program Directors ARPA-E's primary method of recruiting program directors is by word of mouth. Current and alumni program directors suggest individuals that ARPA-E may wish to consider, and they inform their professional networks of whether they think ARPA-E is a valuable place to work. Read the entire page →
From page 58... ... An Assessment of ARPA-E 58 AN ASSESSMENT OF ARPA-E appointment meant that to make an impact, they would need to work hard on identifying and supporting promising ideas. They stated they found this a motivating factor, and contrasted it with the likelihood of building constituencies or maintaining independence from corporate or government bureaucracy that often accompanies long-term positions. Read the entire page →
From page 59... ... An Assessment of ARPA-E ARPA-E'S INTERNAL OPERATIONS 59 unreasonable to expect that a technology could move from concept to market within the 3-year (or less) timeframe of an ARPA-E project. Read the entire page →
From page 60... ... An Assessment of ARPA-E 60 AN ASSESSMENT OF ARPA-E Allen Hamilton, 2012) Read the entire page →
From page 61... ... An Assessment of ARPA-E ARPA-E'S INTERNAL OPERATIONS 61 1962) Read the entire page →
From page 62... ... An Assessment of ARPA-E 62 AN ASSESSMENT OF ARPA-E for award negotiation. The overall selection rate resulting from this two-step application process was 5 percent. Read the entire page →
From page 63... ... An Assessment of ARPA-E ARPA-E'S INTERNAL OPERATIONS 63 percent for some of the focused programs. 13 Figure 3-3 shows the selection rate from concept paper through full application and on to selection for award negotiation for many of the programs that the committee's consultants were able to analyze. Read the entire page →
From page 64... ... An Assessment of ARPA-E 64 AN ASSESSMENT OF ARPA-E FIGURE 3-4 Selection of full applications by focused and OPEN programs. NOTES: Data are percentage of 2,335 selected applications submitted to ARPA-E through December 31, 2015, by program. Read the entire page →
From page 65... ... An Assessment of ARPA-E ARPA-E'S INTERNAL OPERATIONS 65 "discretionarily selected" for this analysis.) Almost half (49 percent) Read the entire page →
From page 66... ... An Assessment of ARPA-E 66 AN ASSESSMENT OF ARPA-E FIGURE 3-5 Evidence of program director discretion in project selection (RANGE program) : Projects selected with a high score and discretionarily selected projects. Read the entire page →
From page 67... ... An Assessment of ARPA-E ARPA-E'S INTERNAL OPERATIONS 67 Recent research supports ARPA-E's selection process as it balances the trade-off between valuable information to be gained and potential bias that comes with peer review. Reviewers may assign lower scores because of bias against novel ideas, the very thing ARPA-E needs to identify. Read the entire page →
From page 68... ... An Assessment of ARPA-E 68 AN ASSESSMENT OF ARPA-E Active Project Management: Quantitative Evidence Once projects have been selected for funding, each project team enters a negotiation phase with ARPA-E during which the parties agree upon financial terms and project-level milestones. Funding levels and project lengths vary significantly across projects. Read the entire page →
From page 69... ... An Assessment of ARPA-E ARPA-E'S INTERNAL OPERATIONS 69 when a milestone was modified, the technical target was updated, but within the database the "old" milestone was not deleted and replaced with a "new" one incorporating the updated criteria. For this analysis, projects were further subdivided into two groups based on the percentage of measured quarters that involved milestone creation or deletion. Read the entire page →
From page 70... ... An Assessment of ARPA-E 70 AN ASSESSMENT OF ARPA-E TABLE 3-1 Transition Matrix for Overall Status Statust+1 Status Red Yellow Green Total Statust Red 66% 29% 5% 100% Yellow 8% 71% 21% 100% Green 1% 21% 78% 100% quarter to the next. The table reveals a high degree of rating persistence. Read the entire page →
From page 71... ... An Assessment of ARPA-E ARPA-E'S INTERNAL OPERATIONS 71 track of the category of milestone changed, to conduct this analysis they would also need to record systematically two other attributes of the milestone changes. First, there would need to be a detailed record of precisely why milestones were changed. Read the entire page →
From page 72... ... An Assessment of ARPA-E 72 AN ASSESSMENT OF ARPA-E project lengths typically range from 2 to 3 years, many projects will experience a change in program director. A majority of projects examined for this analysis (70 percent) Read the entire page →
From page 73... ... An Assessment of ARPA-E ARPA-E'S INTERNAL OPERATIONS 73 controlling for organization type, initial award amount, and initial project length, the likelihood of a project's filing a patent application increased if the program director changed (Goldstein and Kearney, 2016, Table 17) Read the entire page →
From page 74... ... An Assessment of ARPA-E 74 AN ASSESSMENT OF ARPA-E Comparisons to and Coordination with DARPA, Other Offices at DOE, and Other Federal Agencies Because ARPA-E was patterned after DARPA, many of the agency's processes have their origins at DARPA. While other DOE offices have utilized some processes and practices similar to those found at ARPA-E, since the agency's creation, some offices have taken steps to adapt ARPA-E's processes to their own contexts in a more systematic fashion. Read the entire page →
From page 75... ... An Assessment of ARPA-E ARPA-E'S INTERNAL OPERATIONS 75 the research in a particular direction and increasing knowledge flow within the research community. DARPA's offices will support activities and focused funding programs to pursue meeting a strategic challenge for as long as doing so makes sense, often for decades. Read the entire page →
From page 76... ... An Assessment of ARPA-E 76 AN ASSESSMENT OF ARPA-E adopt an open system like DARPA's without a larger budget and less budgetary uncertainty. Table 3-2 summarizes the most significant points of comparison between ARPA-E and DARPA. Read the entire page →
From page 77... ... An Assessment of ARPA-E ARPA-E'S INTERNAL OPERATIONS 77 TABLE 3-2 Continued Attribute DARPA ARPA-E Set Aside Full No: annual appropriations Yes: to mitigate uncertainty of Multiyear Project sufficiently certain to enable future budgets Funding in Year annual allocations One Procedures and Processes Orchestration of Yes: specific strategy for all Aspirational, but not yet a Technology office directors and program specific strategy Directions managers in support of achieving technical outcomes and agency mission Funding of Entire Yes No: budget too small Platform of Technologies Necessary to Achieve Particular Goals Program Yes: no peer review Yes: program directors utilize Managers/Directors information from peer review, Have Autonomy in but are not obligated to follow Funding score cutoff or "rack and stack" Recommendations Effort to Build New Yes: specific strategy for all Unclear: at present, ad hoc with Research office directors and program anecdotal evidence Communities managers in support of achieving technical outcomes and agency mission; aim is to connect previously disconnected researchers to enable otherwise infeasible research Active Yes Yes: regular site visits, review Management of of data with performers, Projects suggestions regarding technical directions of research and project team Go/No-Go Yes, but how much has varied Yes: legislatively required; Technical by decade and program quarterly review of progress Milestones for toward achieving milestones Projects and feasibility (Continued) Copyright National Academy of Sciences. Read the entire page →
From page 78... ... An Assessment of ARPA-E 78 AN ASSESSMENT OF ARPA-E TABLE 3-2 Continued Attribute DARPA ARPA-E Program or Yes: focused on military Yes: specific program and Procedures to application; Adaptive personnel to assist awardees Ensure Awardees Execution Office seeks to with orienting to eventual Plan/Prepare for accelerate transition from market entry, identifying Eventual DARPA project to Defense commercial applications, and Commercialization Department capability; focus finding sources of funding to on commercializing for defense continue past ARPA-E award with increasing interest in dual- period use potential; most applications envision some application of technology; expectation that good technologies and teams will find new funding sources to continue past DARPA granting period since "DARPA is not in the business of sustaining the technology" Intensive Yes: intensive problem-focused No Gatherings of gatherings of thought leaders to Entire Research and determine specific problems to Innovation solve Community to Identify Priorities and Directions Inducement Prizes Yes No, presumably because of and Competitions, budget limitations and Grand Challenges Program Features Programs Focused Yes Yes on Specific Technical Outcomes with Measurable Goals Broad Area or Yes Restricted: OPEN program is Open Programs to made available roughly every Capture Promising 3 years, with an application Ideas That Do Not window of approximately Fit within Particular 120 days Focused Programs Copyright National Academy of Sciences. All rights reserved. Read the entire page →
From page 79... ... An Assessment of ARPA-E ARPA-E'S INTERNAL OPERATIONS 79 TABLE 3-2 Continued Attribute DARPA ARPA-E Fund Competing Yes Yes Technologies Aimed at Solving the Same Problem Project Management Features Suggest Milestone Yes Modifications Based on Findings from the Project Regular Contact Yes Yes, at least quarterly with Research Teams Suggest Milestone Yes Modifications Based on Findings from Other Projects across Program and Agency Suggest Teaming Yes Partners Suggest Changes in Yes Personnel or Subcontractors Comparing ARPA-E's Mission and Goals with Other DOE Research Offices and Programs Although ARPA-E funds some projects that fall within broad categories of technologies also funded in other parts of DOE, it funds specific research ideas or individual technology ideas that are out of scope, at different stages of development, or have different risk profiles for other parts of DOE. The Office of Science focuses its support on fundamental scientific research. Read the entire page →
From page 80... ... An Assessment of ARPA-E 80 AN ASSESSMENT OF ARPA-E inform potential applicants that the agency funds "applied research and development" as defined by the Office of Management and Budget. Its funding announcements have also directed readers to contact the Office of Science if their project is for basic research, while directing those with "projects focused on the improvement of existing technology platforms along defined roadmaps" to consider seeking support from one or more of DOE's applied offices (ARPA-E, 2015a) Read the entire page →
From page 81... ... An Assessment of ARPA-E ARPA-E'S INTERNAL OPERATIONS 81 funded projects, the Office of Fossil Energy funded 3 (out of 15) of the projects (NETL, n.d.) Read the entire page →
From page 82... ... An Assessment of ARPA-E 82 AN ASSESSMENT OF ARPA-E white spaces in which ARPA-E projects could complement what EERE is doing in such areas as grid-scale power storage and batteries for electric vehicles. ARPA-E's program directors and EERE's program managers coordinate these efforts without formal institutional support. Read the entire page →
From page 83... ... An Assessment of ARPA-E ARPA-E'S INTERNAL OPERATIONS 83 with specific milestones so the awards can be terminated early if they cannot meet those milestones. 19 Office of Fossil Energy An FE representative stated that "coordination between FE and ARPA-E has been poor and partial. Read the entire page →
From page 84... ... An Assessment of ARPA-E 84 AN ASSESSMENT OF ARPA-E such practices because, unlike ARPA-E program directors who are with the agency for 3 years, OE program managers typically have been with the office for a very long time and have developed constituents, inhibiting robust debate. Comparisons with Other Federal Departments and Agencies ARPA-E engages with over 70 organizations and agencies within the federal government. Read the entire page →
From page 85... ... An Assessment of ARPA-E ARPA-E'S INTERNAL OPERATIONS 85 settings. Their limited tenure and freedom to pursue ideas with potential to have a large impact empowers them to bridge market failures and create value that can help ensure that the "United States maintains a technological lead in developing and deploying advanced energy technologies." 22 Recommendation 3-1: ARPA-E should preserve its distinctive and flexible management approach that empowers program directors and stresses active project management. Read the entire page →
From page 86... ... An Assessment of ARPA-E 86 AN ASSESSMENT OF ARPA-E Recommendation 3-3: ARPA-E should reconceptualize its "technology-to-market" (T2M) program to account for the wide variation in support needed across programs and performers with respect to prospective funding, commercialization, and deployment pathways. Read the entire page →
From page 87... ... An Assessment of ARPA-E ARPA-E'S INTERNAL OPERATIONS 87 contrast, ARPA-E is widely seen as having a nimble, efficient, and innovation supporting culture. Several DOE offices have adapted some ARPA-E practices in an effort to improve their own operations. Read the entire page →
From page 88... ... An Assessment of ARPA-E 88 AN ASSESSMENT OF ARPA-E reviewer scores, often ranking applications by scores and utilizing a strict cutoff that does not allow for discretion on the part of the agency or program directors. Strong and consistent evidence indicates that projects selected through ARPA-E's process have the potential to yield measurable outcomes at least as good as, if not better than, those of projects that would have been selected had less discretion been allowed. Read the entire page →
From page 89... ... An Assessment of ARPA-E ARPA-E'S INTERNAL OPERATIONS 89 personnel or subcontractors. They regularly engage with performers to discuss a project's technical approach and collaborate to revise it based on results to date. Read the entire page →
From page 90... ... An Assessment of ARPA-E Copyright National Academy of Sciences. All rights reserved. Read the entire page →

From page 47...

... An Assessment of ARPA-E 3 ARPA-E's Internal Operations: Culture, People, and Processes In accordance with its statement of task (Box 1-1 in Chapter 1) , the committee undertook an operational assessment to "appraise the appropriateness and effectiveness of the agency's structure to position it to achieve its mission and goals." This chapter describes and assesses ARPA-E's internal operations, addressing the following specific charges in the statement of task: • Evaluate ARPA-E's methods and procedures to develop and evaluate its portfolio of activities; • Examine appropriateness and effectiveness of ARPA-E programs to provide awardees with non-technical assistance such as practical financial, business, and marketing skills; • Assess ARPA-E's recruiting and hiring procedures to attract and retain qualified key personnel; • Examine the process, deliverables, and metrics used to assess the short and long term success of ARPA-E programs; • Assess ARPA-E's coordination with other Federal agencies and alignment with long-term DOE objectives; • Provide guidance for strengthening the agency's structure, operations, and procedures; • Evaluate, to the extent possible and appropriate, ARPA-E's success at implementing successful practices and ideas utilized by the Defense Advanced Research Projects Agency (DARPA)