Summary

The Department of Energy (DOE) asked the National Academies of Sciences, Engineering, and Medicine to conduct a quadrennial review of its Small Business Innovation Research (SBIR) and Small Business Technology Transfer (STTR) programs, in accordance with a legislative mandate. Using quantitative and qualitative analyses of data, the committee convened by the National Academies to conduct the assessment produced this report on the operations and outcomes stemming from the SBIR/STTR grants awarded by DOE.

The report undertakes a detailed examination of the SBIR and STTR programs at DOE in compliance with the legislative mandate. Drawing on published research plus existing data, the committee analyzed (1) the effectiveness of DOE’s SBIR/STTR processes and procedures on topic and awardee selection; (2) DOE outreach efforts to SBIR and STTR applicants, particularly to increase applications from small businesses that are new to the programs, from under-represented states, woman-owned, or minority-owned; (3) collaborations created between small businesses and research institutions on account of the programs; (4) a range of direct economic and non-economic impacts to awardees; and (5) the role of SBIR/STTR programs in stimulating technological innovation and contributing to DOE’s research and development needs, whether directly from awardees or indirectly through spillovers from other firms.

The committee found that DOE’s SBIR and STTR programs are effective at funding small businesses that provide and stimulate research, experimentation, and innovation in the energy sector. The programs stimulate energy innovation, contribute to DOE’s research and development needs, and advance the national energy innovation system. DOE’s SBIR/STTR programs encourage research in technical areas identified by DOE as important to their research and development (R&D) needs directly through patents and products produced by the grantees and indirectly by stimulating complementary innovation.

The majority of DOE SBIR/STTR awards go to firms that are not only small but also are young. This type of firm offers the potential for significant

technological advances. A sizable fraction of awards goes to small firms that receive multiple awards over time, and the committee explored the underlying reasons for this pattern. The SBIR/STTR programs also help to diversify the geographic reach of DOE research activities. While the bulk of DOE research activities are centered at a relatively small number of institutions (17 national labs and a small number of public and private research universities), the SBIR/STTR programs provide a channel for DOE research funding and technical expertise to be accessed by small businesses throughout the United States.

However, the programs attract only a small number of successful applications from businesses that are (a) woman-owned, (b) minority-owned, or (c) from under-represented states. DOE has not had a measurable impact on increasing the incidence of successful applications from these groups since 2012. The committee recommends that DOE modify its diversity and outreach efforts as a means of improving its ability to find firms with creative solutions to mission-oriented scientific needs.

Finally, the committee found that although some firms have achieved significant commercialization outcomes, there has not been large-scale commercialization on a systematic basis by the awardees. In part this is due to the inherent risk in developing new technologies. The committee recommends that improvements be made to the commercialization assistance provided to awardees. Recent legislative changes have increased the amount available for assistance, but this assistance must be provided through third-party vendors. This commercialization assistance cannot be used to recruit and retain in-house marketing and business talent which might improve long-run commercial prospects for the small business.

From its set of 16 findings, the committee developed eight recommendations, listed in full at the end of this summary, in Boxes S-1 and S-2. This summary presents an overview of the study and highlights the 10 findings and six recommendations the committee believes are most important.

SCOPE OF THE ASSESSMENT

Congress first requested that the National Academies undertake a study of the SBIR program as part of the Small Business Reauthorization Act of 2000. This study mandate was expanded in the National Defense Authorization Act for Fiscal Year 2012, wherein Congress directed agencies with SBIR program budgets of more than $50 million to engage with the National Academies to conduct a quadrennial assessment of each agency’s SBIR and STTR programs.¹ The congressional mandate calls for assessments of how the SBIR and STTR

___________________

¹ Two previous studies were completed by the National Academies in response to the legislative mandate. The first was completed in 2009 and included a study of the SBIR programs at the Department of Defense, National Institutes of Health, Department of Energy, National Aeronautics and Space Administration, and National Science Foundation. The second was completed in 2016 and included studies of both SBIR and STTR at the same agencies.

programs have stimulated technological innovation, have used small businesses to meet federal research and development needs (SBIR), and have stimulated technology transfer (STTR), and requests several specific analyses and evaluations, including the value and quality of R&D conducted under the programs, and their economic and non-economic benefits.²

STUDY METHODOLOGY AND LIMITATIONS

The committee based its review on a wide range of evidence. This included descriptive and qualitative evidence of agency-level outcomes; patterns in the landscape of awards, firms, and related geographic and demographic characteristics; and patterns in the administration of the SBIR/STTR program from the perspective of the agency and its offices. The committee also used descriptive evidence of program impacts on collaborations, firm structure and orientation, and other system-level outcomes that cannot be easily determined using standard econometric techniques, and supplied causal evidence of direct and indirect effects of the program on innovation, jobs, and commercialization.

The quantitative and qualitative data used by the committee included data gathered from (1) the Small Business Administration (SBA) SBIR/STTR Company and Award Listing, (2) the SBA Dynamic Small Business Search (DSBS), (3) the U.S General Services Administration System for Award Management (SAM), (4) USASpending, (5) U.S. Economic Development Administration Cluster Map (CLUSTER), (6) the Federal Procurement Data System (FPDS), (7) publication data from the Web of Science, (8) patent data from PatentsView, (9) venture capital funding data from Crunchbase, and (10) firm-level data from the National Establishment Time-Series (NETS). Data from these different sources were compiled, matched, and verified to provide the empirical basis for this study.

The committee also relied on qualitative data, including presentations by personnel from the DOE SBIR/STTR Programs Office and the Small Business Administration, consultations with current and former DOE and SBA personnel, and researchers specializing in energy innovation. The committee used archival data available from the DOE website such as webinars, publicly available documents, and funding opportunity announcements. The committee also interviewed a number of past and current DOE SBIR/STTR awardees.

Most importantly, the committee looked at a broad set of outcomes when assessing the effectiveness of the SBIR/STTR programs. Early scholars of innovation policy tended to assess the SBIR program by looking at metrics such as follow-on funding, patents, or published papers produced by the awardees. The committee cautions against a narrow focus on these traditional outcomes in evaluating the programs and notes that the programs have a role in the broader

___________________

² Section 108 of the Small Business Reauthorization Act of 2000 as amended by FY2012 National Defense Authorization Act.

system of innovative activity that is much more complex and wide-reaching than just the direct interaction between DOE and awardees. In addition to looking at the direct impact on awardees, this report includes an evaluation of the programs’ innovation outputs that result from spillovers to other firms.

Neither DOE nor SBA maintains a systematic dataset with unitary commercialization outcomes, and this made the assessment of commercialization outcomes difficult. The committee was not able to obtain access to any internal scoring for any of the SBIR/STTR applicants nor did it gain access to information about non-awardees. This limited the committee’s ability to determine differences in commercial success between awardees and unsuccessful applicants. However, the committee was able to rely on previously published studies that did have access to these data to develop its recommendations.

KEY FINDINGS

The committee developed a number of findings based on the available evidence.³ The first set of key findings relates to the role of economic and non-economic outcomes from SBIR/STTR programs.

As the findings above show, the committee found evidence that the programs produce economic growth other than by federal procurement as it considered economic factors beyond direct measures of employment—by directing attention to the research orientation of the firm and examining the role of partnership and idea diffusion through agglomeration economies. These findings are based on the committee’s analyses of the role of SBIR/STTR programs in stimulating technological innovation and contributing to DOE’s research and development needs, in accordance with the missions of DOE

___________________

³ The committee’s findings and recommendations are numbered according to the chapter and ordering where they appear.

program offices, whether directly from awardees or indirectly through spillovers from other firms. The committee used both qualitative and quantitative evidence to support these findings on technology and innovation.

While the bulk of DOE research activities are centered at a small number of national labs and a relatively small number of leading research universities, the SBIR/STTR programs provide a channel for DOE’s research funds and technical expertise to be accessed by a more geographically dispersed set of small businesses. The average distance between awardees and the nearest DOE-sponsored national lab is approximately 200 miles.

The committee was asked to analyze, to the extent practicable, the employment impact of SBIR/STTR awards on firms. The evidence suggests that SBIR does not produce greater direct employment when compared to other government program participants. It should be noted that the programs have neither a legislative mandate nor an operational model to serve as an intentional “jobs” program. However, the programs do encourage or require partnerships with research institutions, and those partnerships support indirect job creation in technical specialties at universities and nonprofit research centers and enhance the national workforce in energy innovation.

Employment for DOE SBIR/STTR awardees does grow over time, and younger firms are more likely to generate employment growth following receipt of an SBIR/STTR award. There is also evidence that a substantial number of SBIR-awarded firms have institutional partners and that the percentage of projects linked to research institutions has increased over time, showing a substantial increase in external partnerships and subcontracting activity. Measurement of employment activity within an SBIR/STTR-awarded firm does not capture employment of researchers who partner with the small business. Additionally, by statute, commercialization assistance provided to SBIR/STTR awardees must be either be provided by an agency-provided vendor or a third-party vendor identified by the awardee, and those employees are not captured by direct employment numbers.

The five findings above support the committee’s overall assessment of the effectiveness of the programs, as shown in the following finding:

While the committee found that the programs encourage research in technical areas that are important to DOE’s research and development needs, perhaps more importantly, there is some evidence that these programs have spillovers into the broader national energy innovation system and DOE’s overall mission. Given the importance of these spillovers, the committee offers a series of findings and recommendations to help realize DOE’s mission even more fully.

The committee had a number of findings, articulated below, about the processes put in place by the DOE SBIR/STTR Programs Office. Described in more detail in Chapter 3, these procedures are clear to the individual DOE offices offering SBIR/STTR grants. The processes and timelines are clearly laid out on the DOE website, with webinars offered to provide details about technical parameters regarding the topics where applications will be sought. However, the number of successful applications from woman-owned, minority-owned, or new to the program small businesses has not increased since 2012.

The committee’s findings regarding awardee’s collaborations were limited to evidence provided by STTR awards, which require documented collaborations and formal intellectual property agreements between small businesses and research institutions such as a university, non-profit, or a national lab. The STTR program is associated with the creation of a significant number of

collaborations between small businesses and research institutions over time that occur at significant geographic distance.

The committee found a substantial difference in geographic distance between woman-owned awardees and their research partners compared to overall STTR awardees. The difference in distances traveled by woman-owned businesses suggest that these firms seem to face greater difficulty in finding R&D partners with the specialized knowledge needed to develop, test, and commercialize their innovations.

KEY RECOMMENDATIONS

On the basis of its findings, the committee formulated eight recommendations intended to help DOE, SBA, and Congress continue to strengthen the programs. Six of these recommendations stand out as key to improving the programs’ stated missions and goals.

The majority of reviewers for applications come from national labs (46 percent) or universities (32 percent) where potential reviewers are more likely to be male and white. Women and under-represented minorities comprise only a small share of leadership or research/technical management positions at the national labs.⁴ Similarly, underrepresented minorities comprise less than 10 percent of full professors, and women comprise 32 percent of all full professors at U.S. colleges and universities.⁵ For engineering departments these numbers are even lower, with women professors of any rank comprising 16 percent of the faculty, African American faculty comprising 2.3 percent, and Latino faculty comprising 3.6 percent.⁶

___________________

⁴ See https://nationallabs.org/staff/diversity/. At the national laboratories, women comprise approximately 25 percent of senior leadership positions and 18 percent of research/technical management positions. Underrepresented minorities comprise about than 8% of senior leadership positions and research/technical management positions.

⁵ See https://nces.ed.gov/fastfacts/display.asp?id=61.

⁶ See https://www.asee.org/documents/papers-and-publications/publications/college-profiles/16Profile-Front-Section.pdf.

Given the significant geographic distances separating potential SBIR/STTR applicants from the pool of prospective R&D partners, providing applicants with historical data on the specific partners associated with past awards would be especially helpful to new applicants. Currently information on research partners is available only for STTR awards, even though SBIR awards constitute the vast majority of all awards. New applicants may be unaware of or unfamiliar with local or nearby research institutions that possess the relevant expertise and capabilities for commercializing energy-related technologies. Ideally, this information would be made publicly available as additional searchable fields linked to the SBA’s existing SBIR/STTR Award and Company Listing Databases. By making it easy for applicants to find and connect to eligible research partners, DOE may be able to substantially improve the breadth and depth of its applicant pool. This may be especially helpful to woman-owned businesses where long distances between the company and its research partner(s) suggest difficulty in finding partners.

Given the large distances between the national labs (the most concentrated sources of R&D expertise) and potential SBIR/STTR applicants, the committee recommends that DOE deploy virtual mentoring programs to connect national labs with grant applicants. Geographic distance appears to be a substantial barrier for many awardee firms, and online technologies may offer a cost-effective way to bridge the distances between awardees and suitable R&D partners. Given the potential budget, personnel, and time resource constraints that DOE faces, DOE could build upon existing capabilities for supporting procurement from small, minority-owned businesses, such as the Mentor-Protégé Program, which fosters long-term business relationships between “small disadvantaged firms certified by the Small Business Administration under Section 8(A) of the Small Business Act”⁷ and prime contractors.

___________________

⁷ For more information, see https://www.energy.gov/osdbu/osdbu-programs-home/mentor-protege-program.

SBIR/STTR allocations are a legislatively mandated percentage set-aside of DOE’s extramural funding. Congress authorizes DOE’s budget as a whole, but also specifies budgets for each office within DOE. Section 301 (d-g) of DOE’s appropriations law directs DOE to adhere to congressional budget control points, and in fiscal year 2018, DOE’s Office of General Counsel determined that Section 301 also applies to the funds allocated to SBIR/STTR. For some offices, such as the Office of Energy Efficiency and Renewable Energy, Congress provides specific guidance across the different technologies. Some offices may not be able to award a Phase II award or even a Phase I STTR award within their budget allocations. The strict application of the Section 301 requirements to SBIR/STTR constrains the flexibility of the program in selecting topics to meet broad agency R&D needs and limits competition for awards within the program. It should not apply to the SBIR/STTR programs.

Even without lifting these restrictions, one way the Secretary of Energy could encourage greater coordination is to centralize the selection of topics and awardees. The current decentralization of topics, reviewers, and awardees, combined with the congressional mandate to use SBIR/STTR to meet the agency’s needs, may be leading to the selection of topics and awardees that may appropriately meet individual office’s needs, but miss opportunities for advancing DOE’s greater mission. The Secretary of Energy could also encourage additional use of collaborations to prioritize broader DOE priorities over individual offices’ missions.

DOE program offices offering SBIR and STTR grants emphasize the technical merit of a proposal. However, successful commercialization often requires development of complementary organizational assets. The committee found that firms that receive SBIR or STTR grants from DOE are more likely to have a founder who is the chief technical officer compared to firms that do not receive SBIR or STTR grants. Moreover, the SBIR and STTR firms are much less likely to expand their top management teams of executives over time. There is

also evidence that SBIR and STTR firms are less likely to bring on a seasoned or professional chief executive officer, which may help the company expand its commercialization outcomes.

FINAL THOUGHTS

The SBIR and STTR programs stimulate energy innovation and experimentation, contribute to DOE’s research and development needs, and advance the national energy innovation system. The programs encourage research in technical areas identified by DOE as important to its research and development mission needs both directly, through patents and products produced by the grantees, and indirectly, by stimulating complementary innovation. The committee offers a series of recommendations designed to help DOE better realize its mission to ensure America’s security and prosperity through transformative science and technology solutions in the energy, environmental, and nuclear fields. At the same time, follow-on funding for DOE SBIR/STTR awardees is low. This low rate of follow-on funding reflects the dearth of venture capital funding for clean energy startups as a whole, which researchers have attributed to the large upfront investment and long time horizons for the returns (Gaddy, Sivaram, and O’Collivan, 2016). More research on ways to improve the clean energy ecosystem would help improve the commercialization rate of DOE’s SBIR/STTR awardees.