STTR: An Assessment of the Small Business Technology Transfer Program (2016)

1

Introduction

Small businesses are an important driver of innovation and economic growth in the United States.¹ Despite the challenges of changing global environments and the impacts of the 2008 financial crisis and subsequent recession, innovative small businesses continue to develop and commercialize new products for the market, improving the health and welfare of Americans while strengthening the nation’s security and competitiveness.²

Created in 1982 through the Small Business Innovation Development Act,³ the Small Business Innovation Research (SBIR) program remains the nation’s largest innovation program for small businesses. The SBIR program offers competitive awards to support the development and commercialization of innovative technologies by small private-sector businesses.⁴ At the same time,

________________

¹See Z. Acs and D. Audretsch, “Innovation in large and small firms: An empirical analysis,” The American Economic Review, 78(4):678-690, 1988. See also Z. Acs and D. Audretsch, Innovation and Small Firms, Cambridge, MA: The MIT Press, 1991; E. Stam and K. Wennberg, “The roles of R&D in new firm growth,” Small Business Economics, 33:77-89, 2009; E. Fischer and A.R. Reuber, “Support for rapid-growth firms: A comparison of the views of founders, government policymakers, and private sector resource providers,” Journal of Small Business Management, 41(4):346-365, 2003; M. Henrekson and D. Johansson, “Competencies and institutions fostering high-growth firms,” Foundations and Trends in Entrepreneurship, 5(1):1-80, 2009.

²See D. Archibugi, A. Filippetti, and M. Frenz, “Economic crisis and innovation: Is destruction prevailing over accumulation?” Research Policy, 42(2):303-314, 2013. The authors show that “the 2008 economic crisis severely reduced the short-term willingness of firms to invest in innovation” and also that it “led to a concentration of innovative activities within a small group of fast growing new firms and those firms already highly innovative before the crisis.” They conclude that “the companies in pursuit of more explorative strategies towards new product and market developments are those to cope better with the crisis.”

³Small Business Innovation Development Act of 1982, P.L. 97-219, July 22, 1982.

⁴SBIR awards can be made as grants or as contracts. Grants do not require the awardee to provide an agreed deliverable (for contracts this is often a prototype at the end of Phase II). Contracts are also governed by federal contracting regulations, which are considerably more onerous from the small business perspective. Historically, all Department of Defense (DoD) and National Aeronautics and Space Administration (NASA) awards have been contracts; all National Science Foundation (NSF)

the program provides government agencies with technical and scientific solutions that address their different missions.

Seeking to bridge the gap between basic science and commercialization of resulting innovations, the Small Business Technology Transfer (STTR) program, created in 1992 by the Small Business Research and Development Enhancement Act of 1992,⁵ sought to expand joint venture opportunities for small businesses and nonprofit research institutions. Under the STTR program, a small business receiving an award must collaborate formally with a research institution.

Both the SBIR and STTR programs share a three-phase structure:

• Phase I provides limited funding (up to $100,000 prior to the 2011 reauthorization and up to $150,000 thereafter) for feasibility studies.
• Phase II provides more substantial funding for further research and development (typically up to $750,000 prior to 2012 and $1 million after the 2011 reauthorization).⁶
• Phase III reflects commercialization without providing access to any additional SBIR/STTR funding, although funding from other federal government accounts is permitted.

The SBIR program has four congressionally mandated goals: (1) stimulate technological innovation, (2) use small business to meet federal research and development (R&D) needs, (3) foster and encourage participation by minority and disadvantaged persons in technological innovation, and (4) increase private-sector commercialization derived from federal research and development.⁷

In comparison, the statutory objective for the STTR program is “to stimulate a partnership of ideas and technologies between innovative small business concerns (SBCs) and Research Institutions through Federally-funded research or research and development (R/R&D).”⁸ However, the Small Business Administration (SBA) web site also gives additional objectives that are largely aligned with the SBIR program: (1) stimulate technological innovation, (2) foster technology transfer through cooperative R&D between small businesses and research institutions, and (3) increase private-sector commercialization of innovations derived from federal R&D. ⁹

________________

and most National Institutes of Health (NIH) awards have been grants, and the Department of Energy (DoE) has used both vehicles.

⁵Small Business Research and Development Enhancement Act, P.L. 102-564, S. 2941, October 28, 1992.

⁶All resource and time constraints imposed by the program are somewhat flexible and are addressed by different agencies in different ways. For example, NIH and to a much lesser degree DoD have provided awards that are much larger than the standard amounts, and NIH has a tradition of offering no-cost extensions to allow for completion of work on an extended timeline.

⁷Small Business Innovation Development Act of 1982, P.L. 97-219, S. 881, July 22, 1982.

⁸SBA, STTR Policy Directive, February 2014, p. 3.

⁹SBA, https://www.sbir.gov/about/about-sttr, accessed on November 27, 2015.

Each of the research agencies has sought to pursue these goals in administering its SBIR and STTR programs, utilizing the administrative flexibility to address its own unique mission needs.¹⁰

Although the SBIR and STTR programs have similar objectives, there are important differences. Under SBIR, the Principal Investigator (PI) must be primarily employed with the small business concern (SBC) at the time of award and for the duration of the project period, but primary employment is not stipulated in the STTR Program. Under STTR, the principal investigator “may be primarily employed by either the small business concern or the collaborating non-profit research institution at the time of award and for the duration of the project period.”¹¹

The programs also differ in that STTR requires the small business concern to formally collaborate with a nonprofit research institution. Research partnerships are permitted under the SBIR program, but the partnering research institution can complete no more than one-third of the Phase I work and no more than one-half of the Phase II work. In contrast, “Under STTR, the small business must perform at least 40 percent of the work and the research institution must perform at least 30 percent. The remaining 30 percent may be … [completed by] the small business concern, the collaborating non-profit research institution, or an additional third party.”¹²

Over time, through a series of reauthorizations, the legislation enabling SBIR and STTR has required federal agencies with extramural R&D budgets in excess of $100 million to set aside a growing share of their budgets for the SBIR program and those with extramural R&D budgets in excess of $1 billion to set aside a growing share of their budgets for the STTR program (see Table 1-1). By fiscal year (FY)2012, the 11 federal agencies listed in Table 1-2 that administer SBIR and STTR programs were disbursing $2.4 billion dollars a year.¹³ Five agencies administer greater than 96 percent of SBIR/STTR funds: Department of Defense (DoD), Department of Health and Human Services (HHS; particularly NIH), National Aeronautics and Space Administration (NASA), National Science Foundation (NSF), and Department of Energy (DoE). Aggregate award amounts for the five largest agencies for FY2015 are provided in Table 1-1. STTR is administered only at these five agencies.

Although STTR is dwarfed in size by SBIR (agency budgets for SBIR are seven or eight times larger than those of STTR), SBA views it as a vehicle to expand funding opportunities in the federal innovation R&D arena. In particular, STTR was designed to combine the strengths of research institutions

________________

¹⁰The committee commended this flexibility in its 2008 assessment of the SBIR program. See Finding C, National Research Council, An Assessment of the SBIR Program, Washington, DC: The National Academies Press, 2008, p. 59.

¹¹See, for example, the NIH web site at https://sbir.nih.gov/about/critical, accessed July 9, 2015.

¹²Ibid.

¹³SBA, SBIR/STTR annual report, http://www.sbir.gov/, accessed July 2015. FY2012 is the most recent year for which SBA publishes comparative data across agencies.

TABLE 1-1 SBIR/STTR Funding by the Five Principal Funding Agencies, FY 2015

SOURCE: Small Business Administration, SBIR/STTR Annual Report, http://www.sbir.gov, accessed October 20, 2015.

TABLE 1-2 Agencies Participating in the SBIR and STTR Programs in 2015

SOURCE: Small Business Administration.

and small firms by introducing entrepreneurial skills to high-tech research efforts. This design has sought to assist the transfer of technologies and products from the laboratory to the marketplace.

HISTORY AND STRUCTURE OF THE SBIR AND STTR PROGRAMS¹⁴

During the 1980s, the perceived decline in U.S. competitiveness due to Japanese industrial growth in sectors traditionally dominated by U.S. firms—

________________

¹⁴Parts of this section are based on the Academies’ previous report on the NIH SBIR program—National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, Washington, DC: The National Academies Press, 2009.

autos, steel, and semiconductors—led to concerns about future economic growth in the United States.¹⁵ A key concern was the perceived failure of American industry “to translate its research prowess into commercial advantage.”¹⁶ Although the United States enjoyed dominance in basic research—much of which was federally funded—applying this research to the development of innovative products and technologies remained a challenge. As the great corporate laboratories of the post-war period were buffeted by change, new models such as the cooperative model utilized by some Japanese keiretsu seemed to offer greater sources of dynamism and more competitive firms.¹⁷

At the same time, new evidence emerged to indicate that small businesses were an increasingly important source of both innovation and job creation.¹⁸ This evidence reinforced recommendations from federal commissions, dating back to the 1960s, that federal R&D funding should provide more support for innovative small businesses (which was opposed by traditional recipients of government R&D funding).¹⁹

Early-stage financial support for innovative technology-based small businesses for developing high-risk technologies with commercial promise was first advanced by Roland Tibbetts at NSF. In 1976, Mr. Tibbetts advocated shifting some NSF funding for this purpose. NSF adopted this initiative first, and after a period of analysis and discussion, the Reagan administration supported its expansion across the federal government. Congress then passed the Small Business Innovation Research Development Act of 1982, which established the SBIR program.

________________

¹⁵See J. Alic, “Evaluating competitiveness at the office of technology assessment,” Technology in Society, 9(1):1-17, 1987, for a review of how these issues emerged and evolved within the context of a series of analyses at a Congressional agency.

¹⁶D.C. Mowery, “America’s industrial resurgence (?): An overview,” in D.C. Mowery, ed., U.S. Industry in 2000: Studies in Competitive Performance, Washington, DC: National Academy Press, 1999, p. 1. Other studies highlighting poor economic performance in the 1980s include M.L. Dertouzos et al., Made in America: The MIT Commission on Industrial Productivity, Cambridge, MA: The MIT Press, 1989; and O. Eckstein, DRI Report on U.S. Manufacturing Industries, New York: McGraw Hill, 1984.

¹⁷Keiretsu is “the name given to a form of corporate structure in which a number of organisations link together, usually by taking small stakes in each other and usually as a result of having a close business relationship, often as suppliers to each other.” See The Economist, “Keiretsu,” October 16, 2009. http://www.economist.com/node/14299720.

¹⁸For an alternate view, see S.J. Davis, J. Haltiwanger, and S. Schuh, Small Business and Job Creation: Dissecting the Myth and Reassessing the Facts, Working Paper No. 4492, Cambridge, MA: National Bureau of Economic Research, 1993. Evaluating the empirical basis for conventional claims about the job-creating prowess of small businesses, the authors find inter alia that conventional wisdom about the job-creating prowess of small business rests on misleading interpretations of the data. According to Per Davidsson, these methodological fallacies, however, “ha[ve] not had a major influence on the empirically based conclusion that small firms are overrepresented in job creation.” See P. Davidsson, “Methodological concerns in the estimation of job creation in different firm size classes,” Working Paper, Jönköping International Business School, 1996.

¹⁹For an overview of the origins and history of the SBIR program, see G. Brown and J. Turner, “The federal role in small business research,” Issues in Science and Technology, Summer 1999, pp. 51-58.

Initially, the SBIR program required agencies with extramural R&D budgets in excess of $100 million²⁰ to set aside 0.2 percent of their funds for SBIR. Program funding totaled $45 million in the program’s first year of operation (1983). Over the next 6 years, the set-aside grew to 1.25 percent.²¹

SBIR Reauthorizations of 1992 and 2000

The SBIR program approached reauthorization in 1992 amidst continued worries about the ability of U.S. firms to commercialize inventions (see Box 1-1). Finding that “U.S. technological performance is challenged less in the creation of new technologies than in their commercialization and adoption,” the Academies recommended an increase in SBIR funding as a means to improve the economy’s ability to adopt and commercialize new technologies.²²

The Small Business Research and Development Enhancement Act (P.L. 102-564) reauthorized the SBIR program until September 30, 2000, and doubled the set-aside rate to 2.5 percent. The legislation also more strongly emphasized the need for commercialization of SBIR-funded

________________

²⁰That is, those agencies spending more than $100 million on research conducted outside agency labs.

²¹Additional information regarding SBIR’s legislative history can be accessed from the Library of Congress. See http://thomas.loc.gov/cgi-bin/bdquery/z?d097:SN00881:@@@L.

²²See National Research Council, The Government Role in Civilian Technology: Building a New Alliance, Washington, DC: National Academy Press, 1992, p. 29.

technologies.²³ Legislative language explicitly highlighted commercial potential as a criterion for awarding SBIR contracts and grants.

At the same time, Congress expanded the SBIR program’s purposes to “emphasize the program’s goal of increasing private sector commercialization developed through federal research and development and to improve the federal government’s dissemination of information concerning the small business innovation, particularly with regard to woman-owned business concerns and by socially and economically disadvantaged small business concerns.”²⁴

The Small Business Reauthorization Act of 2000 (P.L. 106-554) extended the SBIR program until September 30, 2008. It also called for a National Research Council (NRC)²⁵ assessment of the program’s broader impacts, including those on employment, health, national security, and national competitiveness.²⁶

STTR Establishment and Reauthorizations

Established by the Small Business Technology Transfer Act of 1992 (P.L. 102-564, Title II), the STTR program was reauthorized until the year 2001 by the Small Business Reauthorization Act of 1997 (P.L. 105-135) and reauthorized again until September 30, 2009, by the Small Business Technology Transfer Program Reauthorization Act of 2001 (P.L. 107-50).²⁷

As explained below, the SBIR/STTR Reauthorization Act of 2011 included a number of changes to the SBIR-STTR programs, including increases in the set-asides over the next 6 years and expanded eligibility for STTR awardees to take part in technical assistance programs.

________________

²³Small Business Research and Development Enhancement Act, P.L. 102-564, S. 2941, October 28, 1992. See also R. Archibald and D. Finifter, “Evaluation of the Department of Defense Small Business Innovation Research program and the Fast Track Initiative: A balanced approach,” in National Research Council, The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative, Washington, DC: National Academy Press, 2000, pp. 211-250.

²⁴Small Business Research and Development Enhancement Act, P.L. 102-564, S. 2941, October 28, 1992.

²⁵Effective July 1, 2015, the institution is called the National Academies of Sciences, Engineering, and Medicine. References in this report to the National Research Council or NRC are used in an historic context identifying programs prior to July 1.

²⁶The current assessment is congruent with the Government Performance and Results Act (GPRA) of 1993: http://govinfo.library.unt.edu/npr/library/misc/s20.html. As characterized by the Government Accountability Office (GAO), GPRA seeks to shift the focus of government decision making and accountability away from a preoccupation with the activities that are undertaken—such as grants dispensed or inspections made—to the results of those activities. See http://www.gao.gov/new.items/gpra/gpra.htm.

²⁷See https://www.sbir.gov/about, accessed on December 3, 2015.

The 2011 SBIR/STTR Reauthorization

The anticipated 2008 reauthorization was delayed in large part by a disagreement between long-time program participants and their advocates in the small business community and proponents of expanded access for venture-backed firms, particularly in biotechnology where proponents argued that the standard path to commercial success includes venture funding at some point.²⁸ Other issues were also difficult to resolve, but the conflict over participation of venture-backed companies dominated the process²⁹ following an administrative decision to exclude these firms more systematically.³⁰

After a much extended discussion, passage of the National Defense Act of December 2011 reauthorized the SBIR/STTR programs through FY2017.³¹ The new law maintained much of the core structure of both programs but made some important changes, which were to be implemented via the SBA’s subsequent Policy Guidance.³²

The eventual compromise on the venture funding issue allowed (but did not require) agencies to award up to 25 percent of their SBIR grants or contracts (at NIH, DoE, and NSF) or 15 percent (at the other awarding agencies) to firms that benefit from private, venture capital investment. It is too early in the implementation process to gauge the impact of this change.³³

The reauthorization made changes to the SBIR program that were recommended in prior Academies reports.³⁴ These included the following:

• Increased award size limits
• Expanded program size
• Enhanced agency flexibility—for example, for Phase I awardees from other agencies to be eligible for Phase II awards or to award an additional Phase II
• Improved incentives for the utilization of SBIR technologies in agency acquisition programs
• Explicit requirements for better connecting prime contractors with SBIR awardees

________________

²⁸D.C. Specht, “Recent SBIR extension debate reveals venture capital influence,” Procurement Law, 45:1, 2009.

²⁹W.H. Schacht, “The Small Business Innovation Research (SBIR) program: Reauthorization efforts,” Congressional Research Service, Library of Congress, 2008.

³⁰A. Bouchie, “Increasing number of companies found ineligible for SBIR funding,” Nature Biotechnology, 21(10):1121-1122, 2003.

³¹SBIR/STTR Reauthorization Act of 2011, P.L. 112-81, December 31, 2011.

³²See SBA post, S. Greene, “Implementing the SBIR and STTR Reauthorizations: Our Plan of Attack,” http://www.sbir.gov/news/implementing-sbir-and-sttr-reauthorization-our-plan-attack, accessed February 21, 2012.

³³See National Research Council, Venture Capital and the NIH SBIR Program, Washington, DC: The National Academies Press, 2009.

³⁴See Appendix B for a list of the major changes to the SBIR program resulting from the 2011 Reauthorization Act.

• Substantial emphasis on developing a more data-driven culture, which has led to several major reforms, including the following:
• • adding numerous areas of expanded reporting
  • extending the Academies’ evaluation
  • adding further evaluation, such as by the Government Accountability Office and Comptroller General
  • tasking the SBA with creating a unified platform for the collection of data

• Expanded management resources (through provisions permitting use of up to 3 percent of program funds for [defined] management purposes)
• Expanded commercialization support (through provisions providing companies with direct access to commercialization support funding and through approval of the approaches piloted in Commercialization Pilot Programs)
• Options for agencies to add flexibility by developing other pilot programs—for example, to allow awardees to skip Phase I and apply for a Phase II award directly or for NIH to support a new Phase 0 pilot program.

The reauthorization also made changes that were not mentioned in previous reports of the Academies. These included the following:

• Expansion of the STTR program
• Limitations on agency flexibility—particularly in the provision of larger awards
• Introduction of commercialization benchmarks for companies, which must be met if companies are to remain in the program. These benchmarks are to be established by each agency.

Other clauses of the legislation affect operational issues, such as the definition of specific terms (such as “Phase III”), continued and expanded evaluation by the Academies, mandated reports from the Comptroller General on combating fraud and abuse within the SBIR program, and protection of small firms’ intellectual property within the program.

PREVIOUS RESEARCH ON SBIR

Studies pre-dating the Academies’ first-round assessment, most notably by the General Accounting Office and the SBA, focused only on specific aspects

or components of the SBIR/STTR programs.³⁵ In addition, prior to the first-round assessment, there had been few internal assessments of agency SBIR/STTR programs. The academic literature on SBIR was also limited,³⁶ except for an assessment in the 1990s by Joshua Lerner of the Harvard Business School, who found “that SBIR awardees grew significantly faster than a matched set of firms over a ten-year period.”³⁷

To help fill this assessment gap for the SBIR/STTR programs, the NRC’s Committee for Government-Industry Partnerships for the Development of New Technologies (GIP, which preceded the NRC’s first-round congressionally mandated study of the SBIR program) convened a workshop to discuss the SBIR program’s history and rationale, review existing research, and identify areas for further research and program improvements.³⁸ In addition, in its report on the SBIR Fast Track Initiative at DoD, the GIP committee found that the SBIR program contributed to mission goals by funding “valuable innovative projects.”³⁹ It concluded that a significant number of these projects would not have been undertaken absent SBIR funding⁴⁰ and that DoD’s Fast Track Initiative encouraged the commercialization of new technologies⁴¹ and the entry of new firms into the program.⁴² The GIP committee also found that the SBIR program improved both the development and utilization of human capital and the diffusion of technological knowledge.⁴³ Case studies provided some evidence that the knowledge and human capital generated by the SBIR program have positive economic value, which spills over into other firms through the movement of people and ideas.⁴⁴ Furthermore, by providing a validation of

________________

³⁵An important step in the evaluation of the program has been to identify existing evaluations of the program. These include U.S. General Accounting Office, Federal Research: Small Business Innovation Research Shows Success But Can Be Strengthened, Washington, DC: U.S. General Accounting Office, 1992; and U.S. General Accounting Office, Evaluation of Small Business Innovation Can Be Strengthened, Washington, DC: U.S. General Accounting Office, 1999. There is also a 1999 unpublished SBA study on the commercialization of SBIR Phase II awards from 1983 to 1993 among non-DoD agencies.

³⁶Early examples of evaluations of the SBIR program include S. Myers, R. L. Stern, and M. L. Rorke, A Study of the Small Business Innovation Research Program, Lake Forest, IL: Mohawk Research Corporation, 1983; and Price Waterhouse, Survey of Small High-tech Businesses Shows Federal SBIR Awards Spurring Job Growth, Commercial Sales, Washington, DC: Small Business High Technology Institute, 1985.

³⁷See J. Lerner, “The government as venture capitalist: The long-run effects of the SBIR program,” Journal of Business, 72(3), 1999.

³⁸ See National Research Council, The Small Business Innovation Research Program: Challenges and Opportunities, Washington, DC: National Academy Press, 1999.

³⁹National Research Council, The Small Business Innovation Research Program: An Assessment of the DoD SBIR Fast Track Initiative, Washington, DC: National Academy Press, p. 32.

⁴⁰Ibid., p. 32.

⁴¹Ibid., p. 33.

⁴²Ibid., p. 34.

⁴³Ibid., p. 33.

⁴⁴Ibid., p. 33.

promising new technologies, SBIR awards encourage further private-sector investment in an award-winning firm’s technology.⁴⁵

FIRST-ROUND ASSESSMENT OF THE SBIR PROGRAM

The 2000 SBIR reauthorization mandated that the NRC complete a comprehensive assessment of the SBIR program.⁴⁶ The separate assessment of the SBIR programs at DoD, NIH, NASA, NSF, and DoE began in 2002 and was conducted in three steps. As a first step, the committee authoring this study developed a research methodology⁴⁷ and gathered information about the program by convening workshops where officials at the relevant federal agencies described their program operations, challenges, and accomplishments. These meetings highlighted the important differences in agency goals, practices, and evaluations. They also served to describe the evaluation challenges that arise from the diversity in program objectives and practices.⁴⁸

The committee implemented the research methodology during the second step. As set out in the methodology, multiple data collection modalities were deployed. These included the first large-scale survey of SBIR award recipients. Case studies of a wide variety of SBIR firms were also developed. The committee then evaluated the results and developed the findings and recommendations presented for improving the effectiveness of the SBIR program.

During the third step, the committee reported on the SBIR program through a series of publications in 2008-2010: five individual volumes on the major funding agencies and an additional overview volume titled An Assessment of the SBIR Program.⁴⁹ Together, these reports provided the first detailed and comprehensive review of the SBIR program and, as noted above, served as an important input into SBIR reauthorization prior to December 2011 (see Box 1-2).

CURRENT, SECOND-ROUND STUDY: CHALLENGES AND OPPORTUNITIES

The first-round study of the SBIR program found that the program was, overall, “sound in concept and effective in practice.”⁵⁰ The current study,

________________

⁴⁵Ibid., p. 33.

⁴⁶SBIR Reauthorization Act of 2000, P.L. 106-554, Appendix I-H.R. 5667, Section 108.

⁴⁷National Research Council, An Assessment of the Small Business Innovation Research Program: Project Methodology, Washington, DC: The National Academies Press, 2004.

⁴⁸Adapted from National Research Council, SBIR: Program Diversity and Assessment Challenges, Washington, DC: The National Academies Press, 2004.

⁴⁹National Research Council, An Assessment of the SBIR Program.

⁵⁰Ibid., p. 54.

described in the Statement of Task in Box 1-3, provides a second snapshot to measure the SBIR program’s progress against its legislative goals. Importantly, the second-round study also includes an assessment of the STTR program.

This volume on the STTR program partially addresses this Statement of Task. It is supplemented by a number of workshops and other publications that assess agency SBIR programs (See Box 1-2). For example, workshops were convened on the participation of women and minorities in the SBIR-STTR programs (February 2013), the evolving role of university participation in the

programs (February 2014), the relationship between state innovation programs and the SBIR program (October 2014), the STTR program (May 2015), and the economics of entrepreneurship in relation to the SBIR program (June 2015). The committee published a report on Innovation, Diversity, and Success in the SBIR/STTR Programs, based on the 2013 workshop. Relevant to this publication, the committee also convened a workshop on the STTR program on May 1, 2015.

Information on which to assess the STTR program has been drawn from the Academies 2011 and 2014 surveys, which is described in detail in Appendix A, company case studies profiled in Appendix E, discussions with university technology transfer officials, a series of ongoing discussions and conversations with agency officials, and the workshop convened by the committee on the STTR program in Washington, DC on May 1, 2015 (see Box 1-4). A guide to the contents, data sources, and organization of this report can be found at the conclusion of this chapter.

STUDY METHODOLOGY

The SBIR-STTR programs are unique in terms of scale and mission focus. In addition, the evidence suggests that there are no truly comparable

programs in the United States, and those in other countries operate in such different ways that their relevance is limited.⁵¹ Thus, it is difficult to identify programs comparable to SBIR-STTR against which to benchmark their results.

Assessing the SBIR-STTR programs is challenging given the diversity of the agencies involved. At DoD and NASA, SBIR-STTR awards are primarily designed to generate tools and capabilities for agency use. At the other study agencies they are instead explicitly designed to generate technologies that will be adopted outside the agency, primarily in the private sector. Thus commercialization success cannot be measured across agencies only in terms of project outputs sold to the agency.

The SBIR-STTR programs are also highly decentralized at some agencies. At NIH for example, the SBIR-STTR program office within the NIH Office of Extramural Programs sets policy and provides critical cross-agency communication flows, as well as links the program to outside stakeholders, but award funding is separately determined by each Institute or Center (IC). ICs take different views of the program and use different approaches to program management. Therefore, generalizations about the SBIR and STTR programs must be made with care.

Focus on Legislative Objectives

This study is focused on assessing the extent to which STTR is meeting its congressionally mandated objectives and on providing recommendations for further program improvements.⁵² It provides assessment‐based findings of the benefits and costs of STTR while seeking to improve the public’s understanding of the program—including the differing perspectives of entrepreneurs, agency staff, research institutions, and other stakeholders—and makes recommendations to improve the program’s effectiveness.

Definition Challenges

Commercialization offers practical and definitional challenges. As described in Chapter 5, several different definitions of commercialization can be used when discussing the SBIR-STTR programs. In fact, it is important to use more than one simple definition. For example, the percentage of funded projects that reach the marketplace is not the only measure of commercial success.

________________

⁵¹See National Academies of Sciences, Engineering, and Medicine, Workshop on “Learning from Each Other: U.S. European Perspectives on Small Business Innovation Programs,” Washington, DC, March 19, 2015.

⁵²These limited objectives are consistent with the methodology developed by the committee. See National Research Council, An Assessment of the Small Business Innovation Research Program: Project Methodology.

In the private sector, commercial success over the long term requires profitability. However, in the short term, the path to successful commercialization can involve many different aspects of commercial activity, from product rollout to licensing to patenting to acquisition. Even during new product rollout, companies often do not generate immediate profits. This second-round assessment uses multiple metrics to address the question of commercialization (see Chapter 5).

Quantitative Assessment Methods

More practically, several issues relate to the application of quantitative assessment methods, including decisions about which kinds of program participants should be targeted for survey deployment, the number of responses that are appropriate, selection bias, nonresponse bias, the design and implementation of survey questionnaires, and the level of statistical evidence required for drawing conclusions in this case. These and other issues were discussed at a workshop and summarized in a 2004 report.⁵³ Also, as noted above, a peer-reviewed report on the study methodology completed by the first-round committee provided the baseline for the initial study and for follow-on studies—including this one.⁵⁴

Survey Development

For the current study, a survey of SBIR and STTR award recipients was developed and deployed. The survey was based closely on previous surveys, particularly the 2005 Survey that focused exclusively on SBIR, but nonetheless included significant improvements.⁵⁵ The description of the survey and improvements, including a discussion of the survey outreach and response, are documented in Appendix A. Most notably, the survey development made an ambitious but ultimately unsuccessful effort to develop a comparison group to provide context and a benchmark for analyzing the results (this effort is also discussed in Appendix A).

The survey covered 1,400 STTR awards made by the five study agencies during the period FY1998-2010. There was a recorded point of contact for each of the 1,400 awards. This is the preliminary survey population. Awards made in FY2010 would only have started to generate commercial products by the time of the survey in FY2014, so following established practice from prior Academies and GAO surveys, no awards made after FY2010 were surveyed.

________________

⁵³National Research Council, SBIR: Program Diversity and Assessment Challenges.

⁵⁴National Research Council, An Assessment of the Small Business Innovation Research Program: Project Methodology.

⁵⁵The survey carried out as part of this study was administered in 2014, and the survey completed as part of the Academies’ first-round assessment of the SBIR program was administered in 2005. In this volume all survey references are to the 2014 Survey unless noted otherwise.

Given the time period covered it is not surprising that many points of contact could not be reached. Of the 1,400 contacts comprising the preliminary survey population, 807 could not be reached, leaving an effective population of 593. Of these, 292 answered questionnaires, generating a response rate of 20.9 percent for the preliminary population and 49.3 percent for the effective population.

Appendix A provides a detailed discussion of the issues related to quantitative methodologies, as well as a review of potential biases. As a result of the relatively small response rate, there are significant limitations on the conclusions that can be drawn from this quantitative assessment, which is reflected in the wording of findings and recommendations (Chapter 6). At the same time, drawing on quantitative analysis is a crucial component of the overall study, reflective of the need to identify and assess outcomes that are found only by querying contacts in participating companies for individual STTR project awards.

A Complement of Approaches

Partly because of these limitations, the 2004 methodology report stressed the importance of utilizing a complement of research modalities, an approach that has been adopted here.⁵⁶ Although quantitative assessment represents the bedrock of our research and provides insights and evidence that could not be generated through any other modality, it is, in and of itself, insufficient to address the multiple questions posed in this analysis. Consequently, we undertook a series of additional activities:

• Case studies. We conducted in-depth case studies of 11 STTR recipients. These companies were geographically and demographically diverse, funded by different agencies, focused on different kinds of technologies, and at different stages of the company lifecycle. The case studies themselves are included as Appendix E.
• Workshops. We conducted workshops, including workshops to discuss the participation of women and minorities and the role of universities in the SBIR-STTR programs, as well as a workshop focused on the STTR program,⁵⁷ to allow stakeholders, agency staff, and academic experts to

________________

⁵⁶National Research Council, An Assessment of the Small Business Innovation Research Program: Project Methodology.

⁵⁷Workshops convened by the committee as part of the overall analysis include NASA Small Business Innovation Research Program Assessment: Second Phase Analysis, January 28, 2010; Early-Stage Capital in the United States: Moving Research Across the Valley of Death and the Role of SBIR, April 16, 2010; Early-Stage Capital for Innovation—SBIR: Beyond Phase II, January 27, 2011; NASA's SBIR Community: Opportunities and Challenges, June 21, 2011; Innovation, Diversity, and Success in the SBIR/STTR Programs, February 7, 2013; Commercializing University Research: The Role of SBIR and STTR, February 5, 2014; SBIR/STTR & the Role of State Programs, October 7, 2014; The Small Business Technology Transfer Program, May 1, 2015, and the

• provide insights into program operations and to identify issues that need to be addressed.

• Analysis of agency data. As appropriate, we analyzed and included data from the five STTR agencies that cover various aspects of program activities.
• Open-ended responses from SBIR-STTR recipients. For the first time, we collected textual responses in the survey. These comments are addressed in Chapter 4.
• Agency consultations. We engaged in discussions with staff at each of the five major agencies about the operation of their STTR programs and the challenges they face.
• Literature review. Since the start of our research in this area, a number of academic and policy papers have been published that address various aspects of the SBIR-STTR programs, many drawing from the survey and other data made available by our reviews. In addition, other organizations—such as the GAO—have reviewed specific parts of the SBIR-STTR programs. The committee has incorporated references to this work, where useful, into its analysis.

Data Sources and Limitations

Multiple research modalities are especially important because limitations still exist in the data collected for the SBIR-STTR programs.

In particular, there is no unified source of data on outcomes. Some agencies have significant data on outcomes—notably DoD and DoE. Other such as NIH and NASA are still developing their systems. Also, in some cases, such as NSF, the agency preferred not to provide outcomes data on privacy grounds.

Accordingly, the survey was used to provide quantitative insights into the program as a whole. While the limits of this methodology are described in detail in Appendix A, the survey provided important information about both outcomes and the views of recipients on various aspects of program operations and management. In particular, the survey allows for an analysis of the ways in which STTR connects research institutions and small businesses, data that are especially important in assessing the extent to which STTR meets its congressional objective

In short, within the limitations described, the study utilizes a complement of tools to ensure that a wide spectrum of perspectives and expertise is reflected in the findings and recommendations. Appendix A provides an overview of the methodological approaches, data sources, and survey tools used in this study.

________________

Economics of Entrepreneurship, June 29, 2015. Each of these workshops was held in Washington, DC.

ORGANIZATION OF THE REPORT

The analysis and conclusions are organized as follows. Chapter 2 provides a review of program operations, describing the program in some detail and addressing a range of issues related to program management. Chapter 3 provides an overview of applications and awards, illustrating trends in the size of the program over time. Chapter 4 provides a qualitative assessment of the program, based on material drawn from interviews conducted with companies and the wider case studies included in Appendix E, as well as textual responses from survey respondents. Chapter 5 draws on the Academies survey to provide a quantitative assessment, covering the congressional objective, other SBA objectives, and the longer term impact of the program on recipient companies, as well as the role of women and minorities in the program. Chapter 6 provides the findings and recommendations from the study.

The report’s appendixes provide additional information. Appendix A sets out an overview of the methodological approaches, data sources, and survey tools used in this assessment. Appendix B describes key changes to the SBIR and STTR programs from the 2011 reauthorization. Appendix C reproduces the Academies survey instrument. Appendix D lists the research institutions involved in STTR awards. Appendix E presents the case studies of selected firms with STTR awards. Appendix F is the data annex for Chapter 3, and Appendix G is the data annex for Chapter 5. Appendix H provides a glossary of acronyms used, Appendix I includes the agenda from the May 1, 2015, Academies STTR workshop, and Appendix J provides a list of references.