1

Introduction

Small businesses continue to be an important driver of innovation and economic growth,¹ given the challenges of changing global environments and the impacts of the 2008 financial crisis and subsequent recession.² In the face of these challenges, supporting innovative small businesses in their development and commercialization of new products is essential for U.S. competitiveness and national security.

Created in 1982 through the Small Business Innovation Development Act, the Small Business Innovation Research (SBIR) program remains the nation’s largest innovation program for small business. The SBIR program offers competitive awards³ to support the development and commercialization of innovative technologies by small private-sector businesses. At the same time,

___________________

¹ See Z. Acs and D. Audretsch, “Innovation in large and small firms: An empirical analysis,” The American Economic Review, 78(4):678-690, 1988. See also Z. Acs and D. Audretsch, Innovation and Small Firms, Cambridge, MA: The MIT Press, 1991; E. Stam and K. Wennberg, “The roles of R&D in new firm growth,” Small Business Economics, 33:77-89, 2009; E. Fischer and A.R. Reuber, “Support for rapid-growth firms: A comparison of the views of founders, government policymakers, and private sector resource providers,” Journal of Small Business Management, 41(4):346-365, 2003; M. Henrekson and D. Johansson, “Competencies and institutions fostering high-growth firms,” Foundations and Trends in Entrepreneurship, 5(1):1-80, 2009.

² See D. Archibugi, A. Filippetti, and M. Frenz, “Economic crisis and innovation: Is destruction prevailing over accumulation?” Research Policy, 42(2):303-314, 2013. The authors show that “the 2008 economic crisis has severely reduced the short-term willingness of firms to invest in innovation” and also that it “led to a concentration of innovative activities within a small group of fast growing new firms and those firms already highly innovative before the crisis.” They conclude that “the companies in pursuit of more explorative strategies towards new product and market developments are those able to cope better with the crisis.”

³ SBIR awards can be made as grants or as contracts. Grants do not require the awardee to provide an agreed deliverable; for contracts there is often a prototype at the end of Phase II. Contracts are also governed by federal contracting regulations, which are considerably more demanding from the small business perspective. Historically, all Department of Defense (DoD) and National Aeronautics and Space Administration (NASA) awards have been contracts, all National Science Foundation (NSF) and most National Institutes of Health (NIH) awards have been grants, and the Department of Energy (DoE) has used both vehicles.

the program provides government agencies with technical and scientific solutions that address their different missions.

Currently, the program consists of three phases:

• Phase I provides limited funding (up to $100,000 prior to the 2011 reauthorization and up to $150,000 thereafter) for feasibility studies.
• Phase II provides more substantial funding for further research and development (typically up to $750,000 prior to 2012 and $1 million after the 2011 reauthorization).⁴
• Phase III reflects commercialization without providing access to any additional SBIR funding, although funding from other federal government accounts is permitted.

The program has four Congressionally mandated goals: (1) to stimulate technological innovation, (2) to use small business to meet federal research and development needs, (3) to foster and encourage participation by minority and disadvantaged persons in technological innovation, and (4) to increase private-sector commercialization derived from federal research and development.

Research agencies have pursued these goals through the development of SBIR programs that in many respects differ from each other, utilizing the administrative flexibility built into the general program to address their unique mission needs.

In recent years, about 18 percent of Phase I applications to NASA resulted in an award, making it a highly competitive program.⁵ Before 2011, Phase II funding could be awarded only to projects that had successfully completed Phase I. Just over half of Phase II applications to NASA were successfully completed (51 percent). Thus, fewer than 10 percent of Phase I applications resulted in a Phase II award.

Over time, through a series of reauthorizations, SBIR legislation has required federal agencies with extramural research and development (R&D) budgets in excess of $100 million to set aside a growing share of their budgets for the SBIR program. Reaching a set-aside of 2.5 percent, the 11 federal agencies administering the SBIR program obligated $1.9 billion to fund 4,792 SBIR awards in fiscal year (FY) 2014.⁶ These agencies include the Department of Agriculture, Department of Commerce, Department of Defense (DoD), Department of Education, Department of Energy (DoE), Department of Health and Human Services (DHHS), Department of Homeland Security, Department

___________________

⁴ All resource and time constraints imposed by the program are somewhat flexible and are addressed by different agencies in different ways. For example, NIH and to a much lesser degree DoD have provided awards that are much larger than the standard amounts, and NIH has a tradition of offering no-cost extensions to see work completed on an extended timeline.

⁵ NASA data provided to the Academies.

⁶ Small Business Association (SBA) SBIR/STTR awards database, https://www.sbir.gov/analyticsdashboard/, accessed March 15, 2016.

of Transportation, Environmental Protection Agency, NASA, and the National Science Foundation (NSF).

Five agencies administer greater than 98 percent of SBIR funds: DoD, DHHS (particularly the National Institutes of Health [NIH]), DoE, NASA, and NSF (see Figure 1-1 for their respective shares). In FY2014, for example, NASA made 467 SBIR awards, requiring $141.8 million in funds obligated, comprising 7 percent of the overall SBIR total obligations.

In December 2011, Congress reauthorized the program for an additional 6 years,⁷ with a number of important modifications. Many of these modifications—for example, changes in standard award size—were consistent with or followed recommendations made in a 2008 National Research Council (NRC)⁸ report on the SBIR program, a study mandated as part of the program’s

___________________

⁷ Section 5137 of P.L. 112-81.

⁸ Effective July 1, 2015, the institution is called the National Academies of Sciences, Engineering, and Medicine. References in this report to the National Research Council, or NRC, are used in an historic context identifying programs prior to July 1.

2000 reauthorization.⁹ The 2011 reauthorization also called for further studies by the Academies.¹⁰

The first-round assessment resulted in 11 reports, including the 2008 report cited above. (See Box 1-1 for the list of reports.) In a follow-up to the first round, NASA requested from the Academies an assessment focused on operational questions in order to identify further improvements to the program.

This introduction provides context for analysis of the program developments and transitions described in the remainder of the report. The first section of the introduction provides an overview of the program’s history and structure across the federal government. This is followed by a summary of the major changes mandated through the 2011 reauthorization and the subsequent Small Business Administration (SBA) Policy Directive; a review of the program’s advantages and limitations, in particular the challenges faced by entrepreneurs using (and seeking to use) the program and by agency officials running the program; an overview of the study methodology; and a summary of the technical challenges to assessment and our solutions to those challenges.

PROGRAM HISTORY AND STRUCTURE¹¹

A review of the program’s origins and legislative history provides context to its place in the U.S. innovation landscape. During the 1980s, the perceived decline in U.S. competitiveness due to Japanese industrial growth in sectors traditionally dominated by U.S. firms—autos, steel, and semiconductors—led to concerns about future U.S. economic growth.¹² A key concern was the perceived failure of American industry “to translate its research prowess into commercial advantage.”¹³ Although the United States enjoyed dominance in basic research—much of which was federally funded—applying

___________________

⁹ National Research Council, An Assessment of the SBIR Program, Washington, DC: The National Academies Press, 2008. The National Research Council’s first-round assessment of the SBIR program was mandated in the SBIR Reauthorization Act of 2000, P.L. 106-554, Appendix I-H.R. 5667, Section 108.

¹⁰ The National Defense Reauthorization Act for Fiscal Year 2012, P.L. 112-81, Section 5137. It is referenced in the text by its calendar date of passage, December 2011; hence, the 2011 Reauthorization Act.

¹¹ Parts of this section are based on the previous report on the NASA SBIR program, National Research Council, An Assessment of the SBIR Program at the National Aeronautics and Space Administration, Washington, DC: The National Academies Press, 2009.

¹² See J. Alic, “Evaluating competitiveness at the office of technology assessment,” Technology in Society, 9(1):1-17, 1987, for a review of how these issues emerged and evolved within the context of a series of analyses at a Congressional agency.

¹³ D.C. Mowery, “America’s industrial resurgence (?): An overview,” in D.C. Mowery, ed., U.S. Industry in 2000: Studies in Competitive Performance, Washington, DC: National Academy Press, 1999, p. 1. Other studies highlighting poor economic performance in the 1980s include M.L. Dertouzos et al., Made in America: The MIT Commission on Industrial Productivity, Cambridge, MA: The MIT Press, 1989; and O. Eckstein, DRI Report on U.S. Manufacturing Industries, New York: McGraw Hill, 1984.

this research to the development of innovative products and technologies that were taken to market remained challenging. As the great corporate laboratories of the post-war period were buffeted by change, new models such as the cooperative model utilized by some Japanese kieretsu seemed to offer greater sources of dynamism and more competitive firms.

At the same time, new evidence emerged to indicate that small businesses were an increasingly important source of both innovation and job creation.¹⁴ This evidence reinforced recommendations from federal commissions dating back to the 1960s, that federal R&D funding should provide more support for innovative small businesses (recommendations that were opposed by traditional recipients of government R&D funding).¹⁵

Early-stage financial support for high-risk technologies with commercial promise was first advanced by Roland Tibbetts at NSF. In 1976, Mr. Tibbetts advocated for shifting some NSF funding to innovative, technology-based, small businesses. NSF adopted this initiative before other agencies, and, after a period of analysis and discussion, the Reagan administration supported an expansion of this initiative across the federal government. Congress then passed the Small Business Innovation Research Development Act of 1982, which established the SBIR program.

The program was ramped up gradually. Initially, the SBIR program required agencies with extramural R&D budgets in excess of $100 million¹⁶ to set aside 0.2 percent of their funds for SBIR. In the program’s first year of operation (1983), funding totaled $45 million. Over the next 6 years, the set-aside grew to 1.25 percent of agency extramural R&D budgets.¹⁷

The SBIR Reauthorizations of 1992 and 2000

The SBIR program approached reauthorization in 1992 amidst continued worries about the ability of U.S. firms to commercialize inventions. (See Box 1-2.) Finding that “U.S. technological performance is challenged less in the creation of new technologies than in their commercialization and

___________________

¹⁴ See S.J. Davis, J. Haltiwanger, and S. Schuh, Small Business and Job Creation: Dissecting the Myth and Reassessing the Facts, Working Paper No. 4492, Cambridge, MA: National Bureau of Economic Research, 1993. According to Per Davidsson, these methodological fallacies, however, “ha[ve] not had a major influence on the empirically based conclusion that small firms are overrepresented in job creation.” See P. Davidsson, “Methodological concerns in the estimation of job creation in different firm size classes,” Working Paper, Jönköping International Business School, 1996.

¹⁵ For an overview of the origins and history of the SBIR program, see G. Brown and J. Turner, “The federal role in small business research,” Issues in Science and Technology, Summer 1999, pp. 51-58.

¹⁶ That is, those agencies spending more than $100 million on research conducted outside agency labs.

¹⁷ Additional information regarding SBIR’s legislative history can be accessed from the Library of Congress. See http://thomas.loc.gov/cgi-bin/bdquery/z?d097:SN00881:@@@L.

adoption,” the NRC recommended an increase in SBIR funding as a means to improve the economy’s ability to adopt and commercialize new technologies.¹⁸

The Small Business Research and Development Enhancement Act (P.L. 102-564) reauthorized the SBIR program until September 30, 2000, and doubled the set-aside rate to 2.5 percent. The legislation also more strongly emphasized the need for commercialization of SBIR-funded technologies.¹⁹ Legislative language explicitly highlighted commercial potential as a criterion for awarding SBIR contracts and grants.

At the same time, Congress expanded the SBIR program’s purposes to “emphasize the program’s goal of increasing private sector commercialization developed through federal research and development and to improve the federal government’s dissemination of information concerning the small business innovation, particularly with regard to woman-owned business concerns and by socially and economically disadvantaged small business concerns.”²⁰

The Small Business Reauthorization Act of 2000 (P.L. 106-554) extended the SBIR program until September 30, 2008. It also called for an NRC

___________________

¹⁸ See National Research Council, The Government Role in Civilian Technology: Building a New Alliance, Washington, DC: National Academy Press, 1992, p. 29.

¹⁹ See R. Archibald and D. Finifter, “Evaluation of the Department of Defense Small Business Innovation Research program and the Fast Track Initiative: A balanced approach,” in National Research Council, The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative, Washington, DC: National Academy Press, 2000, pp. 211-250.

²⁰ The Small Business Research and Development Enhancement Act (P.L. 102-564), Sec. 102(b)(4).

assessment of the program’s broader impacts, including those on employment, health, national security, and national competitiveness.²¹

The 2011 Reauthorization

The anticipated 2008 reauthorization was delayed in large part by a disagreement between long-time program participants and their advocates in the small business community and proponents of expanded access for venture-backed firms. The issue of venture backing was particularly relevant in biotechnology where proponents argued that the standard path to commercial success invariably includes venture funding at some point.²² Other issues were also difficult to resolve, but the conflict over participation of venture-backed companies dominated the process²³ following an administrative decision to exclude these firms more systematically.²⁴

After a much extended discussion, passage of the National Defense Act of December 2011 reauthorized the SBIR and Small Business Technology Transfer (STTR) programs through FY2017.²⁵ The new law maintained much of the core structure of both programs but made some important changes, which were to be implemented via the SBA’s subsequent Policy Guidance.²⁶

The eventual compromise on the venture funding issue allowed (but did not require) agencies to award up to 25 percent of their SBIR grants or contracts (at NIH, DoE, and NSF) or 15 percent (at the other awarding agencies) to firms that benefit from private, venture capital investment. It is too early in the implementation process to gauge the impact of this change.

The reauthorization made changes in the SBIR program that were recommended in prior Academies reports.²⁷ These included the following:

___________________

²¹ The current assessment is congruent with the Government Performance and Results Act (GPRA) of 1993: http://govinfo.library.unt.edu/npr/library/misc/s20.html. As characterized by the Government Accountability Office (GAO), GPRA seeks to shift the focus of government decision making and accountability away from a preoccupation with the activities that are undertaken—such as grants dispensed or inspections made—to a focus on the results of those activities. See http://www.gao.gov/new.items/gpra/gpra.htm.

²² D.C. Specht, “Recent SBIR extension debate reveals venture capital influence,” Procurement Law, 45:1, 2009.

²³ W.H. Schacht, “The Small Business Innovation Research (SBIR) program: Reauthorization efforts,” Congressional Research Service, Library of Congress, 2008.

²⁴ A. Bouchie, “Increasing number of companies found ineligible for SBIR funding,” Nature Biotechnology, 21(10):1121-1122, 2003.

²⁵ National Defense Authorization Act for Fiscal Year 2012, P.L., December 31, 2011. The STTR program refers to the Small Business Technology Transfer Program. Although similar to the SBIR program, the STTR program requires the small business to “formally collaborate with a research institution in Phase I and Phase II.” Small Business Administration website, “About STTR,” https://www.sbir.gov/about/about-sttr, accessed June 15, 2015.

²⁶ See SBA post, S. Greene, “Implementing the SBIR and STTR Reauthorizations: Our Plan of Attack,” http://www.sbir.gov/news/implementing-sbir-and-sttr-reauthorization-our-plan-attack, accessed February 21, 2012.

²⁷ See Appendix B for a list of the major changes to the SBIR program resulting from the 2011 Reauthorization Act.

• Increased award size limits.
• Expanded program size through modification to the percentage set-aside. The set-aside was increased from 2.5 to 2.6 percent for FY2012, and it will increase by 0.1 percentage point in each subsequent fiscal year until it reaches 3.2 percent. Thereafter, the set-aside percentage will remain at 3.2 percent.²⁸
• Enhanced agency flexibility—for example, for Phase I awardees from other agencies to be eligible for Phase II awards or to add a second Phase II.
• Improved incentives for the utilization of SBIR technologies in agency acquisition programs.
• Explicit requirements for better connecting prime contractors with SBIR awardees.
• Substantial emphasis on developing a more data-driven culture, which has led to several major reforms, including the following:
  • adding numerous areas of expanded reporting
  • extending the Academies’ evaluation
  • adding further evaluation, such as by the Government Accountability Office and Comptroller General
  • tasking the SBA with creating a unified platform for data collection.
• Expanded management resources (through provisions permitting use of up to 3 percent of program funds for [defined] management purposes).
• Expanded commercialization support (through provisions providing companies with direct access to commercialization support funding and through approval of the approaches piloted in Commercialization Pilot Programs).
• Options for agencies to add flexibility by developing other pilot programs—for example, to allow awardees to skip Phase I and apply for a Phase II award directly or for NIH to support a new Phase 0 pilot program.

The reauthorization also made changes that were not mentioned in previous reports of the Academies. These included the following:

• Expansion of the STTR program.²⁹
• Limitations on agency flexibility—particularly in the provision of larger awards.

___________________

²⁸ See “Key Changes in SBIR and STTR Policy Directives-Funding,” http://www.sba.gov/content/key-changes-sbir-and-sttr-policy -directives.

²⁹ The first round study assessed only the SBIR program.

• Introduction of commercialization benchmarks for companies, which must be met if companies are to remain in the program. These benchmarks would be established by each agency.

Other clauses of the legislation affect operational issues, such as the definition of specific terms (such as “Phase III”), continued and expanded evaluation by the Academies, mandated reports from the Comptroller General on combating fraud and abuse within the program, and protection of small firms’ intellectual property within the program.

PREVIOUS RESEARCH ON SBIR

Although there have been previous studies, most notably by the General Accounting Office and the SBA, they have focused on specific aspects or components of the program.³⁰ Prior to the first-round assessment by the Academies, there had been few internal assessments of agency programs. The academic literature on SBIR was also limited,³¹ except for an assessment in the 1990s by Joshua Lerner of the Harvard Business School, who found “that SBIR awardees grew significantly faster than a matched set of firms over a ten-year period.”³²

To help fill this assessment gap and to learn about a large, relatively under-evaluated program, the NRC’s Committee for Government-Industry Partnerships for the Development of New Technologies (GIP—which preceded the NRC’s first-round congressionally mandated study of the SBIR) convened a workshop to discuss the SBIR program’s history and rationale, review existing research, and identify areas for further research and program improvements.³³ In addition, in its report on the SBIR Fast Track Initiative at the Department of Defense, the GIP committee found that the SBIR program contributed to mission goals by funding “valuable innovative projects.”³⁴ It concluded that a

___________________

³⁰ An important step in the evaluation of the program has been to identify existing evaluations of the program. These include U.S. Government Accounting Office, Federal Research: Small Business Innovation Research Shows Success But Can Be Strengthened, Washington, DC: U.S. General Accounting Office, 1992; and U.S. Government Accounting Office, Evaluation of Small Business Innovation Can Be Strengthened, Washington, DC: U.S. General Accounting Office, 1999. There is also a 1999 unpublished SBA study on the commercialization of SBIR Phase II awards from 1983 to 1993 among non-DoD agencies.

³¹ Early examples of evaluations of the SBIR program include S. Myers, R. L. Stern, and M. L. Rorke, A Study of the Small Business Innovation Research Program, Lake Forest, IL: Mohawk Research Corporation, 1983; and Price Waterhouse, Survey of Small High-tech Businesses Shows Federal SBIR Awards Spurring Job Growth, Commercial Sales, Washington, DC: Small Business High Technology Institute, 1985.

³² See J. Lerner, “The government as venture capitalist: The long-run effects of the SBIR Program,” Journal of Business, 72(3), 1999.

³³ See National Research Council, The Small Business Innovation Research Program: Challenges and Opportunities, Washington, DC: National Academy Press, 1999.

³⁴ National Research Council, An Assessment of the Department of Defense Fast Track Initiative, see Chapter III: Recommendations and Findings, p. 32.

significant number of these projects would not have been undertaken absent SBIR funding³⁵ and that DoD’s Fast Track Initiative encouraged the commercialization of new technologies³⁶ and the entry of new firms into the program.³⁷

The GIP committee also found that the SBIR program improved both the development and utilization of human capital and the diffusion of technological knowledge.³⁸ Case studies provided some evidence that the knowledge and human capital generated by the SBIR program have positive economic value, which spills over into other firms through the movement of people and ideas.³⁹ Furthermore, by acting as a “certifier” of promising new technologies, SBIR awards encourage further private-sector investment in an award-winning firm’s technology.⁴⁰

THE ROUND-ONE STUDY OF SBIR

The 2000 SBIR reauthorization mandated that the NRC complete a comprehensive assessment of the SBIR program. ⁴¹ The assessment of the programs at DoD, NIH, NASA, NSF, and DoE began in 2002 and was conducted in three steps. During the first step, the committee developed a research methodology⁴² and gathered information about the program by engaging in discussion with officials at the relevant federal agencies and by inviting those officials to describe program operations, challenges, and accomplishments at two major conferences. These conferences highlighted the important differences in agency goals, practices, and evaluations. They also served to describe the evaluation challenges that arise from the diversity in program objectives and practices.⁴³

The committee implemented the research methodology during the second step. The committee deployed multiple data collection modalities including the first large-scale survey of SBIR recipients, and its researchers conducted case studies of a wide variety of SBIR firms. The Committee then evaluated the results and developed the findings and recommendations presented in their reports for improving the effectiveness of the SBIR program.

During the third step, the committee reported on the program through a series of publications in 2008-2010: five individual volumes on the five major funding agencies and an additional overview volume titled An Assessment of the

___________________

³⁵ Ibid, p. 32.

³⁶ Ibid, p. 33.

³⁷ Ibid, p. 34.

³⁸ Ibid, p. 33.

³⁹ Ibid, p. 33.

⁴⁰ Ibid, p. 33.

⁴¹ SBIR Reauthorization Act of 2000, P.L. 106-554, Appendix I-H.R. 5667, Section 108.

⁴² National Research Council, An Assessment of the Small Business Innovation Research Program: Project Methodology, Washington, DC: The National Academies Press, 2004.

⁴³ Adapted from National Research Council, SBIR: Program Diversity and Assessment Challenges.

SBIR Program.⁴⁴ Together, these reports provided the first detailed and comprehensive review of the SBIR program and, as noted above, became an important input into SBIR reauthorization prior to December 2011. (See Box 1-1.)

THE CURRENT, SECOND-ROUND STUDY: CHALLENGES AND OPPORTUNITIES

The set of reports from the Academies’ first-round study of the SBIR program found that the program was, overall, “sound in concept and effective in practice.”⁴⁵ Furthermore, in its 2009 review of the NASA SBIR program, the committee concluded, “The NASA SBIR program is making significant progress in achieving the congressional goals for the program. [emphasis in original] Keeping in mind NASA’s unique mission and the recent significant changes to the program, the SBIR program is sound in concept and effective in practice at NASA.”⁴⁶ The current study, described in the Statement of Task in Box 1-3, provides a second snapshot to measure the program’s progress against its legislative goals.

Along with the current volume, several workshops and other publications will fully address this Statement of Task. The workshops convened included one on the participation of women and minorities in the SBIR/STTR programs (February 2013), one on the evolving role of university participation in the program (February 2014), and one on the relationship between state innovation programs and the SBIR program (October 2014—See Box 1-4).

The current volume is focused on updating the 2009 assessment of the NASA SBIR program, by updating data, providing new descriptions of recent programs and developments, and providing fresh company case studies. Guided by the Statement of Task, we have sought answers to questions such as the following:

• Are there initiatives and programs within NASA that have made a significant difference to outcomes and in particular to agency take-up of SBIR-funded technologies?
  • Can they be replicated and expanded?
• What are the main barriers to meeting Congressional objectives more fully?
• What program adjustments would better support commercialization?
• Are there tools that would expand utilization by woman- and minority-owned firms and participation by female and minority principal investigators?

__________________

• Can links with universities be improved? In what ways and to what effect?
• Are there aspects of the program that make it less attractive? Could they be addressed?
• What can be done to expand access in underserved states while maintaining the competitive character of the program?
• Can the program generate better data on both process and outcomes and use those data to fine-tune program management?

TABLE 1-1 SBIR State Match Funds

SOURCE: Presentation by M. P. Feldman, “SBIR State Matching Programs: Science Experiments in the Laboratories of Democracy” at the National Academy of Sciences, Engineering, and Medicine workshop on “SBIR/STTR and the Commercialization Challenge,” Washington, DC, April 12, 2016.

STUDY METHODOLOGY

The SBIR/STTR programs are unique in terms of scale and mission focus. In addition, the evidence suggests that no truly comparable programs exist in the United States, and those in other countries operate in such different ways that their relevance is limited.⁴⁷ Thus, it is difficult to identify comparable programs against which to benchmark the SBIR/STTR program results.

Assessing the SBIR program at NASA is challenging for other reasons as well. Unlike some agencies where the mission and related objectives have remained stable, NASA’s core missions have been in a state of flux for much of the past 10 years. For example, the role of human exploration in space remains uncertain, areas of focus may change, and the role of the private sector in space exploration is evolving rapidly. In addition, NASA has undergone numerous major reorganizations, which has resulted in significant changes to the pathway into NASA for emerging technologies.

Focus on Legislative Objectives

It is important to emphasize at the outset that this volume—and this study—do not seek to provide a comprehensive review of the value of the SBIR program, particularly as measured against other possible alternative uses of federal funding. Such a review is beyond our scope. Rather, our work focuses on assessing the extent to which the SBIR program at NASA has met the congressional objectives set for the program, determining in particular whether recent initiatives have improved program outcomes, and providing recommendations for further improvements to the program.⁴⁸

Thus, as in the first round, the objective of this study is not to consider whether or not SBIR should exist. Congress has already decided affirmatively on this issue, most recently in the 2011 reauthorization of the program.⁴⁹ Rather, this study is charged with “providing assessment‐based findings of the benefits and costs of SBIR . . . to improve public understanding of the program, as well as recommendations to improve the program’s effectiveness.” Also following the first round, this study “will not seek to compare the value of one area with other areas; this task is the prerogative of the Congress and the Administration acting through the agencies. Instead, the study is concerned with the effective review of each area.”⁵⁰

__________________

Defining Commercialization

Commercialization offers practical and definitional challenges. As described in Chapter 5, several different definitions of commercialization can be used when describing the SBIR program. For this reason, the study uses more than one simple definition. For example, the simple percentage of funded projects that reach the marketplace is not the only measure of commercial success.

In the private sector, commercial success over the long term requires profitability. However, in the short term, commercialization can involve many different aspects of commercial activity, from product rollout to patenting to licensing to acquisition. Even during new product rollout, companies often do not generate immediate profits, and they do not necessarily earn a profit on all product offerings. This report uses multiple metrics to address the question of commercialization (see Chapter 5).

In the case of NASA, there are special challenges to defining commercialization. NASA is an “acquisition agency”—it utilizes the results of at least a portion of SBIR-funded research—unlike NSF and NIH. It also differs from DoD—the other major acquisition agency—in that it rarely purchases a sufficient quantity of any product to create a viable commercial market. Therefore, we stress that commercialization at NASA includes as a primary characteristic the take-up of SBIR-funded technologies for use within NASA, regardless of whether a viable commercial market results. In addition, the study recognizes that some NASA’s SBIR awards support the development of aeronautics-related technologies for which NASA has no direct acquisition activity and that have commercialization potential outside of NASA programs.

Quantitative Assessment Methods

From a more practical perspective, several issues relate to the application of quantitative assessment methods, including decisions about which kinds of program participants should be targeted for survey deployment, the number of responses that are appropriate, selection bias, nonresponse bias, the design and implementation of survey questionnaires, and the level of statistical evidence required for drawing conclusions in this case. These and other issues were discussed at a workshop and summarized in a 2004 report.⁵¹ The peer-reviewed study methodology developed by the first-round committee provided the baseline for that study and for follow-on studies—such as this one.⁵²

__________________

Survey Development

For the current study, the committee developed and deployed a new survey of SBIR recipients. Although the committee based the survey⁵³ closely on previous surveys, particularly one deployed in 2005, it made several improvements. Most notably, it made an ambitious but ultimately unsuccessful effort to develop a comparison group to provide context and a benchmark for analyzing results (this effort is discussed in Appendix A).⁵⁴ Randomly assigned control groups were found to be not a possible alternative because of the nature of the SBIR program. Efforts to develop comparison groups from Phase I awardees that had not received a Phase II award, or from Phase II SBIR awardees, were also not successful. Likewise, efforts to identify SBIR-like companies from industry data sources to serve as a comparison group were not successful because sufficiently detailed and structured information about companies was not available.

The survey more deeply explored the program’s demographics. It also included questions about the role of agency liaisons, who deal with award operations and thereby provide a link between individual projects and NASA. Furthermore, it offered unique opportunities to collect qualitative opinions and recommendations for improvement from award recipients. The survey generated 179 responses from NASA Phase II awardees and is an important component of the research conducted for this volume. Appendix A provides a detailed discussion of the issues related to quantitative methodologies, as well as a review of potential biases.

It is recognized that there are significant limitations on the conclusions that can be drawn from this quantitative assessment, and this recognition is reflected in the language of the findings and recommendations (Chapter 8). At the same time, drawing on quantitative analysis is a crucial component of the overall study, particularly given the need to identify and assess outcomes that are to be found only at the level of individual projects and participating companies.

__________________

A Complement of Approaches

Partly because of these limitations, the 2004 review of methodology stressed the importance of utilizing a complement of research modalities,⁵⁵ an approach that has been adopted here. Although quantitative assessment represents the bedrock of the research and provides insights and evidence that could not be generated through any other modality, it is, in and of itself, insufficient to address the multiple questions posed in this analysis. Consequently, we undertook a series of additional activities:

• Case studies. We conducted in-depth case studies of 10 NASA SBIR awardees. These selected companies were geographically and demographically diverse, sponsored by several different NASA Centers, and at different stages of the company life cycle. Lessons learned from the case studies are described in Chapter 7, and the case studies themselves are included as Appendix E.
• Workshops. We conducted workshops, including workshops to discuss the participation of women and minorities, as well as the role of universities, in the SBIR program⁵⁶ to allow stakeholders, agency staff, and academic experts to provide insights into program operations and to identify issues that need to be addressed.
• Analysis of agency data. As appropriate, we analyzed and included data from NASA that cover various aspects of SBIR activities.
• Open-ended responses from SBIR awardees. For the first time, we collected textual responses in the survey. More than 150 awardees provided narrative comments, which are discussed in Chapter 7.
• Agency consultations. We engaged in discussions with agency staff at several NASA centers and facilities about the operation of their program and the challenges they face.
• Literature review. Since the start of our research in this area, a number of papers have been published that address various aspects of the SBIR program. In addition, other organizations—such as the Government Accountability Office—have reviewed specific parts of the SBIR program. We incorporated references to this work, when useful, into our analysis.

__________________

Data Sources and Limitations

Multiple research modalities are especially important because limitations still exist in the data collected for the SBIR program. As described in Chapter 3, NASA has not developed or maintained a comprehensive dataset on outcomes from awards and did not provide data about the take-up of SBIR-funded technologies within NASA. In addition, NASA has not made a systematic effort to utilize the Federal Procurement Data System (FPDS), which tracks federal contracts; and based on the committee’s research, it appears that NASA contract officers are not trained to recognize and record contracts based on SBIR technologies as Phase III contracts. However, as noted in Chapter 3, utilization of NASA’s Electronic Handbook has the potential to greatly enhance NASA’s access to data on its SBIR program.

The lack of data from NASA makes the 2011 Survey data all the more important in assessing SBIR outcomes and processes at NASA. That said, the Survey data also has limitations due to a small response rate, but the data have been analyzed to extract as much information on outcomes as possible.⁵⁷ Future evaluation studies may be able to draw on NASA’s Electronic Handbook (EHB) for in-house data.

Cooperation with NASA

In general, we received sufficient cooperation from NASA and the NASA Centers. Numerous discussions took place between agency staff and the Academies to identify and request information, and NASA followed through in providing the requested data, papers, and presentations.

Late in the committee’s deliberative process, it received from a reviewer of this report a draft copy of a 2015 report, commissioned by NASA, which analyzed the SBIR/STTR programs in the mission directorates. Since the analysis was based on data from the Electronic Handbooks (EHBs), it was of particular interest to this committee’s work. Because the report had not been publicly released, the committee requested official copies of this and two other consultant reports commissioned by NASA. However, SBIR program executives at the agency were either unwilling or unable to provide these reports.

With regard to data from the Electronic Handbooks, the NASA Program Office had initially indicated that EHB data would not be provided for the study. Some data were eventually delivered in March 2015, but they were very incomplete and deemed unusable for the study. The committee urges that in any future evaluations of the SBIR/STTR programs, NASA provide access to EHB data as well as to any relevant studies, so that the resulting reports can be based on the full range of information that exists on the programs.

In short, within the limitations described, the study has used a complement of tools to ensure that a full spectrum of perspectives and expertise

__________________

is reflected in the findings and recommendations. Appendix A provides an overview of the methodological approaches, data sources, and survey tools used in this study.

ORGANIZATION OF THE REPORT

Analyses and findings are organized as follows: Chapter 2 reviews program operations, including the role of agency liaison offices, auditing, and contracts. Chapter 3 describes and analyzes agency initiatives that have been developed and implemented over the past 8 to 10 years, largely aimed at improving program outcomes. Chapter 4 reviews NASA data concerning applications and awards, drawing out demographic and geographic differences as well as previous experience with the program. Chapter 5 provides a quantitative assessment of the program. This is based primarily on the 2011 Survey given the paucity of data from NASA or other sources. Chapter 6 addresses data and NASA efforts concerning the participation of women and minorities in the program. Chapter 7 draws on company case studies and the textual responses from survey respondents to provide a qualitative picture of program operations, issues, and possible solutions. Chapter 8 provides the findings and recommendations from the study.

The report’s appendixes provide additional information. Appendix A provides an overview of the methodological approaches, data sources, and survey tools used in this assessment. Appendix B describes key changes to the SBIR program from the 2011 reauthorization. Appendix C reproduces the 2011 Survey instrument. Appendix D lists the universities involved in NASA SBIR awards. Appendix E presents the case studies of selected NASA SBIR firms. Appendix F and Appendix G serve as annexes to Chapter 5, the first with additional data from the 2011 Survey and the second with supplementary data from DoD about NASA SBIR awards. Finally, Appendix H provides a glossary of acronyms used, and Appendix I provides a list of references.