SBIR at the National Science Foundation (2015)

1

Introduction

Small businesses continue to be an important driver of innovation and economic growth,¹ despite the challenges of changing global environments and the impacts of the 2009 financial crisis and subsequent recession.² In the face of these challenges, supporting small businesses in their development and commercialization of new products is essential for U.S. competitiveness and national security.

The SBIR was started as a pilot program at the National Science Foundation (NSF) in the late 1970s. It received legislative authorization in 1982, through the Small Business Innovation Development Act, and subsequently

___________________

¹See Zoltan Acs and David Audretsch, “Innovation in Large and Small Firms: An Empirical Analysis,” The American Economic Review 78(4):678-690. See also Zoltan Acs and David Audretsch, Innovation and Small Firms (Cambridge, MA: The MIT Press, 1991); Erik Stam and Karl Wennberg, “The Roles of R&D in New Firm Growth,” Small Business Economics 33(2009):77-89; Eileen Fischer and A. Rebecca Reuber, “Support for Rapid-Growth Firms: A Comparison of the Views of Founders, Government Policymakers, and Private Sector Resource Providers,” Journal of Small Business Management 41(4):346-365; Magnus Henrekson and Dan Johansson, “Competencies and Institutions Fostering High-Growth Firms,” Foundations and Trends in Entrepreneurship 5, no. 1 (2009): 1-80.

²Citing recent data, some analysts have recently questioned the strength of small businesses as a driver of economic growth. See, for example, Newmark et al. “Do Small Businesses Create More Jobs? ...” in The Review of Economics and Statistics 93, issue 1, pp 16-23. Others have pointed out the negative but temporary impact of the 2008 financial crisis on credit and investment on small business led growth. See Daniele Archibugi, Andrea Filippetti, and Marion Frenz, “Economic Crisis and Innovation: Is Destruction Prevailing Over Accumulation?” Research Policy 42, no. 2 (March 2013): 303-314. The authors show that “the 2008 economic crisis has severely reduced the short-term willingness of firms to invest in innovation” and that it “led to a concentration of innovative activities within a small group of fast growing new firms and those firms already highly innovative before the crisis.” They conclude that “the companies in pursuit of more explorative strategies towards new product and market developments are those to cope better with the crisis.”

the Small Business Research and Development Enhancement Act of 1992, and the Small Business Innovation Research Program Reauthorization Act of 2000. The Small Business Innovation Research (SBIR) program is the nation’s largest innovation program for small business. Through FY2014, it had made more than 150 thousand awards totaling nearly $40 billion.³

The SBIR program offers competitive awards to support the development and commercialization of innovative technologies by small private-sector businesses. At the same time, the program provides government agencies with technical and scientific solutions that address their different missions.

Currently, the SBIR program provides funding in three phases:

• Phase I provides limited funding (up to $100,000 prior to the 2011 reauthorization and up to $150,000 thereafter) for feasibility studies.
• Phase II provides more substantial funding for further research and development (typically up to $750,000 prior to 2012 and $1 million after the 2011 reauthorization).⁴
• Phase III reflects commercialization without providing access to any additional SBIR funding, although funding from other federal government accounts is permitted.

Congress mandated four goals for the program: “(1) to stimulate technological innovation; (2) to use small business to meet federal research and development needs; (3) to foster and encourage participation by minority and disadvantaged persons in technological innovation; and (4) to increase private sector commercialization derived from federal research and development.”

The research agencies have pursued these goals through the development of SBIR programs that differ from each other in many respects, utilizing the administrative flexibility built into the general program to address their unique mission needs.

SBIR awards are highly competitive. Between 2003 and 2012,⁵ about 18 percent of Phase I applications and 44 percent of Phase II applications to the National Science Foundation (NSF) resulted in an award.

Over time, through a series of reauthorizations, SBIR legislation has required federal agencies with extramural research and development (R&D) budgets in excess of $100 million to set aside a growing share of their budgets for the SBIR program. The set-aside reached 2.5 percent following the 2000

___________________

³Small Business Administration, <http://www.sbir.gov/past-awards>.

⁴All resource and time constraints imposed by the program are somewhat flexible and are addressed by different agencies in different ways. For example, the National Institutes of Health (NIH), and to a much lesser degree the Department of Defense (DoD), have provided awards that are much larger than the standard amounts, and NIH has a tradition of offering no-cost extensions to see work completed on an extended timeline.

⁵National Science Foundation (NSF) data provided to the Academies.

FIGURE 1-1 SBIR/STTR funding by federal agency share, FY2010.

SOURCE: Small Business Administration (SBA) SBIR website. Accessed November 1, 2013.

reauthorization. In fiscal year (FY) 2010, the 11 federal agencies administering the SBIR program disbursed $2.51 billion, and in FY 2014 they disbursed $2.08 billion.⁶

Five agencies administer greater than 96 percent of SBIR/STTR funds: the Department of Defense (DoD), the Department of Health and Human Services (HHS) (particularly the National Institutes of Health [NIH]), the Department of Energy (DoE), the National Aeronautics and Space Administration (NASA), and NSF. Figure 1-1 shows the percentages of SBIR and related Small Business Technology Transfer (STTR) funding in FY2010 provided by each of these five federal agencies, as well as the remaining combined percentage provided by all other agencies that administer SBIR/STTR programs. In FY 2010, for example, NSF made 549 SBIR/STTR awards amounting to $118.7 million. In FY 2014, NSF made 411 SBIR/STTR awards amounting to $130.2 million.

In December 2011, Congress reauthorized the program for an additional 6 years,⁷ with a number of important modifications. Many of these modifications—for example, changes in standard award size—were based on recommendations

___________________

⁶Small Business Association SBIR/STTR annual report, accessed April 2015, <http://www.sbir.gov>.

⁷Section 5137 of Public Law 112-81.

made in a 2008 National Research Council (NRC)⁸ report on the SBIR program.⁹ The reauthorization also called for further studies by the Academies.

In a follow-up to the first-round assessment, NSF requested that the Academies provide a subsequent round of assessment, focused on operational questions with a view to identifying further improvements to the program.

This introduction provides a context for the analysis of program developments and transitions described in the remainder of the report. The first section provides an overview of the SBIR program’s history across the federal government. This is followed by a summary of the major changes mandated by the 2011 reauthorization and the subsequent Small Business Administration (SBA) Policy Directive; a review of the program’s advantages and limitations, in particular the challenges faced by entrepreneurs using (and seeking to use) the program and by agency officials running the program; and a summary of the technical challenges facing this assessment and the committee’s solutions to those challenges.

PROGRAM HISTORY AND STRUCTURE¹⁰

During the 1980s, the perceived challenge of Japanese industrial growth in sectors traditionally dominated by U.S. firms—autos, steel, and semiconductors—led to serious concerns about U.S. competitiveness.¹¹ A key concern was the perceived failure of American industry “to translate its research prowess into commercial advantage.”¹² Although the United States enjoyed dominance in basic research—much of which was federally funded—applying this research to the development of innovative products and technologies remained challenging. As the great corporate laboratories of the post-war period were buffeted by

___________________

⁸Effective July 1, 2015, the institution is called the National Academies of Sciences, Engineering, and Medicine. References in this report to the National Research Council, or NRC, are used in a historic context identifying programs prior to July 1.

⁹National Research Council, An Assessment of the SBIR Program, (Washington, DC: The National Academies Press, 2008). The Academies’ first-round assessment of the SBIR program was mandated in the SBIR Reauthorization Act of 2000, Public Law 106-554, Appendix I-H.R. 5667, Section 108.

¹⁰Parts of this section are based on the Academies’ previous report on the NSF SBIR program—National Research Council, An Assessment of the SBIR Program at the National Science Foundation (Washington, DC: The National Academies Press, 2008).

¹¹See John Alic, “Evaluating Competitiveness at the Office of Technology Assessment,” Technology in Society 9, no. 1 (1987): 1-17 for a review of how these issues emerged and evolved within the context of a series of analyses at a Congressional agency.

¹²David C. Mowery, “America’s Industrial Resurgence (?): An Overview,” in U.S. Industry in 2000: Studies in Competitive Performance, David C. Mowery, ed., (Washington, DC: National Academy Press, 1999), p. 1. Other studies highlighting poor economic performance in the 1980s include Michael L. Dertouzos et al., Made in America: The MIT Commission on Industrial Productivity (Cambridge, MA: The MIT Press, 1989); and Otto Eckstein, DRI Report on U.S. Manufacturing Industries (New York: McGraw Hill, 1984).

change, new models such as the cooperative model utilized by some Japanese kieretsu offered new sources of dynamism and more competitive firms.¹³

At the same time, new evidence emerged to indicate that small businesses were an increasingly important source of innovation and job creation. David Birch and others suggested that national policies should promote and build on these developments.¹⁴ This evidence reinforced recommendations from federal commissions dating back to the 1960s, that is, that federal R&D funding should provide more support for innovative small businesses (which was opposed by traditional recipients of federal R&D funding).¹⁵

Early-stage financial support for high-risk technologies with commercial promise was first advanced by Roland Tibbetts at NSF, who in 1976 advocated for shifting some NSF funding to innovative technology-based small businesses. Following a period of analysis and discussion, and support by the Reagan administration for an expansion of the practice across federal agencies, Congress passed the Small Business Innovation Research Development Act of 1982, establishing the SBIR program.

The scale of the SBIR program was gradually increased. At the start, the SBIR program required agencies with extramural R&D budgets in excess of $100 million¹⁶ to set aside 0.2 percent of their funds for SBIR. Program funding totaled $45 million in the program’s first year of operation (1983). Over the next 6 years, the set-aside grew to 1.25 percent.¹⁷

___________________

¹³David L. Birch, “Who Creates Jobs?” National Affairs, <http://www.nationalaffairs.com/doclib/20080708_1981651whocreatesjobsdavidlbirch.pdf>. Accessed August 13, 2014. Birch’s work greatly influenced perceptions of the role of small firms. Over the past 20 years, it has been carefully scrutinized, leading to the discovery of some methodological flaws, namely making dynamic inferences from static comparisons, confusing gross and net job creation, and admitting biases from chosen regression techniques. See Steven J. Davis, John Haltiwanger, and Scott Schuh, “Small Business and Job Creation: Dissecting the Myth and Reassessing the Facts, Working Paper No. 4492 (Cambridge, MA: National Bureau of Economic Research, 1993). According to Per Davidsson, these methodological fallacies, however, “ha[ve] not had a major influence on the empirically based conclusion that small firms are over-represented in job creation.” See Per Davidsson, “Methodological Concerns in the Estimation of Job Creation in Different Firm Size Classes” (working paper, Jönköping International Business School, 1996).

¹⁴David Birch, “Who Creates Jobs?”

¹⁵For an overview of the origins and history of the SBIR program, see George Brown and James Turner, “Reworking the Federal Role in Small Business Research,” Issues in Science and Technology, Summer 1999, pp. 51-58.

¹⁶That is, those agencies spending more than $100 million on research conducted outside agency labs.

¹⁷Additional information regarding the SBIR program’s legislative history can be accessed from the Library of Congress. See <http://thomas.loc.gov/cgi-bin/bdquery/z?d097:SN00881:@@@L>.

The SBIR Reauthorizations of 1992 and 2000

The SBIR program approached reauthorization in 1992 amidst continued worries about the U.S. economy’s capacity to commercialize inventions. Finding that “U.S. technological performance is challenged less in the creation of new technologies than in their commercialization and adoption,” the Academies recommended an increase in SBIR funding as a means to improve the economy’s ability to adopt and commercialize new technologies.¹⁸

The Small Business Research and Development Enhancement Act (P.L. 102-564) reauthorized the SBIR program until September 30, 2000, and doubled the set-aside rate to 2.5 percent. The legislation also more strongly emphasized the need for commercialization of SBIR-funded technologies.¹⁹ The 1992 legislative language explicitly highlighted commercial potential as a criterion for awarding SBIR grants, as indicated in Box 1-1.

At the same time, Congress expanded the SBIR program’s purposes to “emphasize the program’s goal of increasing private sector commercialization developed through Federal research and development and to improve the federal government’s dissemination of information concerning the small business

___________________

¹⁸See National Research Council, The Government Role in Civilian Technology: Building a New Alliance (Washington, DC: National Academy Press, 1992), p. 29.

¹⁹See Robert Archibald and David Finifter, “Evaluation of the Department of Defense Small Business Innovation Research Program and the Fast Track Initiative: A Balanced Approach,” in National Research Council, The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative (Washington, DC: National Academy Press, 2000), pp. 211-250.

innovation, particularly with regard to woman-owned business concerns and by socially and economically disadvantaged small business concerns.”

The Small Business Reauthorization Act of 2000 (P.L. 106-554) extended the SBIR program until September 30, 2008. It also called for an NRC assessment of the program’s broader impacts, including those on employment, health, national security, and national competitiveness.²⁰

THE 2011 REAUTHORIZATION

The anticipated 2008 reauthorization was delayed in large part by a disagreement between long-time program participants and their advocates in the small business community and proponents of expanded access for venture-backed companies, particularly in biotechnology where proponents argued that the standard path to commercial success involves venture funding at some point.²¹ Other issues were also difficult to resolve, but the conflict over participation of venture capital-backed companies dominated the process²² following an administrative decision to exclude these companies more systematically.²³

After a much extended discussion, the SBIR and STTR programs were reauthorized through FY2017 through passage of the National Defense Act of December 2011. The new law maintained much of the core structure of both programs but made some important changes, which were to be implemented via the SBA’s subsequent Policy Guidance.

The eventual compromise on the venture funding issue allowed (but did not require) agencies to set aside 25 percent of SBIR funding (at NIH, DoE, and NSF) or 15 percent (at the other awarding agencies) for participation by firms benefiting from private, venture capital investment. It is too early in the implementation process to gauge the impact of this change.

Several changes to the program made through reauthorization reflected recommendations by the Academies in prior reports, including the following:

• Increased award size limits
• Expanded program size

___________________

²⁰The current assessment is congruent with the Government Performance and Results Act (GPRA) of 1993: <http://govinfo.library.unt.edu/npr/library/misc/s20.html>. As characterized by the Government Accountability Office (GAO), GPRA seeks to shift the focus of government decision making and accountability away from a focus on the activities that are undertaken—such as grants dispensed or inspections made—to a focus on the results of those activities. See <http://www.gao.gov/new. items/gpra/gpra.htm>.

²¹Damien C. Specht, “Recent SBIR Extension Debate Reveals Venture Capital Influence,” Procurement Law 45 (2009): 1.

²²Wendy H. Schacht, “The Small Business Innovation Research (SBIR) Program: Reauthorization Efforts,” Congressional Research Service, Library of Congress, 2008.

²³Aaron Bouchie, “Increasing Number of Companies Found Ineligible for SBIR Funding,” Nature Biotechnology 21, no. 10 (2003): 1121-1122.

• Enhanced agency flexibility—for example, to utilize Phase I awards from other agencies or to add a second Phase II
• Improved incentives for the utilization of SBIR technologies in agency acquisition programs
• Explicit requirements to better connect prime contractors with the SBIR program
• Substantial emphasis on developing a more data-driven culture. Addressing Academies recommendations in this area has led to several significant reforms, including the following:
  • adding numerous areas of expanded reporting
  • extending the Academies’ evaluation
  • adding further evaluation from other expert bodies, such as the Comptroller General
  • tasking the SBA with creating a unified data platform
• Expanded management resources (through provisions permitting use of up to 3 percent of program funds for defined management purposes)
• Expanded commercialization support (through provisions providing companies with direct access to commercialization support funding and through approval of the approaches piloted in the Commercialization Pilot Program)
• Flexibility for agencies to develop other pilot programs—for example, to skip Phase I, or for NIH to support a new Phase 0 pilot program.

In addition, the reauthorization made changes that were not recommended in previous Academies reports, including the following:

• Expansion of the STTR program
• Limitations on some aspects of agency flexibility, particularly caps on the provision of larger awards
• Introduction of commercialization benchmarks, established by agencies, that companies must meet to remain in the program.

Other clauses of the legislation related to operational issues, such as the definition of specific terms (e.g., “Phase III”), continued and expanded evaluation by the Academies and mandated reports from the Comptroller General on combating waste, fraud, and abuse within the program, and measures of the protection of small companies’ intellectual property within the program.

PREVIOUS RESEARCH ON SBIR

Previous studies, most notably by the General Accounting Office and the Small Business Administration, have focused on specific aspects or components

of the program.²⁴ Prior to the first round of the assessment by the Academies, there had been few internal assessments of agency programs. The academic literature on SBIR was also limited.²⁵ Writing in the 1990s, Joshua Lerner of the Harvard Business School positively assessed the program, finding “that SBIR awardees grew significantly faster than a matched set of firms over a ten-year period.”²⁶ These findings were consistent with the corporate finance literature on capital constraints and the growth literature on the importance of localization economies.²⁷

To help fill this assessment gap, and to learn about a large, relatively under-evaluated program, the Academies’ Committee for Government-Industry Partnerships for the Development of New Technologies (GIP Committee) prepared the first comprehensive discussion of the SBIR program’s history and rationale, reviewed existing research, and identified areas for further research and program improvements.²⁸ It reported the following:

• The SBIR program enjoyed strong support of parts of the federal government, as well as of the country at large.
• The size and significance of the SBIR program underscored the need for more research on its effectiveness.
• The primary emphasis on commercialization within the SBIR program required further clarification.
• Evaluation methodologies required additional work.²⁹

___________________

²⁴An important step in the current evaluation of the SBIR program has been to identify already existing evaluations. These include U.S. General Accounting Office, Federal Research: Small Business Innovation Research Shows Success But Can Be Strengthened RCED-92-32, (Washington, DC: U.S. General Accounting Office, 1992); and U.S. General Accounting Office, Federal Research: Evaluation of Small Business Innovation Can Be Strengthened, T-RCED-99-198 (Washington, DC: U.S. General Accounting Office, 1999). There is also a 1999 unpublished SBA study of the commercialization of SBIR surveys Phase II awards from 1983 to 1993 among non-DoD agencies. Effective July 7, 2004, the GAO’s legal name was changed from the General Accounting Office to the Government Accountability Office.

²⁵Early examples of evaluations of the SBIR program include Summer Myers, Robert L. Stern, and Marcia L. Rorke, A Study of the Small Business Innovation Research Program (Lake Forest, IL: Mohawk Research Corporation, 1983); and Price Waterhouse, Survey of Small High-tech Businesses Shows Federal SBIR Awards Spurring Job Growth, Commercial Sales (Washington, DC: Small Business High Technology Institute, 1985).

²⁶Joshua Lerner, “The Government as Venture Capitalist: The Long-Run Effects of the SBIR Program,” September 1996, National Bureau of Economic Research, Working Paper No. 5753, <http://www.nber.org/papers/w5753.pdf>. Accessed August 13, 2014.

²⁷See Michael Porter, “Clusters and Competition: New Agendas for Government and Institutions,” in On Competition (Boston, MA: Harvard Business School Press, 1998).

²⁸See National Research Council, The Small Business Innovation Research Program: Challenges and Opportunities (Washington, DC: National Academy Press, 1999).

²⁹National Research Council, An Assessment of the DoD SBIR Fast Track Initiative (Chapter III: Recommendations and Findings), p. 32.

In a later, more comprehensive review of the DoD SBIR program, the GIP Committee found that the program contributed to mission goals by funding valuable innovative projects. It also concluded that a significant number of these projects would not have been undertaken absent SBIR funding and that Fast Track (an initiative to expedite decision-making for SBIR awards to companies that have commitments from outside investors) encouraged the commercialization of new technologies and the entry of new firms into the program.³⁰

The GIP Committee also found that the SBIR program affected the development and utilization of human capital and the diffusion of technological knowledge. Case studies showed that the knowledge and human capital generated by the SBIR program have positive economic value, which spills over into other firms through the movement of people and ideas. Furthermore, by acting as a “certifier” of promising new technologies, SBIR awards encourage further private-sector investment in an award-winning firm’s technology.³¹

THE ROUND-ONE STUDY OF SBIR

Drawing on these findings and recommendations, the 2000 SBIR reauthorization mandated a comprehensive assessment of the nation’s SBIR program, which was conducted in three steps. During the first step, the committee charged with carrying out the study developed a research methodology, which was approved by an independent Academies’ panel of experts. This committee gathered information about the program by interviewing officials at the relevant federal agencies and by inviting these officials to describe program operations, challenges, and accomplishments at two major conferences. These conferences highlighted the important differences in agency goals, practices, and evaluations, as well as the evaluation challenges that arise from the diverse program objectives and practices.³²

The research methodology was implemented during the second step of the study. The committee deployed multiple survey instruments and conducted case studies of a wide variety of SBIR companies. The committee then evaluated the results and developed the findings and recommendations for improving the effectiveness of the SBIR program. It is important to stress that the respondents to the survey represent only a subset of all awardees.³³

During the third step of the study, the committee reported on the program through a series of publications in 2008-2010: five individual volumes on the five major funding agencies and an additional overview volume. Together, these reports provided the first detailed and comprehensive review of the SBIR pro-

___________________

³⁰Ibid, 33.

³¹Ibid, 33.

³²Adapted from National Research Council, Small Business Innovation Research Program: Program Diversity and Assessment Challenges (Washington, DC: The National Academies Press, 2004).

³³Box A-1 in Appendix A gives and overview of potential sources of bias in survey results.

gram and, as noted above, became an important input into SBIR reauthorization prior to December 2011. Box 1-2 highlights accomplishments of the round-one, mandated assessment of the SBIR.

THE CURRENT, SECOND-ROUND STUDY: CHALLENGES AND OPPORTUNITIES

The set of reports from the Academies’ first-round study of the SBIR program established that, overall, the program is “sound in concept and effective in practice.” The second-round study seeks to understand how the program could work better. Box 1-3 highlights the Statement of Task of the second-round study.

The current volume is focused on updating the round-one committee’s 2007 assessment of the NSF SBIR program, by updating data, providing new descriptions of recent program and developments, providing fresh company case studies. This volume, in particular, focuses on the efforts made at NSF in recent years to improve the SBIR program. Guided by this Statement of Task, we have sought answers to questions such as the following:

• Are there initiatives and programs within NSF that have made a significant difference to outcomes and in particular to commercialization of SBIR-funded technologies?
• Can they be replicated and expanded?
• What are the main barriers to meeting congressional objectives more fully?
• What program adjustments would better support commercialization?
• What tools would expand utilization by woman and minorityowned firms and participation by female and minority principal investigators?
• Can links with universities be improved?
• Why do some firms simply drop out of the program?
• Are there aspects of the program that make it less attractive? Could they be addressed?
• What can be done to expand access in underserved states while maintaining the competitive character of the program?
• Can the program generate better data on both process and outcomes and use those data to fine-tune program management?

STUDY METHODOLOGY

It is always useful when assessing government programs to identify comparable programs for appropriate benchmarking. However, in the committee’s experience, there are no truly comparable programs in the United States, and those in other countries operate in such different ways that their relevance is limited. The SBIR/STTR programs are relatively unique in terms of scale, integrity, and mission focus.

The SBIR program’s diversity offers other challenges to evaluation and comparison. “The SBIR program” is in fact a multiplicity of agency-specific programs; it is important to ensure that research focuses on the SBIR at the NSF, not on a generic multiagency conceptualization of the SBIR program.

Focus on Legislative Objectives

It is important to note at the outset that this volume—and this study—does not seek to provide a comprehensive review of the value of the SBIR program, in particular measured against other possible alternative uses of federal funding. This is beyond our scope. Our work is focused on assessing the extent to which the SBIR program at NSF has met the congressional objectives set for the program, to determine in particular whether recent initiatives have improved program outcomes, and to provide recommendations for improving the program further.³⁴

Thus, as in the first-round study, the objective of this second round study is “not to consider if SBIR should exist or not”—Congress has already decided affirmatively on this question, most recently in the 2011 reauthorization of the program.³⁵ Rather, we are charged with “providing assessment-based findings of the benefits and costs of SBIR . . . to improve public understanding of the program, as well as recommendations to improve the program’s effectiveness.” As with the first-round, this study “will not seek to compare the value of one area with other areas; this task is the prerogative of the Congress and the Administration acting through the agencies. Instead, the study is concerned with the effective review of each area.”³⁶

Defining Commercialization

Commercialization offers practical and definitional challenges. As described in Chapter 4, several different definitions of commercialization are used in discussions of the SBIR program. We have concluded that it is important to use more than one simple definition: For example, it is not appropriate to use the number or percentage of projects that reach the marketplace as the sole metric for commercial success. In the private sector, commercial success over the long term requires profitability. But in the short term, commercialization can involve many different aspects of commercial activity, from product rollout to licensing to patenting to acquisition. Even during new product rollout, companies often do

___________________

³⁴These limited objectives are consistent with the methodology developed by the committee. See National Research Council, An Assessment of the Small Business Innovation Research Program: Project Methodology (Washington, DC: The National Academies Press, 2004).

³⁵National Defense Authorization Act of 2012 (NDAA) HR.1540, Title LI.

³⁶National Research Council, An Assessment of the Small Business Innovation Research Program: Project Methodology.

not generate immediate profits. In this report we use multiple metrics to address the question of commercialization.

Quantitative Assessment Methods

More practically, several issues relate to the application of quantitative assessment methods, including decisions about which kinds of program participants should be targeted, the number of responses that are appropriate, selection bias, nonresponse bias, the design and implementation of survey questionnaires, and the level of statistical evidence required for drawing conclusions in this case. These and other issues were discussed at a workshop and published in a 2004 report.³⁷ Also prepared was a peer-reviewed report on study methodology, which provided the baseline for the initial study and for follow-on studies—such as this one.

Survey Development

For the current study, a new survey of SBIR recipients was developed and deployed. This 2011 Survey³⁸ was based closely on previous surveys, particularly one deployed by the Academies in 2005 as part of its round-one assessment of the SBIR program, but nonetheless included some significant improvements. The description of the survey and improvements are documented in Appendix A of this report. The 2011 survey delves more deeply into the demographics of the program. It addresses in detail the role of agency liaisons. Finally, it provided unique opportunities to collect opinions and recommendations for program improvement from NSF award recipients. The survey generated more than 600 responses from recipients and provided an important pillar to the research conducted for this volume.³⁹Appendix A provides a description of the survey methods, including a discussion of the survey outreach and response.

Complementing the 2011 survey was a 2010 Phase IIB Survey of NSF Phase II awards that was carried out on behalf of the committee with the objective of comparing outcomes between awardees receiving standard Phase II awards and those also receiving Phase IIB enhancements.

Issues related to quantitative methodologies are discussed in detail in Chapter 4 and Appendix A. We recognize that the conclusions that can be drawn from this kind of assessment are limited. However, we also conclude that drawing on quantitative analysis is a crucial component of the overall study, given the need

___________________

³⁷National Research Council, Program Diversity and Assessment Challenges.

³⁸The survey carried out as part of this study was carried out in 2011, and the survey completed as part of the first-round assessment of the SBIR program was administered in 2005. In this volume all Academies survey references are to the 2011 survey unless noted otherwise.

³⁹National Research Council, An Assessment of the Small Business Innovation Research Program: Project Methodology.

to identify and assess outcomes that are to be found only at the level of individual projects and participating companies.

A Complement of Approaches

Partly because of these limitations, the 2004 methodology report⁴⁰ stressed the importance of utilizing a complement of approaches, which has been adopted here. Although quantitative assessment represents the bedrock of our research and provides insights and evidence that could not be generated through any other modality, it is in and of itself not sufficient to address the multiple questions posed in this analysis. Consequently, we undertook a series of additional activities:

• Case studies. We conducted in-depth case studies of 10 NSF SBIR recipients. These companies were geographically and demographically diverse and were at different stages of the company life cycle. Lessons from the case studies are described in Chapter 6, and the cases themselves are included as Appendix E.
• Workshops. We conducted workshops to allow stakeholders, agency staff, and academic experts to provide insights into program operations, as well as to identify questions that should be addressed.
• Analysis of agency data. As appropriate, we analyzed and included data provided by NSF about the various aspects of its SBIR activities.
• Open-ended responses from SBIR recipients. For the first time, we collected textual survey responses. More than 400 recipients provided narrative comments.
• Agency consultations. We engaged in discussions with NSF staff about program operations program and the challenges faced.
• Literature review. Since the start of our research in this area, a number of papers have been published addressing various aspects of the SBIR program. In addition, other organizations—such as the Government Accountability Office (GAO)—have reviewed specific parts of the SBIR program. We incorporated references to their work, where useful, into its analysis.

In short, within the limitations described, we utilized a complement of tools to ensure that the study reflects a full spectrum of perspectives and expertise. Appendix A provides an overview of the methodological approaches, data sources, and survey tools used in this study.

___________________

⁴⁰Ibid.

ORGANIZATION OF THE REPORT

Our analysis and conclusions are organized as follows. Chapter 2 provides background on the mission of the NSF, relates how the SBIR and STTR fit within the NSF’s Office of Industrial Innovation and Partnerships, reviews program operations, examining the emergence of what could be called the NSF model of program management, and addresses the congressional mandate to foster the participation of women and minorities and the current status. Chapter 3 reviews NSF data concerning applications and awards to the program, drawing out differences by demographic, geographic, and industrial sector. Chapter 4 analyzes program outcomes related to achieving NSF’s SBIR goals. Chapter 5 describes and analyzes in some detail the NSF Phase IIB program, an important initiative that provides matching funds for commercially viable projects at the end of Phase II. Chapter 6 draws on company case studies and on the textual responses from survey respondents to provide a qualitative picture of program operations, issues, and possible solutions. Chapter 7 provides our findings and recommendations.

End-of-chapter Annexes provide additional detail about the chapter topics, and six report appendixes provide additional information. Appendix A sets out an overview of the methodological approaches, data sources, and survey tools used in this assessment. Appendix B describes key changes to the SBIR program from the 2011 Reauthorization. Appendix C reproduces the 2011 Survey instrument. Appendix D shows the 2010 Phase IIB Survey instrument. Appendix E presents the case studies. Appendix F contains a list of references.