SBIR at the National Science Foundation (2015)

4

Commercial and Knowledge Outcomes

This chapter uses available data from NSF and two award recipient surveys of NSF SBIR winners to analyze outcomes related to achieving three of the four goals of the congressional mandate for the SBIR as was described in Chapter 1. Although the statutory goals of the SBIR program are fourfold, subsequent legislation passed by Congress, as well as administrative policies pursued by NSFs, has focused on the fourth goal, the commercialization of SBIR technologies. Commercialization is among the more measurable outcomes of the SBIR program, and has become the benchmark for program performance. The focus on commercialization, however, should not be allowed to obscure the fact that the program is designed to meet all four congressional mandated objectives. Therefore, this chapter also analyzes knowledge effects, which relate to both the first and second mandated goals, stimulating technological innovation, and helping to meet the NSF mission goals related to expanding scientific and technical knowledge. Outcomes related to the third goal, “to foster and encourage participation by socially and economically disadvantaged small businesses”¹ (defined to include woman-owned small businesses), are treated separately in Chapter 2.

DATA SOURCES AND COMPARISON ANALYSIS

To develop an effective quantitative analysis of the commercial and knowledge outputs of the NSF SBIR program, we have drawn on two main sources: data from the National Science Foundation (NSF), and responses to a large-scale survey of SBIR recipients at NSF. This 2011 survey is based on the 2005 survey

___________________

¹Small Business Administration, SBIR Policy Directive, October 18, 2012, p. 3.

deployed by the Academies,² with some additions and modifications.³Appendix A provides a detailed description of the survey methodology, including discussion of the response rate and discussions of potential survey bias. Table A-3 shows that the overall number of responses for NSF was 393, reflecting a response rate of 46.2 percent. The 2011 Survey questionnaire is reproduced in Appendix C.

COMMERCIALIZATION

Under previous and current management, the National Science Foundation (NSF) has made a concerted effort to focus on commercialization, the fourth mandated objective. Indicative of this focus, Dr. Kesh Narayanan, a former NSF SBIR program manager, often said that 97.5 percent of NSF funding went to basic science, and that he was in charge of the 2.5 percent of all NSF funding that was aimed at achieving commercial results. Congressional interest in this area is also high.⁴ Commercialization is important to the nation’s economic prosperity. Commercialization is also among the more quantifiable outcomes, and, therefore, has become a measuring stick for program outcomes.

Some important conceptual challenges must be addressed in tracking commercialization. Like many apparently simple concepts, tracking commercialization becomes progressively more challenging and complex as it is subjected to further scrutiny. A series of questions underscores these challenges:

• Should commercialization include just sales or other kinds of revenue such as licensing fees and funding for further development?
• What is the appropriate benchmark for sales?
  • The fact of any sales whatsoever?
  • Sales equivalent to the cost of awards?
  • Sales that reflect a profit?
  • Sales that drive overall company growth?
  • Very substantial commercial success—which would need to be bench-marked at a specific level of sales (e.g., $5 million, $10 million, or some other number)?
• Should commercialization include sales by licensees, which may be many multiples of the revenues provided to the SBIR company through licenses?

___________________

²Effective July 1, 2015, the institution is called the National Academies of Sciences, Engineering, and Medicine. References in this report to the National Research Council, or NRC, are used in a historic context identifying programs prior to July 1.

³See National Research Council, An Assessment of the SBIR Program, (Washington, DC: The National Academies Press, 2008), Appendix A.

⁴Dr. Narayanan was Division Director of Industrial Innovation and Partnerships from 1994 to 2011 and led the SBIR at NSF in that capacity.

This chapter draws from responses of Phase II SBIR recipients to a large-scale 2011 survey of SBIR recipients at NSF (2011 Survey), as well as previous survey of Phase II and Phase IIB recipients in 2010 (2010 Phase IIB Survey). The 2010 and 2011 surveys are themselves based on a 2005 survey deployed by the Academies, with some additions and modifications.⁵ This has permitted the Committee to cast a broad net to capture a wide range of potentially useful data. These data have been analyzed in a variety of ways to provide a range of insights into this complex topic.

Sales and Licensing Revenue

A much used metric for assessing SBIR-type programs is the sum of sales and licensing fees. In previous assessments, the Academies warned extensively against overuse of this metric.⁶ The Committee heeded these warnings by adopting a wide range of metrics for use in the current assessment; however, sales and licensing fees are still an important consideration.

The current survey eliminated some questions about licensee activities that have sometimes been collected in the past. Although these activities can be important, the principal investigators leading SBIR-funded projects generally have little insight into the activities of licensees. In addition, descriptions of licensee activities are often subject to nondisclosure provisions. Hence, given the limitations of the data provided, it did not seem appropriate to include questions about licensee activities in the updated survey.

Reaching the Market

The first question addressed by the survey concerns reaching the market: Did the project generate sales, and, if not, are sales expected? The second part of this question is necessary because some projects have long cycle times. Responses are summarized in Table 4-1.

Overall, almost 70 percent of Phase II respondents reported some sales from either the company or a licensee. This percentage is higher than that reported for other agencies (typically 45 percent to 60 percent). A further 19 percent reported that they expected to generate future sales from the surveyed project—a percentage that in part reflects the relatively recent date of some awards in the sample.⁷

___________________

⁵The 2011 Survey and 2010 Phase IIB Survey appear in appendixes C and D.

⁶See, for example, National Research Council, An Assessment of the Small Business Innovation Research Program, Section 4.2.2.

⁷2011 Survey, Question 35.

TABLE 4-1 Sales Revenue

^a 2010 Phase IIB Survey question was asked only with regard to projects with an active status.

NOTE: 2010 Phase IIB Survey did not break out sales by type.

SOURCE: 2011 Survey, Question 35; and 2010 Phase IIB Survey.

Amount of Sales Revenues

The percentage of projects reaching the market is an important metric, but it is not sufficient. It is also important to understand the dollar distribution of sales revenue. The survey asked respondents who reported some sales from the surveyed project to also report the amount of sales by tier (see Table 4-2).

TABLE 4-2 Distribution of Total Sales Revenue, by Range

SOURCE: 2011 Survey, Question 36, B1.

Slightly more than 90 percent of Phase II respondents reported sales below $5 million. Twenty-two percent reported sales below $100 thousand. Forty-two percent reported sales of more than $100 thousand but less than $1 million. Thirty-six percent reported sales of $1 million or more, and 9 percent were at $5 million or above. None reported sales of $50 million or greater in project related revenues.

Markets by Sector

As shown by Table 4-3, more than one-half of Phase II respondents reported sales to the domestic private sector in the United States. Fifteen percent reported export sales. In addition, overall sales by Phase II respondents to government agencies and prime contractors for DoD or NASA in combination reached 16 percent.

Employment

As with prior surveys, respondents were asked about the size of the company at the time of the award and the size at the time of the survey, in terms of number of employees. As shown by Table 4-4, Phase II funding was more likely to be awarded to firms with 5 or more employees than to those with fewer than 5 employees.

TABLE 4-3 Markets for SBIR Products and Services—Percentage of Total Sales (Mean of All Responses/Category)

SOURCE: 2011 Survey, Question 37, and 2010 Phase IIB Survey.

TABLE 4-4 Company Size by Number of Employees at Time of Award

SOURCE: 2011 Survey, Question 18A.

The mean number of employees was 16 for Phase II respondents, and the median size was 6. The difference between mean and median likely indicates an influence on the mean by outliers, that is, a few companies with high numbers of employees. A striking characteristic of the NSF program is that it makes awards to very small companies—at other agencies the median has been considerably higher.⁸

Respondents were also asked to report the current number of employees. Results are affected by selection bias toward surviving companies. As Table 4-5 shows, the median size grew for Phase II companies from 6 at the time of the award to 10 at the time of the survey.

The percentage of Phase II respondents reporting 50 or more employees about doubled—from 9 percent at the time of award to nearly 17 percent at the time of the survey, which indicates that some NSF SBIR recipients have become relatively large enterprises (within the general universe of small companies).

Further Investment

The ability of SBIR projects and companies to attract further investment has traditionally been an important defining metric for SBIR commercialization outcomes.⁹ According to the survey results shown in Table 4-6, 63 percent of

___________________

⁸For example, the median number of employees at the Department of Defense is 17. See National Research Council, SBIR at the Department of Defense (Washington DC: National Academies Press, 2014), p.62.

⁹See National Research Council, An Assessment of the SBIR Program.

TABLE 4-5 Company Size (Employees) at Time of Survey

SOURCE: 2011 Survey, Question 18B.

TABLE 4-6 Additional Investment after SBIR Award

SOURCE: 2011 Survey, Question 33.

Phase II projects received additional funding (relatively unchanged from the 2009 survey¹⁰). This indicates that SBIR funded projects are seen to have sufficient value to persuade non-SBIR sources to invest in them.

As with prior surveys, the distribution of additional non-SBIR funding is substantially skewed (see Table 4-7). While 61 percent of Phase II companies reported no funding from non-SBIR sources, 39 percent did. Most reporting funding were in the category of $100 to $500 thousand. Two percent of Phase II respondents reported additional non-SBIR investments of at least $20 million.

___________________

¹⁰See National Research Council, An Assessment of the SBIR Program at the National Science Foundation (Washington, DC: The National Academies Press, 2008), Appendix B.

TABLE 4-7 Additional Non-SBIR Federal Funding by Amount

SOURCE: 2011 Survey, Question 34.1.

As shown by Table 4-8, self-funding by one’s own company was the prevalent source of additional funding reported by Phase II respondents, while many also reported using personal funds. Direct funding from federal agencies was the second most-cited outside source. Funding by other companies, private equity/ angels, and state and local governments were each cited by between 25 and 32 percent of Phase II respondents. Fifteen percent of Phase II companies reported receiving venture funding.

Company-Level Development

Table 4-9 shows company-level changes resulting from the SBIR program. SBIR firms that commercialize their technology often do so through mergers or other company-level activities. Fifteen percent of Phase II respondents indicated that their company had spun off one or more new companies, and 7 percent reported that they had been acquired by or merged with another firm. However, the answers of the majority of respondents indicated that their companies had, as of the time of the survey, not been acquired, had not implemented or planned an initial public offering (IPO), and had not established a spin-off.

Respondents also reported on a range of market-related activities that involved agreements between the surveyed company and other organizations, which can be considered an indication of commercial activity. Table 4-10 shows

TABLE 4-8 Sources of Additional Funding, for Projects Reporting Additional Funding

^a Actual number depends on type of funding.

NOTE: Additional federal funding was non-SBIR.

SOURCE: 2011 Survey, Question 34.

TABLE 4-9 Company-Level Changes Resulting from SBIR Program

NOTE: Responses may not sum to 100 percent because respondents could select more than one answer.

SOURCE: 2011 Survey, Question 10.

TABLE 4-10 Market-Oriented Activities—Finalized Agreements with U.S. Companies and Investors

NOTE: Responses may not sum to 100 percent because respondents could check more than one response.

SOURCE: 2011 Survey, Question 38.1.

finalized agreements between Phase II respondents and U.S. companies and investors by type. Half of Phase II respondents reported completion of at least one R&D agreement with U.S.-based companies or investors. Thirty-nine percent of Phase II respondents reported licensing agreements, and 60 percent reported either marketing/distribution agreements or customer alliances. These four—R&D agreements, licensing agreements, marketing/distribution agreements, and customer alliances—were the most prevalent types of agreements reported by Phase II recipients with U.S. companies and investors.

Table 4-11 shows the percentage of Phase II respondents that entered into various types of agreements with foreign partners. As with domestic partners, the most prevalent types of agreement with foreign partners were R&D agreements reported by 42 percent, licensing agreements reported by 35 percent, marketing/ distribution agreements reported by 37.2, and customer alliances reported by 26 percent.

A comparison of forms of agreements of Phase II recipients with U.S. partners and with foreign partners shows the following: R&D agreements, licensing agreements, and customer alliances were reported more often with U.S. partners, while marketing/distribution agreements, joint venture agreements, and sale of company were reported more often with foreign partners.

TABLE 4-11 Finalized Agreements with Foreign Partners

NOTE: Responses may not sum to 100 percent because respondents could check more than one response.

SOURCE: 2011 Survey, Question 39.1.

Commercialization Training and Marketing

Federal agencies have in recent years increased the amount of commercialization training for SBIR awardees. In some cases, this training has been mandatory. The Committee added a question focused on such training to the 2011 Survey, and the results are reported in Table 4-12.

NSF has focused its commercialization training on Phase II awardees. The responses summarized in Table 4-12 indicate that 61 percent of Phase II respondents had participated in training related to the surveyed award.

The Committee also added a question to the 2011 Survey about the existence of full-time marketing staff at the company, as another metric to gauge the extent to which the company has focused on commercialization. The results are shown in Table 4-13. Forty-seven percent of Phase II respondents reported having one or more full-time marketing staffer. These data suggest that, even with Phase II funding, the very small size of the companies makes it difficult for the majority to support a full-time marketing staffer.

Conclusions: Commercialization Data

Evidence from the 2011 Survey provides useful insight into the commercialization record of SBIR companies at NSF on a number of dimensions. Overall,

TABLE 4-12 Participation in Commercialization Training

SOURCE: 2011 Survey, Question 17.

TABLE 4-13 Full-Time Marketing Staff

NOTE: Statistical tests were run on responses weighted by company.

SOURCE: 2011 Survey, Q12.

the evidence strongly suggests that NSF is meeting its mandate to encourage the commercialization of federally funded research.

Phase II survey respondents reported that almost 70 percent of Phase II projects had already recorded sales of products or services derived from the surveyed award. A further 18 percent expected future sales. Given the relatively short time between the award date for some of the awards and the survey date, these expectations are not unreasonable.

Sales revenue of Phase II projects which had sales were highly skewed. At the upper end of the distribution, 9 percent reported sales revenue of $5 million or more, and at the lower end of the distribution, 22 percent reported sales revenue under $100 thousand. The remaining nearly 70 percent fell in between, with 42 percent reporting more than $100 thousand, but less than $1 million, and 27 percent reporting $1 million or more but less than $5 million.

Additional non-SBIR investment is another important metric of commercialization. Sixty-two percent of Phase II respondents reported that they had acquired additional funding for the project, which is evidence of the value perceived by investors even before market outcomes are achieved.

The range of market-related activities with other organizations—domestic and foreign—is yet another indication of commercial activity of Phase II SBIR award recipients. The most prevalent types of agreements reported were R&D agreements, licensing agreements, marketing and distribution agreements, and customer alliances. Multiple metrics support the idea that NSF Phase II awardees are making progress toward commercialization.

KNOWLEDGE EFFECTS

Among the four congressionally mandated objectives for the SBIR program is to “stimulate technological innovation,” an objective for which knowledge effects serve as metrics. Another mandated objective is to help meet the federal agency mission, which in the case of NSF entails expanding and disseminating scientific and technical knowledge. Again, knowledge effects are pertinent metrics for assessing progress.

To measure progress toward stimulating technological innovation, patenting and patent citation analysis are potent metrics. However, in the context of small business, this standard metric of innovation does not capture the entire story. Many companies interviewed for this report indicated a preference to keep their technology secret or to rely on first-mover advantages and other market-based leverage to defend their technologies. Yet, standard metrics do provide a starting point for quantitative analysis. Consequently, the survey included several metrics related to intellectual property (IP) (i.e., patents, trademarks, copyrights, and peer-reviewed papers).

Patents

Patents are to some degree the lifeblood of high-tech firms. Because patents at small companies often result from multiple contracts in multiple projects, it is important to capture patents related to the surveyed SBIR project as well as patents more generally attributable to SBIR-funded research.

Table 4-14 shows the SBIR-related metrics provided in this study. More than 70 percent of Phase II respondents reported at least 1 patent related to an SBIR-funded technology, and 12 percent of Phase II respondents reported at least 10 such patents. The mean number of patents reported by Phase II respondents is 5.

Regarding patents awarded for the specific SBIR-funded project surveyed, Table 4-15 shows that 70 percent had at least one or more patents, while more than 30 percent of Phase II respondents reported no patents.

Respondents expressed limited interest in either trademarks or copyrights, the two other primary forms of legal protection for intellectual property. Details on these questions are provided in Appendix C.

TABLE 4-14 Number of Patents (at Least in Part) Related to All Company SBIR Awards

SOURCE: 2011 Survey, Question 11.

TABLE 4-15 Patents Related to Surveyed Project

SOURCE: 2011 Survey, Question 39.1.2.

Peer-Reviewed Publications

A broader measure of knowledge effects is provided by taking into account peer-reviewed publications. Other potential measures not included here include tacit knowledge advances such as human capital gains of researchers and others, as well as enhancements of knowledge networks.

TABLE 4-16 Peer-Reviewed Scientific Publications Related to the Surveyed Project

SOURCE: 2011 Survey, Question 39.4.1.

Data from the survey suggest that companies and their staff publish widely, despite the fact that technical knowledge is often the prime source of competitive advantage of a small business. Survey data on publishing is given in Table 4-16. It shows that more than 80 percent of Phase II respondents indicated that an individual at the surveyed company submitted at least one scientific paper related to the surveyed award. Thirty-two percent reported more than three scientific papers submitted.

Links to Universities

An additional metric for knowledge transfer is the development of linkages with universities, as universities are the acknowledged source of many technological innovations in the US. Previous Academies studies have also utilized this metric for program evaluation. Although the STTR program focuses explicitly on these linkages, data from the survey indicate that SBIR projects also often develop close university ties.

The survey asked several questions about the use of university staff and facilities on the surveyed Phase II projects, and reported the results in Table 4-17. Fifty-eight percent of Phase II respondents reported a university connection of some kind, which represents an increase from the 47 percent of Phase II respondents reported in the 2007 survey.¹¹ The most reported types of linkage were a

___________________

¹¹National Research Council, Assessment of the SBIR Program at the National Science Foundation, p. 265.

TABLE 4-17 Linkages to Universities

SOURCE: 2011 Survey, Question 59.

faculty member working on the project but not as a principal investigator (PI), graduate students employed on the project, and a university or college as a subcontractor on the project. Also notable among the reported types of linkages was the origination of the project technology from a university or college, either by development by one of the project participants or by having been licensed from a university or college.

Respondents identified 114 different universities and colleges with which they worked in various capacities on the surveyed project. Those universities or colleges mentioned by three or more respondents are listed in Table 4-18.

Many of the institutions are large state universities, some of which have in recent years focused on technology transition as well as basic research. At the top of the tabular listing for having been specifically mentioned as playing a role in at least five surveyed NSF SBIR projects are the Universities of Colorado, Florida, Texas, Georgia, and California-San Diego, plus MIT and Columbia. When all mentions are taken into account (including those universities and branches that received fewer than 3 and therefore are not included in Table 4-18), the University of California system had 8 mentions and the University of Texas had 7. Although far from a perfect metric, the data shown in the table provide a preliminary indication of the connections between specific universities, university systems, and the NSF SBIR program.

Also indicative of a strong linkage between academia and the SBIR was the reporting that about 80 percent of companies had at least one founder with an

TABLE 4-18 University Participants Mentioned by Three or More Respondents

SOURCE: 2011 Survey, Question 60.

academic background, and for slightly less than 30 percent of companies, at least one founder was most recently employed at a college or university.

Conclusions: Knowledge Effects

The survey data reveal that SBIR-funded projects generate a substantial amount of technical knowledge that enters the public domain, thus meeting the congressional objectives for the program to stimulate technological innovation,

and to help meet NSF’s agency mission to advance the nation’s scientific and technical knowledge. What emerges from the survey data is a picture of companies that are embedded in the innovation economy and that use the patent system to protect their intellectual property. More than 70 percent of Phase II companies received at least one patent based on their work under SBIR contracts, while 70 percent received at least one patent related to the surveyed project alone.

That SBIR companies help expand the nation’s knowledge base is also indicated by their publishing in peer-reviewed journals. More than four-fifths of the NSF Phase II companies published at least one article based on the SBIR-funded work, and about one-quarter published at least three such papers. These data do not include less formal and perhaps more ubiquitous knowledge transmission, such as conference papers and presentations, and tacit forms of knowledge building, such as embodying human capital in researchers and entrepreneurs.

Finally, many SBIR companies are closely connected to universities. Fifty-eight percent of NSF Phase II respondents reported a university connection on the surveyed project, and seven universities were specifically mentioned as playing a role in at least five surveyed NSF SBIR projects.

ASSESSING THE IMPACT OF NSF SBIR AWARDS USING COUNTERFACTUALS

It is difficult to determine the impact of a given SBIR award or program. Not only do many factors affect the success and failure of companies and projects, but also determining whether a specific factor was a necessary condition for success is challenging. Furthermore, the large number of factors and the multiple paths to success and failure lessen the ability to state with confidence that a particular intervention—in this case an SBIR award—constitutes a sufficient condition for a project’s or company’s success.

An alternative, generally practicable approach to assessing impact in applied program evaluation when experimental or quasi-experimental design is infeasible is to use a “counterfactual approach.” This entails asking recipients for their views of the program’s impact on their project or company by asking, “What would have happened to the project absent the SBIR award? Would it have proceeded anyway and, if so, in what form?” This section addresses responses to the counterfactual questions.

Project Go-Ahead Absent SBIR Funding

The survey asked Phase II recipients whether the surveyed project would have been undertaken absent SBIR funding and, if so, whether the scope and timing would have been affected. Table 4-19 summarizes the responses. Fourteen percent of the respondents believed that the project likely would have proceeded without SBIR funding, but only 2 percent were certain. In contrast, more than

TABLE 4-19 Project Undertaken in the Absence of This SBIR Award

SOURCE: 2011 Survey, Question 24.

two-thirds believed the project likely would not have proceeded absent SBIR funding: 26 percent were definite and a further 42 percent thought it unlikely. Data published in the Academies’ 2009 report were very similar.¹² In fact, when project funding cannot be secured in a timely way, the assembled team may dissipate to find other sources of income, leaving the applicant unable to continue.

These data have interesting wider implications for debates about early-stage funding: they suggest a weakness in the “crowding out” hypothesis, because it appears that awardees—presumably those with the closest knowledge of funding prospects for the project—overwhelmingly believed it unlikely that funding alternative to SBIR could be found.¹³

Project Scope Absent SBIR Funding

A second area of review concerns the impact of funding on the scope of the project. It seems likely that SBIR funding would lead to an expansion of the project scope. However, the selection of SBIR awards is based on criteria that

___________________

¹²National Research Council, An Assessment of the SBIR Program of at the National Science Foundation, Appendix B.

¹³“Crowding out effect” refers to the hypothesis that increasing public sector investment in an area will replace (drive down) private sector spending. See review of crowding out literature in Dirk Czarnitzki and Andreas Fier, “Do Innovation Subsidies Crowd Out Private Investment? Evidence from the German Service Sector,” ZEW Discussion Papers, No. 02-04, 2002.

can be drawn quite narrowly. It is therefore possible that tailoring a project to the requirements of a particular solicitation could reduce the scope of the project.

The Committee’s analysis focused only on those respondents who were certain that the project would have proceeded without SBIR funding. Table 4-20 shows that, although 75 percent of respondents indicated that the absence of SBIR funding would have limited the project’s scope, 8 percent indicated that the project’s scope had been limited by participation in the program, most likely to address the specific, narrowly drawn requirements of SBIR topics.

Project Delays Absent SBIR Funding

As with project scope, the immediate supposition is that, absent SBIR funding, projects would have been delayed while other funding was identified and acquired. However, as discussed in Chapter 2, SBIR awards carry delays of their own, which can be substantial in some cases. Thus, this survey question seeks to determine a balance between delays imposed by the need for new funding and delays inherent in the SBIR program.

As shown in Table 4-21, a large majority of respondents who were certain that the project would have proceeded absent SBIR funding agreed that the absence of SBIR funding would have delayed the project. Sixty-five percent of NSF Phase II respondents reported that the project would have been delayed by at least 12 months, 26 percent by at least two years, and 9 percent by at least 3 years. Given that gaps and delays can significantly impact the ability of small companies with limited resources to retain their technical teams, this is a potentially important result. However, the small sample size indicates that these results, although important, should be treated with caution.

TABLE 4-20 SBIR Impact on Scope of Project

NOTE: Only includes respondents who were certain that project would proceed absent SBIR funding.

SOURCE: 2011 Survey, Question 25.

TABLE 4-21 SBIR Impact on Project Delay

SOURCE: 2011 Survey, Question 26B.

SBIR Funding and Project Duration

The survey also asked respondents to determine how the absence of SBIR funding would have affected the duration of the project. Table 4-22 shows the results. Eighty-three percent reported that the project would have taken longer, while 4 percent thought it would have taken less time without the SBIR award. This too is a potentially important finding in that project delays can delay potential market entry, which can be critical because the window for market entry can be narrow. Again, the small sample size indicates that these results, although important, should be treated with caution.

Long-Term Impacts on the Recipient Company

Although SBIR awards have direct effects on specific projects, they can also have a longer term effect on the trajectory of company development, creating capacity and in some cases providing a needed input that transforms long-term outcomes.

The survey asked respondents about the SBIR program’s impact on the company’s long-term development. Table 4-23 reveals overwhelmingly positive long-term impact on Phase II companies. Overall, 35 percent of Phase II respondents indicated that the SBIR program had a transformative effect on their company, and an additional 54 percent reported a strongly positive effect. Negative effects were negligible.

Key Aspects of SBIR-Driven Transformation

It is not easy to summarize the numerous ways in which NSF SBIR awards helped to transform recipient companies. Therefore, what follows is a limited list of impacts drawn from verbatim comments from respondents who

TABLE 4-22 SBIR Impact on Project Duration

SOURCE: 2011 Survey, Question 26B.

TABLE 4-23 Long-Term Impacts on Recipient Companies

SOURCE: 2011 Survey, Question 45.

reported that SBIR awards had transformed their companies. Respondents indicated that SBIR

• provided first dollars,
• funded areas that did not interest venture capitalists and other funders,
• opened doors to many potential stakeholders in specific technologies, including agencies, prime contractors, investors, suppliers, subcontractors, and universities,
• assisted entry into niche markets too small for major players/funders,
• funded technology development,
• enabled projects with high levels of technical risk,
• supported adaptation of technologies to new uses, markets, and industry sectors,

• diversified expertise and allowed for hiring of specialists,
• provided critical funding during downturns in the business cycle,
• attracted and developed young researchers,
• redirected company activities to new opportunities,
• reduced costs,
• addressed needs that require high tech at low volume and relatively low cost,
• provided companies with credibility because SBIR research is considered to be peer reviewed,
• funded researchers to enter business full time,
• transformed company culture to become more market driven,
• drove researchers to focus on technology transition,
• created new companies and kept companies in business (that would not exist without SBIR funding),
• supported feasibility testing for high-risk/high-payoff projects,
• supported projects with longer time horizons/long sales cycles,
• provided the basis for spin-off companies,
• funded proof of concept,
• encouraged R&D companies to transition into manufacturing,
• funded disruptive technologies,
• provided significant mentoring especially for new businesses,
• stimulated international collaboration, and
• reduced technological risk.

Overall, the strongest conclusion to be drawn from these responses is that small innovative companies are highly sensitive to the impact of exogenous variables. The sudden withdrawal of a sponsor can crush a company; a single contract can provide funding for 2 or 3 years of growth. And above all, these small companies are highly path dependent: what happens to them at a given moment can dramatically affect long-term outcomes. The butterfly effect could have been invented specifically to apply to these kinds of companies.

In the end, the SBIR program has often been for many companies a profoundly positive exogenous variable: one that provides funding, validation, and often market access not otherwise available. Even though it seems tenuous to link one SBIR award to the eventual success of a large company, in fact SBIR awards have been pivotal factors in the growth of what are now large and successful companies. The evidence from survey respondents suggests that this powerful positive jolt is not an uncommon effect of SBIR awards.

Key aspects of SBIR-driven transformation are explored in more detail in Chapter 6, which draws extensively on the numerous open-ended comments received in response to a question about the long-term impact of SBIR awards on recipient companies, as well as on the case studies.

COMPANY AND PROJECT CHARACTERISTICS: INDEPENDENT VARIABLES

The outcomes section above strongly suggests that, overall, Phase II SBIR funding at NSF correlates with commercial success. However, a variety of other factors may play a role, and the survey sought to address these.

Founders and Company Foundation

Venture investors focus heavily on the composition of the company team when deciding whether to make an investment. Accordingly, it seemed appropriate to explore the characteristics of company founders.

Table 4-24 shows most Phase II companies reported that they had multiple founders. Only 30 percent reported a single founder; another 30 percent reported two cofounders; the remaining 41 percent (rounding error) reported three or more cofounders.

An analysis of the number of previous companies (of any kind—not just SBIR recipients) founded by given company founders makes it possible to estimate the use of SBIR awards by serial company founders. As may be determined by Table 4-25, more than half of recipient companies had at least one founder who had previously founded another company, while 46 percent had not. More than a quarter of recipient companies had founders who had previously founded multiple companies.

TABLE 4-24 Number of Founders per Responding Company

NOTE: The survey received responses from multiple respondents per company. To address company-level questions, these responses were weighted equally so that taken collectively responses from each company had equal impact on statistical analysis. This approach was used for tables based on Questions 3-12 inclusive in the survey questionnaire (see Appendix C).

SOURCE: 2011 Survey, Question 4A.

TABLE 4-25 Number of Previous Companies Started by Founders

SOURCE: 2011 Survey, Question 4B.

It is worthwhile to learn whether founders had a business background or an academic background before founding their current company. Together the following series of tables provides a good overview of founder background: Table 4-26 shows the share of companies that have founders with a business background; Table 4-27 shows the share with an academic background; and Table 4-28 shows the share with founders most recently employed in business, academia, government, or other areas.

Table 4-26 shows that 64 percent of Phase II companies had at least one founder with a business background, and 23 percent had more than one founder with a business background.

Table 4-27 shows that about 80 percent of Phase II respondents reported at least one founder with an academic background. More than 40 percent of Phase II respondents reported at least two founders with academic backgrounds.

Table 4-28 shows that 73 percent of Phase II respondents reported at least one company founder most recently employed at another private company, while 28 percent of Phase II reported at least one founder previously employed at a college or university. Only 9 percent of Phase II respondents reported most recent prior employment in government and 5 percent in other areas.

Previous Academies studies concluded that, for at least some companies, SBIR funding provided opportunities that led directly to company formation. Table 4-29 shows that 45 percent of Phase II respondents indicated that the surveyed award contributed to some degree to company formation. More than

TABLE 4-26 Number of Founders with Business Backgrounds

SOURCE: 2011 Survey, Question 4C.

TABLE 4-27 Number of Founders with Academic Backgrounds

SOURCE: 2011 Survey, Question 4D.

TABLE 4-28 Most Recent Employment of Founders

NOTE: Totals do not sum to 100 percent because respondents were permitted to check multiple responses.

SOURCE: 2011 Survey, Question 5.

TABLE 4-29 Company Founded Because of SBIR Program

SOURCE: 2011 Survey, Question 6.

20 percent of Phase II respondents indicated that the company had been founded “because of SBIR.”

These data on founders and company foundation indicate that SBIR company founders tended to have strong connections to both business and academia. Companies with founders having a business background were prevalent among recipients of Phase II funding, as were those companies having founders with an academic background, and those companies having more than one founder. It may be that more experience in the private sector and a better understanding of how to address multiple needs within company operations are associated with Phase II success. The observation overall is consistent with the dual emphasis of the NSF SBIR program on both technical strength and commercial potential. Consistent with previous survey findings, SBIR funding had a strong positive effect for many of the respondents on the decision to form a company.

Industry Sector

Previous analysis of the SBIR program did not address a potentially important intervening variable: industry sector. To the extent that commercialization outcomes are affected by the average cycle time of product development in different sectors, the industry sectors of SBIR awards are relevant to impact assessment. For example, product cycle time is much shorter in software than in materials or medical devices.

Table 4-30 shows the distribution of responses of Phase II respondents to a question designed to provide an approximate map of activities by industry sector. There is considerable overlap between some categories, and respondents had substantial leeway to define sectors differently, so these results should be viewed as highly preliminary.

Two key points emerge from the responses. First, engineering and materials were the dominant sectors (at 43 percent and 47 percent of all responses, respectively), followed by information technology (IT) at 21 percent. To a considerable degree, the responses reflect the decisions made at NSF to put forward selected topics in specific areas (as explained in Chapter 2), rather than the endogenous interests of companies themselves.

Project Status and Review of Discontinued Projects

Because the 2011 survey covers 10 years of Phase II awards, projects were at different stages of completion at the time of the survey. Therefore, as noted in previous Academies analyses, project outcomes were in aggregate substantially underreported because of the number of projects whose entire life cycle was not yet complete. Table 4-31 shows the status of the projects at the time of the survey.

Phase II respondents reported products or processes in use at a rate of 47 percent. Fifteen percent of respondents reported that the project is continuing post-award technology development, while 22 percent reported that the project had been discontinued by the company recipient of the award.¹⁴

As shown in Table 4-32, the survey sought reasons for why projects had been discontinued. Phase II companies were most likely to discontinue their SBIR projects for multiple reasons, among which facing a market that was too small dominated at 35 percent. This problem was followed by not enough funding (23 percent) and technical failure or difficulties (also 23 percent). In fact, the Phase II reasons largely focused on a mismatch between the company’s product or service and the marketplace. Thus, it may be that helping companies to address this mismatch or better screening out projects without market potential at the selection stage could be a focus of further initiatives to improve commercialization rates.

___________________

¹⁴Sometimes SBIR-funded research is picked up later at another company. This survey focused on the original recipient.

TABLE 4-30 Respondent Identification of Phase II Awards by Industry Sector

NOTE: Answers do not sum to 100 percent because respondents could select more than one response. Answers within categories do not sum to category totals for the same reason.

SOURCE: 2011 Survey, Question 20.

TABLE 4-31 Current Status of Surveyed Projects

SOURCE: 2011 Survey, Question 30.

TABLE 4-32 Reasons for Project Discontinuation

SOURCE: 2011 Survey, Question 31.

TABLE 4-33 Primary Reason for Project Discontinuation

^aA Phase II project may fail because it did not receive a second related Phase II at a later date.

SOURCE: 2011 Survey, Question 32.

These findings are further supported by responses to an additional question, which asked respondents to indicate which factor was the primary reason for discontinuation. As shown in Table 4-33, a lack of market demand at 25 percent was the single largest reason given for Phase II project discontinuation, followed by not enough funding at 18 percent.

Based on the data provided in these tables, it seems reasonable to conclude that SBIR funding makes a substantive difference to the plans of companies, and that the absence of alternative funding is often a reason for project discontinuation. This finding in turn suggests that the crowding out hypothesis is at best of limited help in explaining company activity in this area.

Company Size by Revenue

The SBIR program is aimed at supporting small firms. As employment data show, most awardee companies are much smaller than the SBA maximum size for small companies (which is typically 500 employees in most sectors).¹⁵

___________________

¹⁵See Small Business Administration, Table of Small Business Size Standards Matched to North American Industry Classification System Codes, accessed October 20, 2013, <http://www.naics.com/ naicsfiles/Size_Standards_Table.pdf>.

Table 4-34 shows company size in terms of company revenue as of the most recent fiscal year prior to the survey. Just slightly under half of Phase II respondents reported company revenue below $1 million for the most recent fiscal year, while 13 percent reported company revenue below $100 thousand. More than a third of Phase II companies reported revenue of at least $1 million but less than $5 million, and 15 percent reported revenue of $5 million or more. The median company size as indicated by amount of revenue was $1.1 million for Phase II respondents.

Company Activities and SBIR

An often raised issue about the SBIR program relates to the existence of SBIR “mills,” that is, companies that live off a stream of SBIR contracts with little (if any) commercialization of the product or service. Previous Academies reports addressed this issue; it is addressed in Chapters 2 and 3 of this report; and it is addressed in company interviews (see Chapter 6). Many interviewees explained that the SBIR program’s role in a company changes over time and that companies generally become less dependent on SBIR funding as products reach the market.

TABLE 4-34 Total Company Revenues, Most Recent Fiscal Year

NOTE: Statistical tests were run on responses weighted by company.

SOURCE: 2011 Survey, Question 8.

SBIR Share of R&D Effort

The survey asked respondents to estimate how much of their company’s total R&D effort—defined as man-hours of work for scientists and engineers—was devoted to SBIR-funded projects.

As Table 4-35 shows, 45 percent of Phase II respondents indicated that SBIR currently constitutes 10 percent or less of overall company R&D man-hour effort, while 11 percent indicated that it amounted to more than 75 percent. For 76 percent of Phase II respondents, SBIR-funded projects accounted for no more than half the company’s R&D effort in the most recent fiscal year.

These responses correspond fairly closely to responses from another survey question about the percentage of company revenues derived from SBIR awards. As shown in Table 4-36, while 34 percent of Phase II respondents reported that their companies received none of their revenue from SBIR sources during the most recent fiscal year, the majority reported that part of their revenue came from SBIR awards. The part of company revenue composed of SBIR funds varied substantially. Fourteen percent of Phase II companies received up to 10 percent of their company revenue in the most recent fiscal year from SBIR awards; 30 percent received up to 25 percent; and 46 percent received as much as 50 percent. Nineteen percent of companies received more than 50 percent of their revenues from SBIR.

TABLE 4-35 Science and Engineering Man-Hours: Percentage Devoted to SBIR Projects, Most Recent Fiscal Year

NOTE: Statistical tests were run on responses weighted by company.

SOURCE: 2011 Survey, Question 7.

TABLE 4-36 Percentage of Company Revenue from SBIR, Most Recent Fiscal Year

SOURCE: 2011 Survey, Question 9.

Prior Use of the SBIR Program

Although standard models of innovation depict a linear sequence whereby ideas are tested in Phase I, prototyped in Phase II, and commercialized in Phase III,¹⁶ there is considerable evidence to suggest that this approach oversimplifies the process. Often, multiple iterations are required, projects must restart with an earlier phase, or multiple efforts are needed to meet specific problems.

The survey asked respondents to indicate how many prior SBIR or STTR Phase I awards had been received by the company that were related to the surveyed award. As Table 4-37 shows, In general, there was little difference between Phase I and Phase II responses. Overall more than 80 percent of companies received at least one prior Phase I award related to the surveyed award. Phase II respondents were more likely to report at least two or more, as well as five or more prior related Phase I awards.

In addition, Phase II respondents were likely to have received at least one prior related Phase II award (see Table 4-38)—almost 85 percent did so, and a third received two or more. Conversely, Phase I respondents were more likely to have not received any related Phase II awards, which was expected because these respondents were explicitly selected to complete the survey because they had not received a Phase II award from the study agencies during the study period.

___________________

¹⁶For a review of the model and its intellectual history, see Benoît Godin, “The Linear Model of Innovation: The Historical Construction of an Analytical Framework,” Project on the History and Sociology of S&T Statistics, Working Paper No. 30, 2005.

TABLE 4-37 Prior SBIR or STTR Phase I Awards Related to the Surveyed Project

SOURCE: 2011 Survey, Question 40.1.1.

TABLE 4-38 SBIR or STTR Phase II Awards Related to the Surveyed Project Technology

SOURCE: 2011 Survey, Question 40.1.2.

These data suggest that projects require more than one Phase II award before they commercialize, which in turn supports the view that innovative products emerge from clusters of activity rather than from a simple linear development from Phase I to Phase II to commercialization.

Prior Investment

One question surrounding the SBIR program related to its role in the sequence of funding that leads from an idea to a product. Table 4-39 probes the sources of funding for the given technology area prior to company receipt of the Phase II SBIR award. About two-thirds of all Phase II respondents reported at least one source of additional funding for the technology areas prior to the Phase II SBIR award. The most prevalent source of funding prior to the Phase II SBIR award was internal company investment (including borrowed money), reported by half the respondents. Previous SBIR/STTR funding (excluding preceding Phase I) was another prevalent source, reported by 45 percent of Phase II respondents. Private investors (including angel funding) as a source of funding was reported by 21 percent of Phase II respondents. Other private companies, prior non-SBIR/

TABLE 4-39 Sources of Funding, Prior to SBIR Phase II Award, for Related Technology

^a Question only asked for Phase II awards.

NOTE: Totals do not sum to 100 percent because respondents could select more than one funding source.

SOURCE: 2011 Survey, Question 21.

STTR federal R&D, and state or local government followed in order of importance as sources of funding, together reported by 40 percent of respondents as funding sources for their companies. Venture capital was reported as a source of funding by only 7 percent of Phase II respondents.

SUMMARY: QUANTITATIVE IMPACT ANALYSIS

The foregoing analysis can be summarized briefly as follows:

1. SBIR-funded projects at NSF tend to reach the market in large numbers—about 70 percent of surveyed Phase II projects reported sales of related products or services, and almost two-thirds of the remainder expected sales in the future.
2. The scale of commercialization is not large for most projects. Of those Phase II projects experiencing sales from the funded technology, only about 9 percent reported sales of $5 million or more. A few projects produce most of the commercial impact.
3. The SBIR program should not be considered an important program for creating jobs. NSF recipient companies are very small, even within the universe of small companies, and although those that survive do add jobs over time, they do not do so at a rate that would make job creation a primary impact of the program.
4. SBIR funding makes a significant difference in the founding of small innovative companies and the decision to proceed with a specific project: 45 percent of respondents reported that the SBIR program played a role in company foundation, and almost 70 percent of Phase II respondents believed that the surveyed project probably or certainly would not have proceeded without Phase II funding.
5. Knowledge effects in the case of NSF serve two mandated congressional goals for SBIR: (1) to stimulate technological innovation, and (2) to help meet federal agency mission, which in the case of NSF relates to advancing scientific and technical knowledge. Patenting and publications are important metrics for measuring contributions to these goals. About 70 percent of Phase II respondents reported receipt of at least one project-related patent. Similarly, over 80 percent of Phase II respondents reported publishing at least one peer-reviewed technical paper related to the surveyed award; about 40 percent had published at least three such papers.
6. NSF SBIR projects have substantial links with universities: about 57 percent of Phase II projects reported some university involvement, 114 different universities and colleges were identified as being involved in the surveyed project, 24 universities were specifically mentioned by at least three respondents, and 7 universities were specifically mentioned by at least five respondents.

7. Most respondents indicated that the project likely would not have proceeded without SBIR funding. And even among those who believed the project would have proceeded without SBIR funding, most expected that it would have been delayed and/or reduced in scope. Overall, respondents reported that SBIR had profoundly positive long-term impacts on their company. More than one-third of Phase II respondents reported that SBIR had a “transformative effect,” and 88 percent reported a substantial positive effect or a transformative effect.
8. Given that the SBIR program supports companies and ideas at earlier stages of development than venture capital investment, the case can be made that it would be appropriate to follow SBIR companies that are showing success over a longer period of time for impact assessment.