2

Framework for Evaluating the SBIR/STTR Programs at the National Institutes of Health

The National Institutes of Health (NIH) has a venerable history of conducting research important to the nation’s public health. NIH traces its origins to 1887, when a laboratory was created within the Marine Hospital Service, the predecessor agency to the U.S. Public Health Service (PHS). In the 1880s, Congress charged the PHS with preventing epidemics by examining passengers on ships arriving from international destinations for clinical signs of infectious diseases. These new obligations coincided with the significant scientific advance in microbiology that offered innovation in diagnostics to assist with disease screening. From this early start, the utility of science to influence public health outcomes was established.

NIH is the largest funder of biomedical research in the world. Its 2020 budget was $46 billion, divided between intramural and extramural research. Intramural research is conducted internally by NIH researchers. Extramural research accounts for about 80 percent of the NIH budget, funding that goes to researchers from universities, institutes and hospitals, and companies. The most common funding mechanism is the Research Project Grant (R01) award, which is provided primarily to academic researchers. Other programs, such as the Graduate Partnerships Program and the Medical Research Scholars Program, among others, are directed at training and enhancing human capital. Key to the distribution of this funding is the rigorous use of a peer review system that is considered the gold standard for scientific review.

The Small Business Innovation Research (SBIR) and Small Business Technology Transfer (STTR) programs promote the translation of scientific findings to technology development in small firms and facilitate the NIH mission to seek fundamental knowledge about the nature and behavior of living systems and the application of that knowledge to enhance health, lengthen life, and reduce illness and disability. As discussed in Chapter 5, the SBIR/STTR programs have funded notable technological advances in new drugs and devices. At their core, these programs represent a critical component of NIH’s mission-specific development of public health innovations, having been instrumental in helping

academic laboratory research achieve markers of nascent commercial development, such as patents and early-stage clinical studies.

The NIH SBIR/STTR programs are a crucial component of the broader set of policies shaping biomedical commercialization. The advent of the SBIR program coincided with the start of the biotechnology industry, in whose development both programs have played an instrumental role. In the early 1980s, passage of the Bayh-Dole Act gave universities the right to apply for patent rights on discoveries from federally funded research and then license that intellectual property (IP) to private firms to advance commercialization. Stanford University’s licensing of the Cohen-Boyer patents on recombinant DNA, the first of which was issued in 1980, marked a commercial success and helped motivate other universities to form technology transfer offices to manage IP (Feldman et al., 2007). Currently, every university in the United States has a technology transfer office through which faculty disclose ideas from federally funded research. Over time, an entrepreneurial ecosystem of incubators, accelerators, advisors, and funders has developed to facilitate the formation of new firms and commercialization of their products. The performance of the life sciences innovation system is grounded in the microeconomic and institutional environment, which has, by and large, been conducive to long-run scientific and technical progress. The SBIR/STTR programs represent a step toward translation, and receiving an award is an often-tracked metric of a firm’s success and that of various entrepreneurial support organizations.

The purpose of this chapter is to provide a framework for understanding the multifaceted nature of the NIH SBIR/STTR programs and the different types of public benefits they provide in advancing the NIH mission and achieving the programs’ stated legislative objectives. The chapter begins by considering the organization of NIH and the broad mandate for the NIH SBIR/STTR programs. It then provides a review of the literature examining the programs’ direct and indirect impacts, which is followed by discussion of the challenges entailed in evaluating the programs. Attention then turns to the ecosystem of innovation relevant to biotechnology and medical devices. The chapter concludes by describing the committee’s approach to the challenges of evaluating the programs in light of their stated objectives, their economic rationale, and the best available data for the evaluation.

NIH ORGANIZATION

NIH is a complex organization comprising 27 individual institutes or centers (ICs), each with a unique health focus, budget, origin date, and internal organization. Over time, Congress has authorized the creation of new institutes in response to changing health priorities. NIH’s funding is provided in the annual Departments of Labor, Health and Human Services, and Education and Related Agencies Appropriations Act. Most of the individual ICs conduct internal research and fund extramural research.

Twenty-four of the 27 ICs administer grants through the SBIR and STTR programs. Currently, each of these ICs sets aside 3.2 percent of its extramural research and development budget for the SBIR program and 0.45 percent for the STTR program. Notably, each IC has great autonomy in administering the programs and defining its portfolio of projects. This diversity creates the first of the challenges entailed in evaluating the SBIR/STTR programs. Rather than two programs, there are, in reality, 24 different programs to evaluate.

THE LIFE SCIENCES ECOSYSTEM

The NIH SBIR/STTR programs are part of a vibrant life sciences innovation ecosystem. This interrelated and interdependent web of institutions and entities contributes to the exploration, development, commercialization, and diffusion of new knowledge and technology. Multiple actors—including universities; large multinational corporations; small entrepreneurial startups; and various intermediary organizations, such as incubators, accelerators, state agencies, and investors—play complementary roles in the U.S. life sciences innovation ecosystem. Discovery has been driven primarily by a robust and independent scientific community focused on the intellectual merit and novelty of investigator-generated research proposals, balanced by input concerning social or governmental priorities.

The overall productivity of the SBIR/STTR programs depends on the structure of the innovation system, which encompasses the participation of and relationships among public and private institutions and the people within them. This system has seen an extraordinary rate of scientific discovery and technological innovation. At the same time, there are ongoing concerns about the ability of this innovation system to translate promising scientific and technical developments into products and tools that can overcome bottlenecks and achieve a high level of diffusion.

Universities play a crucial role in the U.S. life sciences innovation system. In 2019, U.S. universities conducted $83.6 billion in research activity, $44.5 billion of which was funded by federal sources; $34 billion by “other” sources, including universities themselves, other nonprofit research institutions, and state and local governments; and $5 billion by industry (NSF, 2021a). Approximately 58 percent of university research was dedicated to life sciences–oriented research (NSF, 2021b).

The rapid expansion of the scale and scope of biotechnology was driven, at least in part, by the early introduction of a few key “blockbuster” biotechnology drugs that originated from university research. The early biotechnology industry was marked by the founding of numerous companies with strong ties to leading university researchers (Zucker et al., 1998). Many of these companies received significant capital from the still-emerging venture capital sector or the newly introduced public risk capital programs such as the SBIR program.

Genentech is an illustrative and particularly important example of the types of companies emerging during this period. Founded by University of

California, San Francisco researcher Herbert Boyer (of the Cohen-Boyer gene-splicing technology) and venture capitalist Bob Swanson in the mid-1970s, Genentech was able to rapidly develop (and patent) two particularly promising applications of the new genetic technologies—human insulin and human growth hormone. These innovations attracted extraordinary interest because they met an important and unmet human health need—the genetically engineered production of human proteins that had fewer side effects compared with previous products. Most notably, Genentech attracted significant venture capital funding, followed by an enormously successful initial public offering (IPO) in 1980.

The life sciences industry has experienced cyclical variation in investment patterns, with only a few lucrative IPO windows. Returns on investment in the industry are now similar to returns seen by other industries. Importantly, variation in private funding has been buffered by more stable federal support for life sciences innovation research, including funding directed explicitly to small-firm innovators, many of which are startups, through the SBIR/STTR programs.

The modern life sciences innovation system began to emerge in the mid-1990s. While no single event or marker distinguishes this more mature system from its earlier incarnation, several events during the mid-1990s altered the system’s character and ultimate scope of the system. First, several enabling platform technologies, such as polymerase chain reaction (PCR), became cost-effective across various applications, greatly expanding the scope of biotechnology-oriented research and innovation. Second, the institutional shifts transformed the structure of interactions between public and private life sciences research organizations, resulting in an extraordinarily complex research network structure. Finally, significant and sustained investment in the system began to show returns, with an increasing share of all new drug development being grounded in biotechnology.

While the particular structural and institutional aspects of the life sciences innovation system are undoubtedly important, a crucial driver of its performance has been the sustained and growing long-term public investment in life sciences research, primarily through the expansion of NIH. Whereas most other areas of nondefense federal research and development (R&D) funding have either stagnated or declined (in real terms) since the 1980s, life sciences funding has more than tripled (and nearly quadrupled) in real terms. As a result, strikingly, the entire increase in the real nondefense R&D budget over the past 30 years can be attributed to increases in funding for life sciences research. In the 1980s, life sciences research was a relatively minor component of all federal R&D spending (less than 25 percent); today, it represents the majority of all nondefense R&D funding since 2000 (Cockburn et al., 2011).

Three interrelated features of this funding warrant emphasis. First, from 1980 through the late 1990s, the growth rate in NIH funding experienced minimal variability, in sharp contrast with the more volatile private funding environment for biotechnology investment. This steady rate of growth allowed universities and other research organizations to make consistent and coherent long-term

investments, particularly given that this was an era in which the physical capital infrastructure of academic medical centers was considerably expanded. The funding pattern since 1998 has been more variable, with a doubling of the NIH budget in nominal dollars between 1998 and 2003, followed by a flat nominal budget from 2003 through 2008. By 2008, the real-dollar declines in funding that began in 2003 meant that the NIH budget had again reached the level it would have reached along the stable growth path that had characterized the 1980–1998 period.

At the same time, while the overall NIH budget saw a relatively low level of variation in growth (at least until 1998), the funding within NIH was much more variable. Over time, the particular focus and emphasis of the NIH budget have undergone significant shifts. For example, while funding for the emerging AIDS crisis during the early 1980s was essentially nonexistent (in a context of political resistance for several years), AIDS funding received dramatic increases starting in the mid-1980s, ultimately coming to account for a significant share of the overall NIH budget. Similarly, in response to the opportunities afforded by high-throughput sequencing enabled by such technologies as PCR, NIH and Congress were able to direct significant increases in funding to genetics and bioinformatics research, through both the peer-reviewed grant system and special initiatives such as the Human Genome Project. Even as a reasonable level of persistence characterizes the funding for each NIH IC and area on a year-to-year basis, emerging scientific opportunities and particular health care needs require that NIH and congressional funders adapt and reallocate their priorities over time.

The modern life sciences innovation ecosystem is characterized by an extraordinarily high degree of complexity and interdependency and is clustered in a small number of critical locations (Giest, 2021). The life sciences innovation network is highly decentralized and involves multiple linkages between and among institutions, including universities, startup firms, established biotechnology companies, pharmaceutical firms, government, and venture capitalists. The geographic clustering of this highly evolved system in a few key locations (such as the Boston area, the San Francisco Bay Area, and the area around San Diego) is an important feature of the system. Each of these regional clusters is marked by a network with a high level of overlap between public and private research organizations of different sizes and maturity (Feldman et al., 2015). An important implication of this network structure is that the system’s performance depends crucially on the effectiveness of the institutions that support structured knowledge production and transfer between and among R&D organizations.¹

Research is the common element in the origins of the firms in these biotechnology clusters, and universities and research centers are sources of this scientific knowledge. The Boston–Cambridge concentration can be attributed

___________________

¹ Research has shown that the performance of the SBIR and STTR programs varies depending on geography, with awardees doing better in areas with greater access to resources and higher concentrations of high-tech industry activity (Gans and Stern, 2003; Lerner, 2000).

directly to the proactive role of the Massachusetts Institute of Technology in fostering entrepreneurship in new technologies. In downstate New York, New Jersey, Maryland, and Pennsylvania (including Washington, DC, and Philadelphia), substantial concentrations of biotechnology activity result from the historical presence of the nation’s largest pharmaceutical manufacturers and their R&D activity. San Diego, Seattle, and Raleigh–Durham–Chapel Hill emerged as biotechnology clusters as a result of their collocation with well-funded medical research establishments.

Many types of state-level initiatives are aimed at capturing the benefits of life sciences commercialization for economic development purposes (Feldman et al., 2014). Nearly every state in the United States now offers some institutional support for the biotechnology industry, but this support varies with respect to the type and amount of resources allocated, with resources including R&D tax credits, dedicated industry resources centers, and strategic plans. In the case of the SBIR/STTR programs, efforts to encourage SBIR/STTR applications range from holding seminars to providing matching funds (Lanahan, 2016). The downside of this level of interest is that such an emphasis on biotechnology has resulted in bidding wars among states attempting to entice scientists and firms to relocate by offering financial incentives (Moretti and Wilson, 2014).

Many life sciences entrepreneurial support organizations have emerged (Clayton et al., 2018). These include publicly and privately funded accelerators, incubators, and shared wet laboratory space designed to provide new companies with support they need to get off the ground. One of the core advantages, of course, of working with an accelerator or an incubator is that it helps new companies raise the first round of funding they need. In 2017–2018, 33 percent of all U.S. startups that successfully raised Series A funding went through an accelerator or incubator, although receipt of such funding is certainly not the only measure of success. In addition, regions often develop networks of lawyers and venture capitalists who provide expertise and facilitate transactions in what has become an increasingly complex web of relationships among academe, entrepreneurs, and downstream firms (Powell et al., 2005).

BROAD POLICY RATIONALE FOR THE SBIR/STTR PROGRAMS

A vital component of the SBIR/STTR programs is the realization of innovation by small firms. Innovation—the creation of new products, more efficient processes, and better-organized firms and networks of firms—is critical for economic growth and the nation’s international competitiveness (Feldman et al., 2016). Small firms have an essential role in creating innovation by conducting research and pursuing ideas that have transformative potential. Fundamental to innovation is the idea that government investment is required for early-stage idea development to account for the existence of market failures that lead to underinvestment by private firms, although of course, commercializing technological innovation requires a supporting system of private firms, both suppliers and customers; follow-on investors; and developed product markets

(Nelson, 1993). The SBIR/STTR programs form an important component of the U.S. innovation system in focusing on high-risk research and the seeding of small firms that set the system in motion.

Small businesses accounted for 66 percent of the nation’s new job creation from 2000 to 2017. Young startups in particular have emerged as the primary drivers of this trend (Decker et al., 2014, 2016; Haltiwanger et al., 2013). While the barriers and transaction costs facing small businesses are well understood as justifying government intervention, it has become clear that younger small businesses are the dominant drivers of traditional metrics of economic growth (Haltiwanger et al., 2013). Firm age, therefore, is an important moderating variable in assessments of any program that aims to support small firms.

This systems-based view of the role of government in financing R&D is critical for understanding the complete impact of and rationale for the SBIR/STTR programs. Routine conceptions of government performance in R&D funding often focus exclusively on the volume of output per unit of investment. The committee hopes that the results and discussion presented in the forthcoming chapters will clarify that such a narrow conception ignores the indispensable role of the SBIR/STTR programs within a much larger ecosystem of biomedical innovation. The programs’ value is much deeper and broader than its role in correcting the market failure associated with the undersupply of innovation. The committee evaluated the programs along four key dimensions consistent with their legislative goals or potential sources of value:

• stimulating technological innovation,
• helping agencies meet federal R&D needs,
• serving as an engine for the creation of human capital through both firm growth and a broader and more diverse pool of entrepreneurs, and
• promoting commercialization of products and technologies.

Stimulating Technological Innovation

For NIH, the SBIR/STTR programs stimulate technological innovation and facilitate the commercialization of pathbreaking technologies in myriad ways: generating patents, producing collaborative partnerships that result in technology transfer, broadening the geographic scope of NIH’s research activities, and regularly identifying and supporting technological and commercial breakthroughs. And yet none of these impacts is amenable to simple input–output analysis. The web of interconnected partnerships resulting from an award or series of awards can span numerous entities, including the firm; the agency; and other collaborators, contractors, and university partners. The value of the SBIR/STTR programs to each of these entities is practically impossible to measure. Similarly, a single firm might use an award to generate a product whose societal value justifies a decade’s worth of expenditure on the programs. However, this impact

will not show up on the margin in calculating an average total effect across all participants.

The committee cautions against a narrow focus on a sole set of outcomes in evaluating the SBIR/STTR programs. Instead, this report emphasizes two distinct, though both important, forms of evidence for program impacts: direct impacts on grantees and the more dispersed impacts that accrue across the public health and biomedical innovation ecosystems. The former type of impact typifies the bulk of extant literature on SBIR/STTR and related programs, as indicated by the committee’s literature survey, summarized below. The latter type of impact has received less scholarly attention, especially for the SBIR/STTR programs specifically. Evaluation of these impacts requires a more holistic approach that includes not just direct effects on firm outputs, such as patents, publications, sales, products, and jobs, but also benefits that accrue to grantee partners, universities, other firms, and the agency itself (Furman et al., 2002).

For instance, early scholars of innovation policy tended to assess the SBIR/STTR programs through the lens of appropriability, or innovators’ inability to capitalize on the full value of their efforts because of inevitable knowledge spillovers. According to this view, the supply of innovations is therefore suboptimal without government intervention. While any framework for evaluating the SBIR/STTR programs that ignores this key outcome is incomplete, the programs can be viewed more broadly for their contribution to the American innovation system. It is reasonable, then, to ask what level of innovation the two programs support for a given level of public funding. Chapter 5 is devoted to evaluating the programs’ innovation outputs and includes measures of innovation outcomes. The results presented in that chapter highlight that direct innovation, measured through innovative awardee outputs, accounts for only one aspect of the public goods rationale for the SBIR/STTR programs. The programs provide additional public benefits attributable to spillover effects that result from those innovations, in that these spillovers generate further innovation in related technology. They point to the programs’ role in a broader system of innovative activity that is much more complex and wide-ranging than the direct interaction between NIH and awardees.

Helping Agencies Meet Research and Development Needs

The SBIR/STTR programs are intended to facilitate agency objectives by supporting basic research and procurement using a network of small firms and their university partners. For more than 30 years, NIH has used SBIR (and for more than 20 years, STTR) to feed the life sciences ecosystem by directing resources strategically toward the agency’s technological mission areas. These mission areas are, of course, rooted in the rationale for the agency itself. This rationale has historically focused on fostering technological growth trajectories that are inherently uncertain and that unfold over long time horizons in ways that are often fundamental to the subsequent construction of private-market solutions.

The SBIR/STTR programs also represent a key lever for NIH in directing resources strategically toward socially desirable mission areas. In addition to promoting indirect benefits through technology spillovers, setting the nation’s technological agendas, and integrating resources into broader innovation ecosystems, NIH uses the programs to address critical health disparities.

Investigator-initiated and peer-reviewed science approaches have driven life sciences innovation. Even when particular public health priorities have emerged (as in the case of AIDS or COVID-19), the source of the ultimate solutions has been grounded in the open scientific community. These solutions are dependent on the exercise of intellectual freedom and scientific openness and on opportunities for experimentation and diversity at the level of individual researchers and institutions. Product-market incentives steer resource allocation to commercial science, but there remains a need for complementary robust blue-sky research that will have public health benefits available to all. NIH’s SBIR/STTR programs promote a diversity of experimentation that is important to a stream of innovation addressing public health challenges, including in research topics that are in their earliest stages or for which there is no market mechanism to realize a profit.

Serving as an Engine for the Creation of Human Capital

A third objective for the SBIR/STTR programs is to serve as an engine for creating human capital by both fostering firm growth and encouraging a broader, more diverse pool of entrepreneurs who may contribute to life sciences innovation ecosystems. A rationale for the two programs lies in their capacity to attract the best ideas from a larger and more diverse population of entrepreneurs, many of whom face inequitable barriers to market entry in the absence of government involvement. As the 2014 congressional testimony excerpt below shows, these R&D programs are a vital part of U.S. economic policy, especially for minority and women technology entrepreneurs:

Joshi and colleagues (2018) examined workforce diversity within granting agencies, looking specifically at women and underrepresented minority groups, and found a strong link between their representation and the successful conversion of Phase I to Phase II grants for those groups, while agencies with lower representation saw lower conversion rates for those groups. NIH has among

the highest levels of representation of these groups among all mission agencies (Joshi et al., 2018), with an 18,000-person workforce that is 58 percent female and 42 percent minority. This diversity may shape acceptance rates across federal agencies operating SBIR and STTR programs.² More women and minorities funded by the SBIR/STTR programs can mean better outcomes for the health of these groups. Women’s patents were found to have a greater likelihood of addressing women’s health issues, such as breast cancer, postpartum preeclampsia, and fibromyalgia (Koning et al., 2021).

Finally, the pattern of NIH funding over the last decade offers a cautionary lesson about the impact of public funding on research activity (Freeman and van Reenen, 2009). During the 5-year doubling of the NIH budget, a large number of universities and other research organizations made significant dedicated physical capital investments in laboratories along with investments in the expansion of graduate programs and postdoctoral positions in those areas that were receiving the largest increases in NIH funding (such as genomics). An emerging body of evidence suggests that the high level of variability in the aggregate NIH budget over the past decade has likely resulted in a less productive innovation system and distorted the incentives and career dynamics of an entire generation of scientists (Bienenstock et al., 2015; Couzin and Miller, 2007; Goolsbee, 1998; Stephan, 2012; Valdivia, 2017).

Promoting Commercialization of Products and Technologies

A legislative mandate for the SBIR program is to examine its above-discussed ability to alleviate capital market imperfections by acting as a source of seed funding for small startups. The rebranding of the program as “America’s Seed Fund” reflects the core program objective of helping competitive but capital-constrained small businesses weather the “valley of death.”

Life sciences innovation faces additional hurdles due to the nature of the innovation process. Many inventions begin in academic laboratories where there is a need to transfer technology outside the university, which may involve a lengthy and complicated negotiation process (Eisenberg and Cook-Deegan, 2018). The life sciences innovation ecosystem is characterized by intense competition. Individual scientific research teams compete for scientific recognition; universities compete to attract faculty, students, and resources; and biotechnology firms compete with each other to attract scientists, venture capital, and commercialization partners.

Product market competition is oriented primarily around quality and innovation. Much innovation in this sector can be characterized as radical, including remarkable advances in treating disease due to advances in such emerging fields as genomics that have led to identifying and exploiting new physiological mechanisms or new classes of drugs, biologics, and devices, such as selective serotonin reuptake inhibitors and HIV/AIDS therapies. Another

___________________

²https://www.edi.nih.gov/data/demographics.

significant source of economic and health benefits has been incremental innovation in the form of the development of follow-on drugs, biologics, devices, or combination products with differentiated properties, as well as efforts to enhance the effectiveness of existing drugs through reformulations or more effective treatment regimens.

This sector is arguably one of the most regulated sectors of the economy. The Food and Drug Administration (FDA) controls new product introductions and production processes, and federal and state legislation and regulation govern the marketing, distribution, and reimbursement rules for drugs. Significantly, while these regulatory structures are a significant and costly constraint on commercial innovation, they also play an important role in shaping competition. Although the high costs of meeting FDA requirements and managing the FDA approval process lower the direct returns realized for any one successful drug, these costs (and strong patent protection) create significant barriers to entry for new firms.

Therefore, one way of understanding the SBIR/STTR programs’ effectiveness is to assess whether participating firms are achieving such outcomes as follow-on funding, commercialization of products, and development of management teams that reflect present or forthcoming commercial success (see Chapter 5). Collaboration among agency offices, awardees, national labs, universities, and other commercial and academic partners represents an additional overarching task for evaluation of the programs. These partnerships are a direct requirement of the STTR program and influence each of the four program dimensions discussed above. Collaboration outcomes are considered in Chapter 4.

The committee highlights the importance of using these lenses in concert when considering the evidence for the effectiveness of the SBIR/STTR programs. Doing so encourages sensitivity to the trade-offs that can occur for any program seeking to promote positive firm outcomes along these four dimensions while also better reflecting the realities of how the government funds science. Some of these trade-offs, along with several additional evaluation challenges, are discussed below.

Finally, it is important to emphasize that the rewards for innovation are highly skewed, even for those products that can navigate the regulatory system. A small number of “blockbuster” products realize very high sales. The top 100 products account for about one-third of all global revenue, and nearly two-thirds of drugs do not generate sufficient market returns to recoup their development costs (Grabowski and Vernon, 1996). As noted in Chapter 5, for example, 23andMe, which provides genetic health assessment and ancestry information, received six SBIR Phase I awards and two SBIR Phase II awards between 2010 and 2017 and was valued at approximately $3.5 billion in 2021³ (Brown, 2021).

Lanahan and Feldman’s (2018) study of noncompetitive state matching programs encourages evaluation of the SBIR/STTR programs within a broader mix of policies and institutions rather than a silo of direct inputs and outputs. They

___________________

³https://www.sbir.gov/node/1308525; https://www.sbir.gov/sbc/23andme-inc.

found that matching state programs increased the likelihood of Phase I applications and the successful conversion of Phase I to Phase II awards. Additional public funds increased the quality of SBIR/STTR Phase II applications. These findings indicate that additional SBIR/STTR funding leads to greater application intensity, which raises the overall quality of funded proposals.

Additional influences on the programs’ effectiveness include regional institutions and collaborative patterns among awardees, universities, and national labs. The value of these collaborations for innovative activity more generally is clear. A substantial literature supports the finding that high-quality research and technology transfer programs in universities induce commercialization (Lockett and Wright, 2005; Siegel et al., 2003).

Haeussler and Colyvas (2011) surveyed more than 2,000 life scientists in the United Kingdom and Germany and identified numerous drivers of commercial activity among academic researchers and universities. They found that publications and patents mattered in predicting commercial success and collaboration with the private sector. They found further that publications and views of the importance of patents were positive predictors of commercial activity and consulting. These effects were found to be most pronounced in the fields of engineering and clinical medicine, in which commercial applications are relatively more likely.

Lastly, studies have identified procurement and commercialization linkages as another major source of value for the SBIR/STTR programs. A previous assessment by the National Academies (NRC, 2008) found that the SBIR program served as a path to procurement, especially for the Department of Defense. While no studies have specifically analyzed the causal effects of the SBIR/STTR programs on procurement, there is substantial evidence that government procurement in general has produced tremendous commercial applications, prominently including weapon systems (Sherwin and Isenson, 1967) and the iPhone (Mazzucato, 2013).

BROAD CHALLENGES FACING SBIR/STTR ASSESSMENT

With the above framework for the rationale for the SBIR/STTR programs in place, this section identifies empirical challenges faced in evaluating the programs. The variety of theoretical lenses for assessing programs suggests multiple outcome measures, some of which are better represented in the available literature than others. Studies have explored the programs’ impact on sales (Siegel and Wessner, 2012), follow-on private financing (Howell, 2017; Lanahan and Armanios, 2018; Toole and Czarnitzki, 2010; Wallsten, 2000), patents (Howell, 2017; Siegel and Wessner, 2012), and copyrights and trademarks (Siegel and Wessner, 2012). For instance, estimates of the programs’ effect on patenting behavior suggest an innovation-based rationale.

A smaller body of work has examined employment outcomes associated with the SBIR/STTR programs (Howell, 2019; Howell and Brown, 2020; Lanahan et al., 2021; Lerner, 2000; Siegel and Wessner, 2012; Wallsten, 2000).

These studies reflect an expectation that the programs create jobs directly in awarded firms. Moreover, in addition to quantitative outcomes at the firm level, unobserved and more qualitative inputs are salient. For example, firms may leverage infrastructure at government labs and universities, which affects the quality of the research and speed to market. These more qualitative outcomes increase the difficulty of measuring and attributing more complex program effects, such as increasing overall technological output in key NIH mission areas.

An important limitation of much of the current SBIR/STTR evaluation literature is its heavy reliance on interviews (Link and Scott, 2000), surveys (Siegel and Wessner, 2012), case studies (Audretsch et al., 2002), and nonexperimental research designs that limit the ability to determine the true causal effect of the programs. The complete lack of experimental research on NIH’s SBIR/STTR programs in the nearly 40 years since the inception of the SBIR program suggests a strong need to consider conducting or funding such studies and providing researchers access to data on the performance of applicants. Seminal studies that capitalize on rich administrative data (Wallsten, 2000) are due for an update. Finally, regression studies often exploit rich detail on firm characteristics (Siegel and Wessner, 2012; Toole and Czarnitzki, 2007), but it is extremely difficult to measure and adjust for all relevant features of firms that may explain differing performance between awardees and nonawardees.

The present study builds upon prior studies of the NIH SBIR/STTR programs carried out by the National Academies (NASEM, 2015; NRC, 2009), which took a survey-based approach. Since the 2015 report was issued, researchers have begun undertaking more rigorous evaluations using a variety of data to provide rigorous evidence, summarized here. The next section examines some more specific assessment challenges in greater detail, highlighting the committee’s approach to dealing with those challenges and appealing to the relevant literature.

Challenges of Measurement and Attribution

Assessment of the SBIR/STTR programs, or indeed of any government R&D program, is challenging for a number of reasons. For one, it is difficult to observe and measure such direct outcomes of program success as the introduction of a new therapeutic or medical device. Scientific progress is incremental, with long lead times involving multiple organizations. Consider the development trajectory of messenger RNA (mRNA). Hundreds of scientists worked on mRNA vaccines beginning in 1978, followed by decades of dead ends, rejections, and dogged persistence before the coronavirus pandemic brought a breakthrough (Dolgin, 2021).

The conflicting evidence on the SBIR/STTR programs’ performance reflects in part the fact that firms can evolve along multiple pathways through the programs and multiple institutional forces at play in shaping program outcomes. For instance, some firms receive a Phase I award and move quickly to other private funding sources. Others proceed to Phase II, perhaps generating new

SBIR/STTR Phase I applications and ultimately procurement relationships. Conventional empirical approaches to program evaluation tend to miss the qualitative differences among these experiences. Further, a myriad of institutions besides the federal government play a role in determining the programs’ effectiveness. For example, 17 U.S. states have noncompetitive state matching programs for SBIR/STTR winners, and 45 states offer some form of Phase 0 support (Lanahan and Feldman, 2015).

The SBIR and STTR programs funds projects carried out within firms. Startup firms may change their names, and many firms have limited lives, merging with or being acquired by other firms or simply ceasing to exist. After a merger occurs, projects may be continued by the larger firm or discontinued (Cunningham et al., 2018).

Projects are also associated with principal investigators (PIs)—individuals who lead a project. While company names are listed as the official award recipients, the resulting research is attributed to the PIs, many of whom circulate to other companies or strike out on their own, illustrating the building of human capital discussed above and adding to the complexity of program evaluation.

Program evaluators tend to focus on additional gains from an intervention for marginal firms, with program impacts reported in terms of average effects. However, the economic impacts of the SBIR/STTR programs derive largely from a small subset of firms that achieve outstanding results. This impact is an artifact of innovation: outcomes are highly skewed, and a focus on average effects underestimates impact (Scherer and Harhoff, 2000).

The Committee’s Approach to Dealing with Evaluation Challenges

The committee’s focus in this study was on linking structure and process to (multiple) program objectives, focusing not on an existential evaluation of whether the SBIR/STTR programs should exist but on how to make them better as part of the national innovation system. To this end, the committee conducted and documents in this report both qualitative and quantitative research.

The committee’s qualitative assessments focus on the programs’ structure and application and award processes. Specifically, in Chapter 3, the report examines what particular elements of the programs and their underlying processes contribute to (or detract from) the project outcomes and how closely linked the programs’ processes are to achieving their proposed objectives.

Underlying the challenges noted above is the overall priority of determining additionality in estimating program outcomes. In other words, do evaluations reflect a plausible comparison between what happened under the SBIR/STTR programs and what would have happened in the programs’ absence? The literature documents numerous attempts to evaluate the programs by identifying this counterfactual condition and using it as a benchmark for measuring program success. The quantitative approach taken in Chapter 5 uses sophisticated data-matching algorithms. Access to administrative records for

applicants allows a comparison of firms that received SBIR/STTR awards against the comparison group of all firms that applied to the programs. Unfortunately, the committee did not have access to priority scores that would have allowed for a more precise assessment of the quality of proposed projects.

The committee also placed great emphasis on empirical approaches that explicitly address additionality concerns. Chapter 5 presents results of statistical models designed to approximate the counterfactual of what would have happened to awardees in the absence of SBIR/STTR funding. However, these approaches are not sufficient, on their own, for a complete evaluation of the SBIR/STTR programs along the requested dimensions. Therefore, the committee used qualitative interviews to assess outcomes and processes not amenable to quantitative and experimental research designs. This approach also takes into account the need for good descriptive data, especially at the far right and left ends of the distribution of performance outcomes.

Finally, this report emphasizes throughout that individual estimates of program performance should not be interpreted in isolation. Rather, these results are part of the SBIR/STTR programs’ performance within a complex innovation system. Strong performance in certain areas may mean corresponding deficits in others, and individual firm performance will not capture the full value of return on SBIR/STTR investment given the spillovers and unobservable influences within these systems discussed previously. In-depth interviews and detailed descriptions of patterns and findings at multiple SBIR/STTR process levels are essential for explaining these dynamics.

This page intentionally left blank.