National Academies Press: OpenBook

An Assessment of the Small Business Innovation Research Program: Project Methodology (2004)

Chapter:4. Developing Operational Definitions and Concepts

« Previous: 3. Clarifying Study Objectives
Suggested Citation:"4. Developing Operational Definitions and Concepts." National Research Council. 2004. An Assessment of the Small Business Innovation Research Program: Project Methodology. Washington, DC: The National Academies Press. doi: 10.17226/11097.
Suggested Citation:"4. Developing Operational Definitions and Concepts." National Research Council. 2004. An Assessment of the Small Business Innovation Research Program: Project Methodology. Washington, DC: The National Academies Press. doi: 10.17226/11097.
Suggested Citation:"4. Developing Operational Definitions and Concepts." National Research Council. 2004. An Assessment of the Small Business Innovation Research Program: Project Methodology. Washington, DC: The National Academies Press. doi: 10.17226/11097.
Suggested Citation:"4. Developing Operational Definitions and Concepts." National Research Council. 2004. An Assessment of the Small Business Innovation Research Program: Project Methodology. Washington, DC: The National Academies Press. doi: 10.17226/11097.
Suggested Citation:"4. Developing Operational Definitions and Concepts." National Research Council. 2004. An Assessment of the Small Business Innovation Research Program: Project Methodology. Washington, DC: The National Academies Press. doi: 10.17226/11097.

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

4. Developing Operational Definitions and Concepts The study will identify core operational terms and concepts in advance of full development of the methodology. The following represents an initial identification of some terms and concepts: The quality of SBIR research Quality is a relative concept by definition, so an assessment of SBIR research quality must compare it to the quality of other research.26 Quality is also subjective, so the realization of value may depend on its perceived utility. The principal comparison here will be with other extra-mural research funded by the same agencies. The question of whether comparisons should focus only on R&D by other small businesses is yet to be addressed; this decision may be made on an agency-by-agency basis.27 SBIR’s value to agency missions Given that agency missions and their associated sub-unit objectives differ substantially among, (and even within) agencies, the issue of SBIR’s value to agency missions will be addressed largely in the context of individual agency analysis. While a more generic set of answers would be helpful, it will be important to emphasize the challenges posed by multiple agencies with multiple missions, executed by multiple subunits. For example, some agencies, such as DoD and NASA, are “procurement agencies,” seeking tools for the nation’s military, while others, such as NSF and NIH, are not. These different goals may change the agency’s vision of SBIR’s role quite fundamentally. Generic mission elements include: • Technology needs (i.e., agency-identified technology gaps, such as a missing vaccine delivery system identified as a priority by NIH28); • Procurement needs (i.e., technologies that the agency needs for its own internal use, e.g., optical advances for smart weapons at DoD). • Expansion or commercialization of knowledge in agency’s field of stewardship (e.g., funding in relatively broad sub-fields of information technology at NSF);29 • Technology transfer (i.e., promoting the adoption of agency-developed technology by others). Technology may become available to the sponsoring agency and others through a variety of paths. These include o Procurement from supply chain providers, o Purchase by the agency on the open market via successful commercialization by the SBIR firm (e.g., purchase of DoD-R&D-supported advanced sonar equipment), o Use by others of the technology whose development is sponsored by the agency and made available through such means as licensing, partnership arrangements, or by purchase on the open market. (e.g., a power plant may adopt technology available on the market and fostered by DoE’s SBIR, or a public health clinic may adopt a new vaccine delivery system available on the market and fostered by NIH’s SBIR). To address agency-specific missions (e.g., national defense at DoD, health at NIH, energy at DoE), the Committee will closely consult with agency staff to develop operational definitions of success--in some cases at the level of sub- units (e.g., individual NIH institutes and centers.) Some overlay will likely occur (e.g., defense-health needs.) Finally, agencies will undoubtedly have their own conceptions of how their SBIR program is judged in relation to their missions, and it is possible (perhaps likely) that some of these views will not fit well in the areas listed. The Committee is sensitive to these distinctions and differences, and will articulate these concepts at an early stage. 26 See K. Buchholz. "Criteria for the analysis of scientific quality," Scientometrics 32 (2), 1995:195-218. 27 See M. Brown, T.R. Curlee and S.R. Elliott, “Evaluating Technology Innovation Programs: The Use of Comparison Groups to Identify Impacts,” Research Policy, 24, 1995. 28 See for example, “Micromachined Ultrasound Ejector Arrays For Aerosol-Based Pulmonary Vaccine Delivery.” Response to SBIR Proposal PHS 2001 NIP Topic 009, Technologies to Overcome the Drawbacks of Needles and Syringes Contract No: 200-2001-00112 29 See, for example, Maryann Feldman and Maryellen Kelley, “Leveraging Research and Development: The Impact of the Advanced Technology Program,” in National Research Council, The Advanced Technology Program, C. Wessner, ed. Washington, D.C.: National Academy Press, 2001, for a potential model of how an agency can use SBIR to help foster specific areas of technical expertise. The paper, however, does not address the extent to which this is a conscious goal of DoD. 12

The extent of commercialization SBIR is charged with supporting the commercialization of technologies developed with federal government support. In many agencies, this requirement is articulated as a focus on the “commercialization” of SBIR supported research.30 At the simplest level, commercialization means, “reaching the market,” which some agency managers interpret as “first sale”: the first sale of a product in the market place, whether to public or private sector clients.31 This definition is certainly practical and defensible. However, it risks missing significant components of commercialization that do not result in a discrete sale. At the same time, it also fails to provide any guidance on how to evaluate the scale of commercialization, which is critical to assessing the degree to which SBIR programs successfully encourage commercialization: the sale of a single widget is not the same as playing a critical role in the original development of Qualcomm’s cell-phone technology. Thus, the Committee’s assessment of commercialization will require working operational definitions for a number of components. These include: • Sales—what constitutes a sale? • Application—how is the product used? For example, products like software are re-used repeatedly. • Measuring scale—over what interval is the impact to be measured. (e.g., Qualcomm’s SBIR grant was by all accounts very important for the company. The question arises as to how long the dollar value of Qualcomm’s wireless related sales, stemming from its original SBIR grant, should be counted.)32 • Licensing—how should commercial sales generated by third party licensees of the original technology be counted. Is the licensing revenue from the licensee to be counted, or the sales of that technology by the licensee— (or both)? • Complex sales—technologies are often sold as bundles with other technologies (auto engines with mufflers for example). Given this, how is the share of the total sales value attributable to the technology that received SBIR funding to be defined? • Lags—some technologies reach market rapidly, but others can take 10 years or more. What is an appropriate discount rate and timeframe to measure award impact? Metrics for assessing commercialization can be elusive. Notably, one cannot easily calculate the full value of developed “enabling technology” that can be used across industries. Also elusive is the value of material that enables a commercial service. In such cases, a qualitative approach to “commercialization” will need to be employed. While the theoretical concept of additionality will be of some relevance to these questions, practicalities must govern, and the availability of data will substantially shape the Committee’s approach in this area. This is particularly the case where useful data must be gathered from thousands of companies, often at very considerable expense in dollars and time. 33 The NRC study will resolve these very practical questions by the early stages of the study’s second phase. The Committee plans to adapt, where appropriate, definitions and approaches used in the Fast Track study for the current study.34 Broad economic effects SBIR programs may generate a wide range of economic effects. While some of these may be best considered in a national context, others fall more directly on participating firms and on the agencies themselves. The Committee will consider these possible benefits and costs in terms of the level of incidence. 30 A key objective of the 1982 Small Business Innovation Development Act is to increase private sector commercialization derived from federal research and development. The role of SBIR in stimulating commercialization was cited as a justification in the reauthorization of the Act in 1992: that SBIR “has effectively stimulated the commercialization of technology development through federal research and development, benefiting both the public and private sectors of the Nation.” 31 For analysis of observed variations in timelines for commercialization, see NISTIR 6917 ”Different Timelines for Different Technologies: Evidence from the Advanced Technology Program” at 32 For a profile of Qualcomm, see 33 Buisseret, T.J., Cameron, H., and Georghiou, L. (1995) “What difference does it make? Additionality in the public support of R&D in large firms”, International Journal of Technology Management, Vol.10, Nos. 4/5/6 pp. 587-600. See also, Luke Georghiou, “Impact and Additionality of Innovation Policy,” Paper presented at the Six Countries Programme on Innovation, Spring 2002, Brussels. 34 See National Research Council, SBIR: An Assessment of the Department of Defense Fast Track Initiative, 2000, op. cit. 13

Participating firms Economic effects on firms include some or all of the following elements: • Revenue from sale or adoption of SBIR-developed products, services, or processes (this tracks quite closely but not 100 percent with commercialization) • Changes in the firm’s access to capital, including ways in which SBIR awards have helped (or hindered) recipient companies access capital markets • Change in firm viability and sustainability, including how the SBIR program helped bridge the gaps between these interrelated stages of the innovation process;35 o Conception o Innovation o Product development o Entry into market • Changes in the propensity to partner and the nature of the partnerships o The impact of SBIR on the frequency with which companies develop partnerships o The nature of the partnerships—are they public or private partners? • Enhanced firm growth, productivity, profitability o Change in employment and capitalization36 o Change in firm productivity o Change in profits • Intellectual property developed by the firm o Work in this section will follow closely on the Fast Track study model, seeking to identify ways in which the recipient firms were affected in the areas listed.37 The agencies Effects on the agencies include the following: • Effects on mission support • Effects on agency research efficiency, including whether -- o SBIR has helped to generate technologies that agencies might not otherwise have developed in the same timeframe without the program o SBIR is an effective way for agencies to fund competitive research, presumably compared to non-SBIR research funding for small scale requirements o There are significant benefits to agency missions from the specific effort of SBIR to capture research by small firms. Benchmark numbers for small business contributions to agency research programs before SBIR (pre-1983), and outside SBIR (other programs?) may be needed. Research efficiency implies a review of the returns to the agency from SBIR investment vs. other research investment. It is important to acknowledge, however, that this analysis will likely not be based on hard rate of return analysis, because the data necessary for such analysis is unlikely to be available at the agency level. • Effects on agency procurement efficiency 35 Though commonly conceived as a linear process, innovation is characterized by significant complexity. For a discussion of this complexity, see, National Research Council, The Small Business Innovation Research Program: Program Diversity and Assessment Challenges, C. Wessner, ed. Washington, D.C.: National Academies Press, 2004. 36 The employment effects of research are most often indirect—through the application or commercialization of the research. 37 Ibid. 14

Effects on society • Social returns refer to the returns to society at large, including private returns and spillover effects. Using the Link/Scott approach from Fast Track as a model is expected to help us conceptualize our approach to this broad effect.38 Other approaches may also be useful. • Small business support refers to the positive social externalities associated with a vibrant small business sector, including community cohesiveness and improvements to life made possible new products. Measuring the impact of program support for small business is a major objective,39 given that support encourages the commercialization of public investment in R&D, the achievement of national missions, and the encouragement of small firm growth. • Training in both business development and in technology and innovation The Audtresch/ J. Weigand/ C. Weigand Fast Track paper provides a good methodological basis for addressing “training.”40 Non-economic benefits While it is possible to view almost all non-economic effects through the lens of economic analysis, pushing all effects to economic measurement is usually not feasible and may not be appropriate. Certain effects have been specifically defined as positive outcomes by Congress, regardless of whether they have any measurable impact on economic well-being. This section addresses non-economic benefits, which will in turn have non-economic metrics attached to them, discussed later in this paper. Knowledge benefits The missions of several agencies explicitly state the requirement of advancing knowledge in the relevant field. For the SBIR program, this requirement can be viewed from two distinct but complementary perspectives: • Intellectual property,41 which is governed by a set of legal definitions, and is susceptible to close measurement via analysis of patent filings and other largely quantitative assessment strategies.42 Intellectual property rights are generally used to convert knowledge to property for commercial benefit of the owner. At the same time, mechanisms of intellectual property can help to disseminate knowledge to others. A patent, for example, gives the holder exclusive rights, but provides information to others. Intellectual property also includes “trade secrets.” In many cases, the “know how” that firms keep proprietary may be the most important intellectual property produced by the research. These may be less susceptible to measurement. • Non-property knowledge is much less well defined but nonetheless of great importance. Non-property knowledge ranges from formal activities (e.g., papers published in refereed journals, and seminars) to very informal activities (e.g., discussions among researchers and worker mobility). Many relevant concepts are discussed in the literature on human capital. Non-property knowledge is related to education and training and encompasses network capital and tacit expertise that an engineer or scientist may possess. 38 See Link, A. N. and Scott, J. T. Public Accountability: Evaluating Technology-Based Institutions, Boston: Kluwer Academic Publishers, 1998. Link and Scott use published agency data and interviews to determine key indicators, including private hurdle rates, additional anticipated development time after Phase II, additional cost, life of the commercialized technology, and proportion of value appropriated by firm. These allow estimates of social and private returns. Link and Scott indicate that the rates of return on the SBIR Phase II investment for Fast Track were 84 percent for society and 25 percent for private investors. 39 See also section 4.5 below. 40 See David Audretsch, Jeurgen Weigand and Claudia Weigand, “The Impact of the SBIR on creating entrepreneurial behavior,” Economic Development Quarterly Vol. 16, No. 1, February 2002, pp. 32-38. Audretsch/Weigand/Weigand identify interesting spillover effects of SBIR grants on non-recipient scientists and engineers, in terms of career paths, entrepreneurial activities and their timing, etc. 41 Intellectual property is divided into two categories: Industrial property, which includes inventions (patents), trademarks, industrial designs, and geographic indications of source; and Copyright, which includes literary and artistic works such as novels, poems and plays, films, musical works, artistic works such as drawings, paintings, photographs and sculptures, and architectural designs. Rights related to copyright include those of performing artists in their performances, producers of phonograms in their recordings, and those of broadcasters in their radio and television programs. See 42 For a summary of such measures, see Table 1 in European Commission, Directorate General Enterprises “Enterprises and SME” Programme, “European Trend Chart on Innovation,” June 2000. Accessed at 15

Other potential non-economic benefits • Environmental impacts • Safety • Quality of life Trends in agency funding for small business For this study, “small business funding” will be defined as synonymous with SBIR. A definition of “small” is needed. The SBIR definition (fewer than 500 employees) is quite broad.43 Dividing firm participants into size subcategories may be advantageous. We plan a breakdown of small firms by size, taking into account existing SBA classifications and based on natural divisions as emerge from the data. The agencies have considerable discretion in defining which agency expenditures and disbursements they consider to be R&D, and thus subject to the percentage requirements of the SBIR set aside. Small firms also receive R&D funding directly from the agencies outside the SBIR program, and receive subcontracts for R&D from primes or other subcontractors, whose original funding source was federal R&D. Since in some cases the prime or intermediate contractor may also be a small business, there is an opportunity for double counting as well as for undercounting. It is important to keep in mind that the congressional intent was to increase the amount of federal R&D funding ultimately reaching small businesses. Small businesses, in many cases, cannot take on more R&D funding, as they do not have the expert staff, or the culture to do R&D. Thus, there might conceivably be a sort of saturation effect. The issue of absorptive capacity also occurs in the case of fast moving high tech firms, which may not willing to risk the overhead and delay involved in seeking federal funds at all. In the present study, saturation effects can be examined in part by investigating the relationship between the growth of grant-program funding and the growth of grant-program applications. Insights into the impacts of expansions in grant funding on small-business response capacity and on research quality may be gained by analyzing ATP’s experience between 1993 and 1994, based on changes in reviewer technical scores and small business application rates as the program was expanded dramatically between 1993 and 1994. Best practices and procedures in operating SBIR programs Issues related to administrative process, both within agencies and across agencies, will be defined over the course of the first phase of the NRC study. Areas to be addressed may include: • Outreach • Topic development • Application procedures and timelines • Project monitoring • Agency management funding • Project funding limitations • Bridge funding • Post SBIR Phase II support 43 According to the Small Business Administration, a small business is a concern that is organized for profit, with a place of business in the United States, and which operates primarily within the United States or makes a significant contribution to the U.S. economy through payment of taxes or use of American products, materials, or labor. Further, the concern cannot be dominant in its field, on a national basis. Finally, the concern must meet the numerical small business size standard for its industry. SBA has established a size standard for most industries in the U.S. economy. The most common size standards are 500 employees for most manufacturing and mining industries, 100 employees for all wholesale trade industries, $6 million for most retail and service industries, $28.5 million for most general & heavy construction industries, $12 million for all special trade contractors, and $0.75 million for most agricultural industries. 16

Next: 5. Potential Metrics for Addressing Study Objectives »
An Assessment of the Small Business Innovation Research Program: Project Methodology Get This Book
Buy Paperback | $43.00 Buy Ebook | $34.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

In response to a Congressional mandate, the National Research Council conducted a review of the SBIR program at the five federal agencies with SBIR programs with budgets in excess of $100 million (DOD, NIH, NASA, DOE, and NSF). The project was designed to answer questions of program operation and effectiveness, including the quality of the research projects being conducted under the SBIR program, the commercialization of the research, and the program's contribution to accomplishing agency missions. This report describes the proposed methodology for the project, identifying how the following tasks will be carried out: 1) collecting and analyzing agency databases and studies; 2) surveying firms and agencies; 3) conducting case studies organized around a common template; and 4) reviewing and analyzing survey and case study results and program accomplishments. Given the heterogeneity of goals and procedures across the five agencies involved, a broad spectrum of evaluative approaches is recommended.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook,'s online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!