SBIR at the Department of Defense (2014)

Appendix A

Overview of Methodological Approaches, Data Sources, and Survey Tools

This report on the Small Business Innovation Research (SBIR) program at the Department of Defense (DoD), is a part of a series of reports on SBIR at the National Institutes of Health (NIH), National Aeronautics and Space Administration (NASA), Department of Energy (DoE), and National Science Foundation (NSF). Collectively, they represent a second-round assessment of the program by the National Research Council (NRC).

The first-round assessment, conducted under a separate ad hoc committee, resulted in a series of reports released from 2004 to 2009, including a framework methodology for that study and on which the current methodology builds.¹ Thus, as in the first-round study, the objective of this second round study is “not to consider if SBIR should exist or not”—Congress has already decided affirmatively on this question, most recently in the 2011 reauthorization of the program.² “Rather, the NRC Committee conducting this study is charged with “providing assessmentǦbased findings of the benefits and costs of SBIR … to improve public understanding of the program, as well as recommendations to improve the program’s effectiveness.” As with the first-round, this study “will not seek to compare the value of one area with other areas; this task is the prerogative of the Congress and the Administration acting through the agencies. Instead, the study is concerned with the effective review of each area.”

These areas refer to the four legislative objectives of the SBIR program:³

__________________

¹National Research Council, An Assessment of the Small Business Innovation Research Program: Project Methodology, Washington, DC: The National Academies Press, 2004.

²National Defense Authorization Act of 2012 (NDAA) HR.1540, Title LI.

³The most current description of these legislative objectives is in the Policy Guidance provided by SBA to the agencies. SBA Section 1.(c) SBIR Policy Directive, October 18, 2012, p. 3.

• Commercialize government-funded research
• Expand the U.S. technical knowledge base
• Improve the participation of women and minorities
• Support agency missions

The SBIR program, on the basis of highly competitive solicitations, provides modest initial funding for selected Phase I projects (up to $150,000) for feasibility testing and further Phase II funding (up to $1 million) for about one-half of Phase I projects.

From a methodology perspective, assessing this program presents formidable challenges. Among the more difficult are the following:

• Lack of data. Only DoD and NSF track outcomes data, and DoD’s tracking is insufficient for careful analysis. There are no systematic efforts by agencies to collect feedback from awardees. No systematic efforts have addressed qualitative data.
• Intervening variables. Analysis of small businesses suggests that they are often very path dependent and, hence, can be deflected from a given development path by a wide range of positive and negative variables. A single breakthrough contract—or technical delay—can make or break a company.
• Lags. Not only do outcomes lag awards by a number of years, but also the lag itself is highly variable. Some companies commercialize within 6 months of award conclusion; others take decades. And often the biggest impacts take many years to peak even after products have reached markets.

ESTABLISHING A METHODOLOGY

The methodology utilized in this second-round study of the SBIR program builds on the methodology established by the NRC committee that completed the first-round study.

Publication of the 2004 Methodology

The committee that undertook the first-round study and the agencies under study formally acknowledged the difficulties involved in assessing SBIR programs. Accordingly, that study began with development of the formal volume on methodology, which was published in 2004 after completing the standard National Academies peer-review process.⁴

__________________

⁴National Research Council, An Assessment of the Small Business Innovation Research Program: Project Methodology, op. cit., p. 2.

The established methodology stressed the importance of adopting a varied range of tools, which meshes with the methodology originally defined by the study committee to include a broad range of tools, based on prior work in this area. The committee concluded that appropriate methodological approaches

The tools identified in the illustration above include many of those used by the NRC committee conducting the first-round study of the SBIR program. Other tools emerged since the initial methodology review.

Tools Utilized in the Current SBIR Study

Quantitative and qualitative tools being utilized in the current study of the SBIR program include the following:

• Case studies. The committee commissioned in-depth case studies of 20 SBIR recipients at DoD. These companies are geographically diverse, demographically diverse, funded by several different components at DoD, and at different stages of the company lifecycle.
• Workshops. The committee convened a number of workshops to allow stakeholders, agency staff, and academic experts to provide unique insights into the program’s operations, as well as to identify questions that need to be addressed.
• Analysis of agency data. A range of datasets covering various aspects of agency SBIR activities were obtained from DoD or DoD components. The committee has analyzed and included these data as appropriate.

__________________

⁵National Research Council, An Assessment of the Small Business Innovation Research Program: Project Methodology, op. cit., p. 2.

TABLE APP A-1 Overview of Approach to SBIR Program Assessment

NOTE: Supplementary tools may be developed and used as needed.
SOURCE: National Research Council, An Assessment of the Small Business Innovation Research Program: Project Methodology, Washington, DC: The National Academies Press, 2004, Table 1, p. 3.

• Open-ended responses from SBIR recipients. For the first time, the committee solicited textual responses in the context of the 2011 survey, drawing more than 700 observations by firms on the SBIR program.
• Agency interviews. Agency staff were consulted on the operation of the SBIR program, and most were helpful in providing information both about the program and about the challenges that they faced.
• Literature review. In the time period since the start of NRC research in this area, a number of papers have been published addressing various aspects of the SBIR program. In addition, other organizations, such as the Government Accountability Office (GAO), have reviewed particular parts of the SBIR program. We have referenced these works in the course of this analysis.

Taken together with our committee deliberations and the expertise brought to bear by individual committee members, these tools provide the primary inputs into the analysis.

We would stress that, for the first-round study and for our current study, multiple research methodologies feed into every finding and recommendation. No findings or recommendations rest solely on data and analysis from NRC surveys; conversely, data from the survey are used to support analysis throughout the report.

COMMERCIALIZATION METRICS AND DATA COLLECTION

Recent Congressional interest in the DoD SBIR program has reflected strong interest in the commercialization of technologies funded through SBIR. This enhanced focus is understandable: the investment made should be reflected in outcomes approved by Congress.

However, no simple definition of “commercialization” exists.⁶ Broadly speaking, it means funding for technology development beyond that provided under Phase II SBIR funding. Given the diversity of components within DoD, it is not surprising that there is considerable variation in the definition of commercialization and in the collection of data that can be used for assessment and measurement.

While all of the components measure “transition,” it is not clear that this means the same thing across DoD. Possible meanings and elements include the following:

• issuance of a certified Phase III contract by a Service or DoD agency (collectively “components”) directly to the small firm;
• adoption of a technology by a program of record (Air Force [AF] simply counts technologies that are flown on AF aircraft);

__________________

⁶See Chapter 4 for related analysis of commercialization in the SBIR program.

• utilization of a technology in weapons systems that are delivered to the war fighter;
• licensing of technologies to prime contractors (primes) and other parties serving DoD components;
• sale of products and services to primes for use on DoD systems (this may or may not include sale of data rights); and
• any sale of goods or services derived from SBIR-funded technologies, to DoD or to other purchases, including the U.S. private sector, other U.S.-based government agencies, and foreign buyers.

Different kinds of commercialization are recorded and reported differently as well. Figure A-1 shows the different kinds of commercialization and the data sources that can be used to track them. Numbers in the figure are as follows:

1. All commercialization: including both DoD-related and private-sector sales and licensing.
2. All DoD-related commercialization: including sales and licensing revenues generated from either DoD directly or through primes to DoD.
3. DoD-only sales and revenues: from small business directly to the DoD (excludes sales through primes).
4. Defense-related sales to the primes only: excluding DoD-direct sales.
5. Non-DoD related sales: including all sales in the U.S. private sector and all exports (to both public- and private-sector entities abroad). These sales may be in the defense sector but not to DoD or the primes.
6. DoD Phase III contracts: This box represents all the DoD sales resulting from SBIR awards that should be designated as Phase III contracts. However, Phase III currently only rarely covers downstream impacts of SBIR awards beyond the first Phase III contract.
7. DoD-sales correctly certified as Phase III: this is the share of all Phase III sales that are correctly entered into the Federal Procurement Data System (FPDS).
8. Exports: part of non-DoD commercialization.

Challenges in Tracking Commercialization

The nested boxes in Figure A-1 indicate something of the scale and complexity of the challenges involved in accurately tracking commercialization. So it is useful to break the tracking issue into three broad components:

• within DoD;
• in the DoD primes; and
• all remaining commercialization.

FIGURE A-1 Accounting for commercialization in the DoD SBIR program.
SOURCE: Committee-generated.

Tracking Commercialization Within DoD

The Federal Procurement Data System (FPDS) is the primary tool for tracking commercialization within DoD. FPDS is designed to contain all federal contracts, including all SBIR contracts and all subsequent direct contracts between the federal government and small businesses.^7,8

All federal contracts for more than $3,000 must be entered into the FPDS.⁹ However, while Phase I and Phase II contracts are almost always entered accurately as SBIR contracts, that is not the case for Phase III.

__________________

⁷Information about DoD use of FPDS is drawn from <http://www.acq.osd.mil/dpap/pdi/eb/federal_procurement_data_system__next_generation_fpds-ng.html>, accessed July 3, 2013, and from interviews with Navy SBIR staff and with the Navy civilian FPDS contractor.

⁸Detailed information about FPDS is contained in the FPDS-NG User Manual, <http://www.fpds-ng.com/wiki/index.php/FPDS-NG_User_Manual>, accessed July 3, 2013.

⁹GAO FPDS-Next Generation FAQ, <https://www.fpds.gov/wiki/index2.php/FPDSNG_FAQ>, accessed July 7, 2013.

Interviews with senior Services SBIR staff confirm that Contracting Officers (COs) do not always recognize that a follow-on SBIR contract with a small firm is a “Phase III contract” and hence do not mark the contract as such in FPDS.

In October 2012, the DoD Inspector General (IG) office reported that SBIR and Small Business Technology Transfer (STTR) awards were often incorrectly labeled under the requirements for effective competition; the report stated that SBIR and STTR awards should be viewed as having been competed, but that they were often not marked as such in the FPDS by contracting officers.¹⁰ As a result, SBIR Phase III contracts are not completely reflected in FPDS, and hence that data and analysis based on FPDS systematically undercount the impact of SBIR transitions within DoD.

Some components have tried to address this problem. In Navy, a civilian contractor has been assigned full time to work with FPDS. The contractor double-checks to ensure that all Phase III contracts with DoD are marked as such in FPDS. So far as can be determined, none of the other DoD components uses a full-time contractor or staff member for this purpose. Because the Navy process consistently finds errors, it seems very likely that an unknown number of SBIR Phase III contracts at other components go unrecorded as such. No estimate is available for the amount of under-reporting that can be attributed to this problem.

There is a second and perhaps more important difficulty in tracking commercialization within DoD itself: the limits of Phase III certification. Phase III certification typically covers only the first contract after the end of Phase II. In some cases, Phase III certification extends beyond the first contract, but that does not appear to be typical. For many technologies, this first Phase III contract is only the start of a long journey for the company and the technology. For the most successful technologies, subsequent contracts may cover thousands of units and tens of millions of dollars. This extensive commercialization is not recorded through FPDS. In some ways this is understandable, because there is no obvious way for contracting officers to certify the original sources of complex technologies. Yet as a result, the long-term impact of SBIR awards eludes capture in FPDS.

The DoD Acquisitions Desk Reference defines Phase III as follows:

1.   Each of the following types of activity constitutes SBIR Phase III work:

__________________

¹⁰DoD IG Report, DODIG-2013-002, “Improvement Needed With DoD Single-Bid Program to Increase Effective Competition for Contracts,” October 4, 2012, p. 12.

i.   commercial application of SBIR-funded R/R&D financed by non-Federal sources of capital (Note: The guidance in this Policy Directive regarding SBIR Phase III pertains to the non-SBIR federally-funded work described in (ii) and (iii) below. It does not address the nature of private agreements the SBIR firm may make in the commercialization of its technology.);

ii.   SBIR-derived products or services intended for use by the Federal Government, funded by non-SBIR sources of Federal funding;

iii.  continuation of R/R&D that has been competitively selected using peer review or scientific review criteria, funded by non-SBIR Federal funding sources.¹¹

Although in principle there is no time limit on Phase III certified technologies—and the Small Business Administration (SBA) Policy Guidance document makes this clear—in practice DoD struggles even to make the link between Phase II and the first Phase III contract. There appears to be no standard process in place at any component to help small businesses ensure that subsequent contracts are also certified as Phase III.

Tracking Through the Primes—The Importance of the Company Commercialization Record (CCR)

Once beyond DoD itself, the ability to track commercialization declines substantially. FPDS no longer applies because primes are not required to enter subcontracts into FPDS. As the Air Force (AF) SBIR program manager said, “We have no vision into private-sector transactions between small businesses and prime contractors.”¹² Primes are not required to identify or to track SBIR award contracts. There are indeed currently no incentives to do so, and some significant incentives not to.

Primes are sometimes required to meet contracting goals focused on small businesses, among other demographics.¹³ However, there is no separate breakout for SBIR contracts. It is doubtful whether the primes themselves track SBIR-related contracts. According to Navy staff, one—but only one—of the primes that serve Navy maintains a database of SBIR-related subcontracts.

__________________

¹¹DoD Acquisitions Desk References, Annex A, Section 3, “Definitions,” <http://www.acq.osd.mil/osbp/sbir/sb/resources/deskreference/annex_a.shtml#Target3>, accessed July 9, 2013.

¹²Interview with David Sikora and Richard Flake, June 28, 2013.

¹³In fiscal year (FY)2012, the target for small business subcontracts at DoD was 36.7 percent of total contract value. See DoD Office of Small Business Programs, “Small Business Goal and Objectives,” <http://www.acq.osd.mil/osbp/about/sbProgramGoals.shtml>, accessed July 9, 2013.

Some other primes appear to be actively hostile to the notion of tracking and including SBIR-based technologies.¹⁴

The data rights attached to SBIR-based technologies provide a substantial disincentive for primes to track and identify Phase III commercialization. Technologies certified as SBIR bestow significant data rights on their owners, which may mean that small businesses have substantially more leverage in negotiations with primes.

In the Navy system commands with the deepest commitment to tracking SBIR outcomes, program managers readily admit that they have very limited capacity to track outcomes through the primes. It appears that primes account for a considerable—though uncountable—share of overall commercialization. Estimates from Navy staff at different program executive offices (PEOs) suggested that this share ranges from 25 percent to 50 percent.

Data from both the current and the previous (2005) NRC survey indicate that about 60 percent of Phase II respondents with sales generated revenues either directly to DoD or to DoD primes. Of these, about two-thirds of revenues came directly from DoD, and one-third through the primes. So FPDS misses the approximately one-third of the overall sales made by SBIR companies that through the primes.

Sales and revenues from the primes are collected through the Company Commercialization Record (CCR) (see Box A-1). CCR was introduced in the early 2000s precisely to capture more of the company’s commercialization activities, beyond direct DoD contracts. It is the only DoD-wide activity to do so.

However, CCR has some important weaknesses. It is entirely self-reported. Although the contractor managing the database seeks to cross-check reported sales above a benchmark amount, no systematic cross-checks are made between reported sales and FPDS or other data sources. In addition, although companies must sign off on their CCR reports under penalty of perjury, there are nonetheless incentives that would encourage firms to overstate their commercial results: CCR scores are sometimes taken into account in determining funding for subsequent projects, and very low scores are expected to exclude companies from the program altogether. However, it is also fair to say that the inherently fuzzy nature of commercialization reporting allows considerable room for leeway, so companies can interpret commercialization results in the most positive light.

On the other hand, updating CCR is a burden, and it is not clear how comprehensive this process has been: firms that have stopped applying for SBIR funding, have graduated from the program, or have been acquired have no incentive to participate. As a result—as with other data sources—the depth and the completeness of the information captured declines over time. It does not appear that DoD has conducted any recent analysis that would help to determine

__________________

¹⁴Interview with Dean Putnam, NAVSEA SBIR Program Manager, June 25, 2013.

the extent to which CCR continues to provide useful and important information, or indeed whether the burden on companies should be changed.

Yet despite these weaknesses, CCR remains the only systematic source of information about SBIR commercialization outside FPDS. Therefore not only

is it the best source of information about commercialization through the primes, but also it provides a view into commercialization within DoD beyond Phase III and outside DoD.

Unfortunately, perhaps as a result of resource constraints, very little appears to have been done in terms of utilizing CCR results for assessing commercialization. Although Navy has been comparing commercialization outcomes between Program Executive Offices (PEO) for a number of years, using both the Federal Procurement Data System (FPDS) and the Company Commercialization Record (CCR), this has not been the case at other Services or apparently at Office of Small Business Program.

The existence of these different outcome metrics makes it possible at least in theory to compare SBIR-related outcomes across different Services, different topics, different sponsoring organizations, different primes, and indeed different types of technologies. Perhaps now that the additional administrative resources are available, these opportunities to understand how different variables affect program success will be exploited more fully.

Other Tracking Tools

Some components have developed additional tools for tracking outcomes. Although mostly focused on ensuring that the award process runs smoothly, the Navy program managers’ database does allow program managers to track contracts that they know about.¹⁵ According to the Navy SBIR Program Office,

The Navy program managers’ database is thus a well-developed tool for helping those responsible for various aspects of SBIR manage the program at Navy. It is not, however, primarily designed for tracking outcomes; to the extent that it does so, it relies primarily on data from FPDS. Some program managers—for example, at the PEO Integrated Warfare Systems (IWS)—also maintain their own database and indeed devote considerable resources to

__________________

¹⁵Private communication, July 11, 2013.

¹⁶Navy SBIR program support staff, private communication.

consistently populating it. IWS tasks a junior staffer to call SBIR recipients on a regular basis after the end of their Phase II contract to determine whether there has been further commercialization, which is then tracked in the IWS database.¹⁷ This intensive data-gathering approach is similar to that adopted in the past at NSF.¹⁸

Why New Data Sources Are Needed

Congress often seeks evidence about the effectiveness of programs or indeed about whether they work at all. This interest has in the past helped to drive the development of tools such as CCR. However, in the long term the importance of tracking lies in its use to support program management. By carefully analyzing outcomes and CCR’s associated program variables, program managers will be able to manage more successfully.

We have seen significant limitations to all of the available data sources. FPDS captures a limited dataset, and even that is not accurate especially with regard to Phase III. CCR is self-reported and subject to a range of conflicting incentives and its own additional limitations. It is also an aggregate measure, not an FY measure, and hence is not compatible with FPDS. In addition, DoD does not make data from CCR and from FPDS available for review, at least not at the disaggregated level necessary for detailed statistical analysis.

OVERVIEW OF THE NRC SURVEY

Our analysis of the SBIR program at DoD makes use of case studies, interviews, and other qualitative methods of assessment. These remain important components of our overall methodology, and a chapter in this report is devoted to lessons drawn from case studies. But qualitative assessment alone is insufficient.

The Role of the NRC Survey

The NRC survey offers some significant advantages over other data sources. It—

• covers all kinds of commercialization inside and outside of DoD;
• provides a rich source of textual information in response to open-ended questions;
• permits some quantitative analysis based on the development of a comparison group, even though the comparison group has limitations;
• probes more deeply into company demographics and agency processes;

__________________

¹⁷Douglas Marker, SBIR Program Manager, IWS, presentation to NRC, June 25, 2013.

¹⁸See National Research Council, An Assessment of the SBIR Program at the National Science Foundation, Washington, DC: The National Academies Press, 2008, Table 5.2-12.

• addresses principal investigators (PIs), not just company business officials;
• allows comparisons with previous data-collection exercises; and
• addresses other Congressional objectives for the program beyond commercialization.

At the same time, however, we are fully cognizant of the limitations of this type of observational survey research in this case. To address these issues while retaining the utility and indeed explanatory power of survey-based methodology, this report contextualizes the data by comparing results to those from the NRC survey conducted as part of the first-round assessment of the SBIR program (referred to below as the “2005 NRC Survey”¹⁹). This report also adds transparency by publishing the number of responses for each question and indeed each subgroup, thus allowing readers to draw their own conclusions about the power of the statistical conclusions being drawn.

We contracted with Grunwald Associates LLC to administer a survey to DoD award recipients.²⁰ This survey is built closely on the 2005 NRC survey but is also adapted to lessons learned and includes some important changes discussed in detail below. A methodology subgroup of the committee was charged with reviewing the survey and the reported results for best practice and accuracy. The survey was carried out simultaneously to a survey focused on the SBIR programs at NSF and NASA.²¹

The primary objectives of the survey were as follows:

• Provide an update of the program “snapshot” taken in 2005, maximizing the opportunity to identify trends within the program;
• Probe more deeply into program processes, with the help of expanded feedback from participants and better understanding of program demographics;
• Improve the utility of the survey by including a comparison group; and

__________________

¹⁹The survey conducted as part of the current, second-round assessment of the SBIR program is referred to below as the “2011 NRC Survey” or simply the “survey.” In general, throughout the report, any survey references are understood to be to the 2011 NRC Survey unless specifically noted otherwise.

²⁰Grunwald Associates LLC is a research and consulting firm located in Bethesda, Md. The firm specializes in assignments that require an in-depth understanding of multiple market segments, and is experienced in deploying state-of-the-art research methodologies. Grunwald Associates has conducted the annual PBS Survey on Educational Technology and Media for eight years. Grunwald Associates is also one of the core partners on the U.S. Department of Education’s Connected Online Communities of Practice initiative, and works with clients such as the National Park Service, AT&T, Adobe, Microsoft, and smaller companies.

²¹Delays at NIH and DoE in contracting with the NRC combined with the need to complete work contracted with DoD NSF and NASA led the Committee to proceed with the survey at three agencies only.

• Reduce costs and shrink the time required by combining three 2005 survey questionnaires—for the firm, Phase I, and Phase II awards—into a single questionnaire.

Survey Characteristics

In order to ensure maximum comparability for a time series analysis, the survey for the current assessment was based as closely as possible on previous surveys, including the 2005 NRC Survey and the 1992 GAO survey.

Given the limited population of winners, the starting point for consideration was to deploy one questionnaire per successful project. However, we were also aware that the survey imposes burdens on respondents. Given the detailed and hence time-consuming nature of the survey, it would not be appropriate to over-burden potential recipients, some of whom were responsible for many awards over the years.

An additional point of consideration was that this survey was intended to add detail on program operations, rather than the original primary focus on program outcomes. Agency clients were especially interested in probing operations more deeply. We decided that it would be more useful and effective to administer the survey to PIs—the lead researcher on each project—rather than to the registered company point of contact (POC), who in many cases would be an administrator rather than a researcher. This decision was reinforced by DoD’s decision on privacy grounds not to provide company-level access to the DoD commercialization database, which contains current POC information.

The survey was therefore designed to collect the maximum amount of relevant data, consistent with our commitment to minimizing the burden on individual respondents and to maintaining maximum continuity between surveys. Survey questionnaires were to be sent to PIs of all projects that met selection characteristics, with a maximum of two questionnaires per PI.

Based on reviewer feedback about the previous round of assessments, we determined that for purposes of contextualization, comparison groups would be developed that would provide the basis for further statistical analysis.

Key similarities and differences between the 2005 and 2011 NRC surveys are captured in Table A-2.

The 2011 NRC Survey included awards made from FY1998 to FY2007 inclusive. This end date allowed completion of Phase II awards (which nominally fund 2 years of research) and provided a further 2 years for commercialization. This time frame was consistent with the 2005 NRC survey, which surveyed awards from FY1992 to FY2001. It was also consistent with a previous GAO study, published in 1992, which surveyed awards made through 1987.

The aim of setting the overall time frame at 10 years was to reduce the impact of difficulties generating information about older awards, because some companies and PIs may no longer in place and because memories fade over time. Reaching back to awards made in 1998, while ensuring comparability, generated few results from older awards.

Determining the Survey Population

Following the precedent set by both the original GAO study and the first-round NRC study of the SBIR program, we differentiated between the total population of SBIR recipients, the preliminary survey target population, and the effective population for this study, which is the population of respondents that were reachable.

The effective survey population was the denominator for the survey, used to determine both response rates and the limits of statistical precision that are achievable given the number of responses received.

Initial Filters for Potential Recipients

Determining the effective study population required the following steps:

• acquisition of data from the sponsoring agencies (DoD, NSF, and NASA) covering record-level lists of award recipients;
• elimination of records that did not fit the protocol agreed upon by the committee—namely, a maximum of two questionnaires per PI (in cases where PIs received more than two awards, the awards were selected by agency [NASA, NSF, DoD, in that order], then by year [oldest], and finally by random number); and

TABLE A-2 Similarities and Differences: 2005 and 2011 NRC Surveys

• elimination of records for which there were significant missing data—in particular, where emails and/or contact telephone numbers were absent.

Note that this approach does not directly limit the number of responses from a single firm, only from a single PI. This process of excluding awards either because they did not fit the selection profile or because the agencies did not provide sufficient or current contact information reduced the total award list

provided by the agencies to a preliminary survey population of approximately 15,000 awards.

Secondary Filters to Identify Recipients with Active Contact Information

This nominal population still included many potential respondents whose contact information was complete, but who were no longer associated with the contact information provided and hence effectively unreachable. This is unsurprising given that there is considerable turnover in both the existence of and the personnel working at small businesses and that the survey reaches back 13 years to awards made in FY1998. Recipients may have switched companies, the company may have ceased to exist or been acquired, or telephone and email contacts may have changed, for example. Consequently, two further filters were utilized to help identify the effective survey population.

• First, contacts were eliminated for which the email address bounced twice. Because the survey was delivered via email, the absence of a working email address disqualified the recipient. This eliminated approximately 30 percent of the preliminary population.
• Second, efforts were made to determine whether non-bouncing emails were in fact still operative. Email addresses that did not officially “bounce” (i.e., return to sender) may still in fact not be active. Some email systems are configured to delete unrecognized email without sending a reply; in other cases, email addresses are inactive but not deleted. So a non-bouncing email address did not equal a contactable PI. In order to identify not contactable PIs, we undertook an extensive telephone survey. Telephone calls were made to every awardee in the preliminary survey population at NASA and NSF and to a random sample of 1,000 thousand awardees at DoD. On the basis of responses to the telephone survey, we were able to ascertain that on average 47 percent of the respondents with ostensibly not bouncing email addresses were in fact not contactable.

There was little variation between agencies or between programs in the quality of the lists provided by the agencies, based on these criteria, although there was, not surprisingly, considerable variation between Phase I and Phase II awards, especially for DoD.²²

__________________

²²The share of preliminary contacts that turned out to be not contactable was higher for this survey than for the previous NRC survey in 2005. We believe this is primarily because company points of contact (POCs) have less churn than do program managers (PMs) (often being senior company executives).

Deployment

The survey opened on October 4, 2011, and was deployed by email, with voice follow-up support. Up to four emails were sent to the effective population (emails discontinued once responses were received). In addition, two voice mails were delivered to non-respondents between the second and third and between the third and fourth rounds of email. In total, up to six efforts were made to reach each questionnaire recipient.

After members of the data subgroup of the committee determined that sufficient responses had been received and that additional efforts to acquire new responses were not likely to be cost effective, the survey was closed on December 19, 2011. The survey was therefore open for a total of 11 weeks.

Response Rates

Standard procedures were followed to conduct the survey. These data collection procedures were designed to increase response to the extent possible within the constraints of a voluntary survey and the survey budget. The population surveyed is a difficult one to contact and obtain responses from as evidence from the literature shows.²³ Under these circumstances, the inability to contact and obtain responses always raises questions about potential bias of the estimates that cannot be quantified without substantial extra efforts that would require resources beyond those available for this work.

The lack of detailed applications data from the agency makes it impossible to estimate the possible impact of non-response bias. We, therefore, no have evidence either that non-response bias exists or that it does not. For the areas where the survey overlaps with other data sources - notably DoD's mandatory Company Commercialization database - results from the survey and the DoD data are similar.

Table A-3 shows the response rates at DoD by phase, based on the effective study population after all adjustments. The extent to which a given response rate is sufficient depends entirely on the uses to which the survey is being put and on the degree of statistical precision required. These issues are addressed below. For purposes of comparison, we provide the response rates for the previous survey and for all three agencies surveyed (see Table A-4).

The higher response rates at NSF for 2011 appear to result from extended agency efforts to encourage participation and from a closer focus on

__________________

²³Many surveys of entrepreneurial firms have low response rates. For example, Aldrich and Baker (1997) found that nearly a third of surveys of entrepreneurial firms (whose results were reported in the academic literature) had response rates below 25 percent. See Howard E. Aldrich and Ted Baker. 1997. Blinded by the Cites? Has There Been Progress in Entrepreneurship Research? pp. 377-400 in Donald L. Sexton and Raymond W. Smilor (eds.), Entrepreneurship 2000. Chicago: Upstart Publishing Company.

TABLE A-3 Response Rates by Agency, Program, Phase, and Population

SOURCE: 2011 NRC Survey.

identifying respondents. The higher response rates for Phase I at DoD in 2011 result from more extensive efforts to qualify potential respondents. More current contact data were provided by DoD in 2005, which were not available on privacy grounds in 2011. This may explain the somewhat lower response rate for DoD Phase II respondents in 2011.

Note that all subsequent references to the 2011 NRC Survey in this appendix address only responses for awards made by DoD.

TABLE A-4 Comparing Response Rates for 2005 and 2011

SOURCES: 2005 NRC Survey and 2011 NRC Survey.

DOD RESPONSES AND RESPONDENTS

Responses by Year of Award

The distribution of responses by year is largely comparable between Phase I and Phase II, except for 2007, which accounts for a much higher percentage of Phase I awards. There were more Phase II awards from 2004 and 2006. Table A-5 shows DoD SBIR responses by year of award.

For both Phase I and Phase II, more responses were received from more recent years. This is not surprising, because it is easier to successfully contact PIs from more recent awards and the probability of survival increases.

Statistical testing indicates that the year of award is significantly different depending on whether the company received a Phase I or Phase II award, at the .05 level of statistical significance.

The survey primarily reached companies that were still in business: overall, 97 percent of respondents indicated that the companies were still in business.²⁴

Effort at Comparison Group Analysis

Several readers of the reports in the first round analysis of SBIR suggested the inclusion of comparison groups in the analysis. There is no simple and easy way to acquire a comparison group for Phase II SBIR awardees. These are technology based companies at an early stage of company development,

TABLE A-5 DoD SBIR Responses by Year of Award

SOURCE: 2011 NRC Survey.

__________________

²⁴2011 NRC Survey, Question 4A.

which have the demonstrated capacity to undertake challenging technical research and to provide evidence that they are potentially successful commercializers. Given that the operations of the SBIR program are defined in legislation and limited by the Policy Guidance provided by SBA, randomly assigned control groups were not a possible alternative. Efforts to identify a pool of SBIR-like companies were made by contacting the most likely sources - Dunn and Bradstreet and Hoovers—but these efforts were not successful, as insufficiently detailed and structured information about companies was available.

In response, we sought to develop a comparison group from among Phase I awardees that had not received a Phase II award from the three surveyed agencies (DoD, NSF, and NASA) during the award period covered by the survey (1999-2008). After considerable review, however, we concluded that the Phase I-only group was also not appropriate for use as a statistical comparison group. In the interests of providing researchers with a full view of the data collected, Appendix G includes tables showing both the Phase I only and Phase II survey responses for questions where both groups were surveyed.