SBIR at the Department of Defense (2014) / Chapter Skim
Currently Skimming:
Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools
Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools
Pages 235-258
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 235... ...
Instead, the study is concerned with the effective review of each area." These areas refer to the four legislative objectives of the SBIR program:3 1 National Research Council, An Assessment of the Small Business Innovation Research Program: Project Methodology, Washington, DC: The National Academies Press, 2004. 2 National Defense Authorization Act of 2012 (NDAA)
|
From page 236... ...
Accordingly, that study began with development of the formal volume on methodology, which was published in 2004 after completing the standard National Academies peer-review process.4 4 National Research Council, An Assessment of the Small Business Innovation Research Program: Project Methodology, op.
|
From page 237... ...
5 National Research Council, An Assessment of the Small Business Innovation Research Program: Project Methodology, op.
|
From page 238... ...
SOURCE: National Research Council, An Assessment of the Small Business Innovation Research Program: Project Methodology, Washington, DC: The National Academies Press, 2004, Table 1, p.
|
From page 239... ...
We would stress that, for the first-round study and for our current study, multiple research methodologies feed into every finding and recommendation. No findings or recommendations rest solely on data and analysis from NRC surveys; conversely, data from the survey are used to support analysis throughout the report.
|
From page 240... ...
6. DoD Phase III contracts: This box represents all the DoD sales resulting from SBIR awards that should be designated as Phase III contracts.
|
From page 241... ...
FPDS is designed to contain all federal contracts, including all SBIR contracts and all subsequent direct contracts between the federal government and small businesses.7,8 All federal contracts for more than $3,000 must be entered into the FPDS.9 However, while Phase I and Phase II contracts are almost always entered accurately as SBIR contracts, that is not the case for Phase III. 7 Information about DoD use of FPDS is drawn from |
From page 242... ...
Yet as a result, the long-term impact of SBIR awards eludes capture in FPDS. The DoD Acquisitions Desk Reference defines Phase III as follows: SBIR Phase III refers to work that derives from, extends, or logically concludes effort(s)
|
From page 243... ...
continuation of R/R&D that has been competitively selected using peer review or scientific review criteria, funded by non SBIR Federal funding sources.11 Although in principle there is no time limit on Phase III certified technologies -- and the Small Business Administration (SBA) Policy Guidance document makes this clear -- in practice DoD struggles even to make the link between Phase II and the first Phase III contract.
|
From page 244... ...
Data from both the current and the previous (2005) NRC survey indicate that about 60 percent of Phase II respondents with sales generated revenues either directly to DoD or to DoD primes.
|
From page 245... ...
These firms are not allowed to receive any subsequent SBIR awards until they have updated their CCR information. Note that this information must be updated for every previous SBIR award.
|
From page 246... ...
Although Navy has been comparing commercialization outcomes between Program Executive Offices (PEO) for a number of years, using both the Federal Procurement Data System (FPDS)
|
From page 247... ...
FPDS captures a limited dataset, and even that is not accurate especially with regard to Phase III. CCR is self-reported and subject to a range of conflicting incentives and its own additional limitations.
|
From page 248... ...
The survey was carried out simultaneously to a survey focused on the SBIR programs at NSF and NASA.21 The primary objectives of the survey were as follows: Provide an update of the program "snapshot" taken in 2005, maximizing the opportunity to identify trends within the program; Probe more deeply into program processes, with the help of expanded feedback from participants and better understanding of program demographics; Improve the utility of the survey by including a comparison group; and 19 The survey conducted as part of the current, second-round assessment of the SBIR program is referred to below as the "2011 NRC Survey" or simply the "survey." In general, throughout the report, any survey references are understood to be to the 2011 NRC Survey unless specifically noted otherwise. 20 Grunwald Associates LLC is a research and consulting firm located in Bethesda, Md.
|
From page 249... ...
Successful and more recently funded firms more likely to respond. Research by Link and Scott demonstrates that the probability of obtaining research project information by survey decreases for less recently funded projects, and it increases the greater the award amount.b Nearly 75 percent of Phase II respondents to the 2011 NRC Survey received awards after 2003, largely because winners from more distant years are more difficult to reach: small businesses regularly cease operations, are acquired, merge, or lose staff with knowledge of SBIR awards.
|
From page 250... ...
Commercialization lag. Although the 2005 NRC Survey broke new ground in data collection, the amount of sales made -- and indeed the number of projects that generate sales -- are inevitably undercounted in a snapshot survey taken at a single point in time.
|
From page 251... ...
Reduce costs and shrink the time required by combining three 2005 survey questionnaires -- for the firm, Phase I, and Phase II awards -- into a single questionnaire. Survey Characteristics In order to ensure maximum comparability for a time series analysis, the survey for the current assessment was based as closely as possible on previous surveys, including the 2005 NRC Survey and the 1992 GAO survey.
|
From page 252... ...
The effective survey population was the denominator for the survey, used to determine both response rates and the limits of statistical precision that are achievable given the number of responses received. Initial Filters for Potential Recipients Determining the effective study population required the following steps: acquisition of data from the sponsoring agencies (DoD, NSF, and NASA)
|
From page 253... ...
This process of excluding awards either because they did not fit the selection profile or because the agencies did not provide sufficient or current contact information reduced the total award list
|
From page 254... ...
There was little variation between agencies or between programs in the quality of the lists provided by the agencies, based on these criteria, although there was, not surprisingly, considerable variation between Phase I and Phase II awards, especially for DoD.22 22 The share of preliminary contacts that turned out to be not contactable was higher for this survey than for the previous NRC survey in 2005. We believe this is primarily because company points of contact (POCs)
|
From page 255... ...
For the areas where the survey overlaps with other data sources - notably DoD's mandatory Company Commercialization database - results from the survey and the DoD data are similar. Table A-3 shows the response rates at DoD by phase, based on the effective study population after all adjustments.
|
From page 256... ...
Note that all subsequent references to the 2011 NRC Survey in this appendix address only responses for awards made by DoD. TABLE A-4 Comparing Response Rates for 2005 and 2011 Phase 2011 NRC Survey 2005 NRC Survey II Effective Number of Response Effective Number of Response Population Responses Rate (Percent)
|
From page 257... ...
2000 or earlier 3.8 2.7 2001 3.8 5.6 2002 5.4 6.9 2003 7.7 10.3 2004 9.0 15.2 2005 16.2 16.2 2006 16.4 22.7 2007 37.7 20.3 100.0 100.0 N= 390 765 SOURCE: 2011 NRC Survey. 24 2011 NRC Survey, Question 4A.
|
From page 258... ...
In the interests of providing researchers with a full view of the data collected, Appendix G includes tables showing both the Phase I only and Phase II survey responses for questions where both groups were surveyed.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.