SBIR at the National Science Foundation (2015) / Chapter Skim
Currently Skimming:
Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools
Appendix A: Overview of Methodological Approaches, Data Sources, and Survey Tools
Pages 203-221
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 203... ...
represents a secondround assessment of the program undertaken by the National Academies of Sciences, Engineering, and Medicine.1 The first-round assessment, conducted under a separate ad hoc committee, resulted in a series of reports released from 2004 to 2009, including a framework methodology for that study and on which the current methodology builds.2 The current study is focused on the twin objectives of assessing outcomes from the programs and of providing recommendations for improvement.3 The 1 Effective July 1, 2015, the institution is called the National Academies of Sciences, Engineering, and Medicine. References in this report to the National Research Council, or NRC, are used in an historic context identifying programs prior to July 1.
|
From page 204... ...
The SBIR programs, on the basis of highly competitive solicitations, provide modest initial funding for selected Phase I projects (up to $150,000) , and for feasibility testing and further Phase II funding (up to $1 million)
|
From page 205... ...
Tools Utilized in the Current SBIR Study Quantitative and qualitative tools being utilized in the current study of the SBIR program include the following: • Surveys. The Committee commissioned two extensive surveys of NSF SBIR award recipients as part of the analysis.
|
From page 206... ...
programs? Measures Peer review Sales; follow up Patent counts Agency scores, funding; progress; and other IP/ procurement Publication IPO employment of products counts, growth, resulting from Citation number of new SBIR work analysis technology firms Tools Case studies, Phase II surveys, Phase I and Phase Program agency program manager II surveys, case manager program surveys, case studies, studies, study of surveys, case studies, study study of repeat repeat winners, studies, agency of repeat winners bibliometric program winners, analysis studies, study bibliometric of repeat analysis winners Key Research Difficulty of Skew of returns; Measures of Major Challenges measuring significant interagency actual success interagency quality and and inter-industry and failure at the differences in of identifying differences project and firm use of SBIR to proper level; relationship meet agency reference of federal and missions group state programs in this context NOTE: Supplementary tools may be developed and used as needed.
|
From page 207... ...
No findings or recommendations rested solely on data and analysis from Academies surveys; conversely, survey data was used to support analysis throughout the report. COMMERCIALIZATION METRICS AND DATA COLLECTION Recent congressional interest in the SBIR program has to a considerable extent focused on bringing innovative technologies to market.
|
From page 208... ...
In this regard, the NSF SBIR program can benefit from access to the Academies survey data. The survey work provides quantitative data necessary to provide an evidence-driven assessment and, at the same time, allows management to focus on specific questions of interest.
|
From page 209... ...
THE ACADEMIES SURVEY ANALYSIS Traditional modes of assessing the NSF SBIR program include case studies, interviews, and other qualitative methods of assessment. These remain important components of the Academies' overall methodology, and a chapter in the current report is devoted to lessons drawn from case studies.
|
From page 210... ...
Very limited information is available about SBIR award recipients: company name, location, and contact information for the PI and the company point of contact, agency name, and date of award (data on woman and minority ownership are not considered reliable)
|
From page 211... ...
On the basis of successive data sets collected from vector editable National Institutes of Health (NIH) SBIR award recipients, it is estimated that total sales from all responding projects will be considerably greater than can be captured in a single survey.e This underscores the importance of follow-on research based on the now-established survey methodology.
|
From page 212... ...
A subgroup of this Committee with particular expertise in survey methodology also reviewed the survey and drew in current best practices. The 2010 Phase II/IIB survey covered only NSF, while the 2011 survey covered NSF, DoD, and NASA simultaneously.10 The primary objectives of the 2011 Survey (in combination with the 2010 Phase IIB Survey)
|
From page 213... ...
This end date allowed completion of Phase II-awarded projects (which TABLE A-2 Similarities and Differences: 2005 and 2011 Surveys Item 2005 Survey 2011 Survey Respondent selection Focus on Phase II winners ✓ ✓ All qualifying awards ✓ PIs ✓ POCs ✓ Max number of questionnaires per respondent < 20 2 Distribution Mail ✓ No Email ✓ ✓ Telephone follow-up ✓ ✓ Questionnaire Company demographics Identical Identical Commercialization outcomes Identical Identical IP outcomes Identical Identical Woman and minority participation ✓ ✓ Additional detail on minorities ✓ Additional detail on PIs ✓ New section on agency staff ✓ New section on company recommendations for SBIR ✓ New section capturing open-ended responses ✓
|
From page 214... ...
Determining the Survey Population Following the precedent set by both the original GAO study and the first round of Academies analysis, this Committee differentiates between the total population of SBIR recipients, the preliminary survey target population, and the effective population for this study, which is the population of respondents that were reachable. Initial Filters for Potential Recipients Determining the effective study population required the following steps: • acquisition of data from the sponsoring agencies -- DoD, NSF, and NASA -- covering record-level lists of award recipients; • elimination of records that did not fit the protocol agreed upon by the Committee -- namely, a maximum of two questionnaires per PI (in cases where PIs received more than two awards)
|
From page 215... ...
In order to identify non-reachable PIs, the Committee undertook an extensive telephone survey. For NSF, telephone calls were made to every targeted award recipient in the preliminary survey population that did not respond to the initial email invitation to participate.
|
From page 216... ...
during the award period covered by TABLE A-3 2011 Survey Response Rates at NSF Preliminary population 996 Missing contact information –212 Contact moved/uncontactable –376 Effective population 408 Responses 186 Surveys as Percentage of Awards Contacted 45.6 Surveys as Percentage of Sample 18.7 SOURCE: 2011 Survey, without inclusion of data from Phase IIB.
|
From page 217... ...
Including Results from the 2010 Phase IIB Survey The 2010 Phase IIB Survey of NSF Phase II awards was carried out with the objective of comparing outcomes between award recipients receiving standard Phase II awards and those receiving Phase IIB enhancements in addition. This survey generated interesting responses and highlighted some important differences between the groups, and these are addressed in Chapter 5.
|
From page 218... ...
1998 2.2 1999 5.9 2000 3.7 2001 2.4 2002 8.6 2003 6.8 2004 11.0 2005 20.5 2006 12.7 2007 11.2 2008 6.8 2009 8.1 100.0 NOTE: Phase II N=186 SOURCE: 2011 Survey; 2010 Phase IIB Survey. The Phase IIB Survey Population NSF provided the Academies with a data set containing records on 285 Phase IIB awards; data included the PI's name and email address.
|
From page 219... ...
. Survey Responses Based on the provision of 281 apparently valid email addresses by NSF and 281 control group awards, the preliminary survey deployment list included 562 email addresses, one per project.
|
From page 220... ...
A more detailed breakout of survey responses by year indicates that with two exceptions, the two groups are quite similar (see Table A-6)
|
From page 221... ...
SOURCE: 2010 Phase IIB Survey.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.