National Academies Press: OpenBook
« Previous: 4 SBIR Program Outcomes
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 105
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 106
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 107
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 108
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 109
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 110
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 111
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 112
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 113
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 114
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 115
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 116
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 117
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 118
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 119
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 120
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 121
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 122
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 123
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 124
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 125
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 126
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 127
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 128
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 129
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 130
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 131
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 132
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 133
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 134
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 135
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 136
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 137
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 138
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 139
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 140
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 141
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 142
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 143
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 144
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 145
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 146
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 147
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 148
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 149
Suggested Citation:"5 Program Management." National Research Council. 2009. An Assessment of the SBIR Program at the National Aeronautics and Space Administration. Washington, DC: The National Academies Press. doi: 10.17226/12441.
×
Page 150

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

5 Program Management 5.1  INTRODUCTION: ASSESSING SBIR IN A RESTRUCTURING NASA As with other parts of NASA, the NASA SBIR program has, experienced sequential waves of reorientation and restructuring. Mission objectives have changed very substantially, far more than at other SBIR agencies. During NASA’s reorganization of 2003-2004, the agency’s SBIR program became a component of the Advanced Space Technology Program within the Exploration Systems Mission Directorate (ESMD), which is charged with imple- menting NASA’s planned exploration of Mars and other space exploration proj- ects. In 2006, further reorganization led a change in the balance of management power between the Mission Directorates and the centers, with the former assum- ing much more direct authority over SBIR topic and award selection. Because of this churn, any assessment of program management at NASA must deal with a moving target. Extensive changes in management structures mean that data regarding past activities is of limited relevance in guiding current management. This chapter details how NASA implements its SBIR management strategy.  It begins by focusing on issues related to the SBIR award cycle, including topic   The NASA SBIR/STTR management team during the course of this analysis was led by:   • Carl G. Ray (Code RC)—SBIR/STTR Executive Director; Oversight-Strategic Direction; SBIR/ STTR Selection Official, NASA Headquarters, Washington, DC.   W. Paul Mexcur (GSFC)—SBIR/STTR Program Manager; SBIR/STTR Program Operations; • Program Management, Goddard Space Flight Center, Greenbelt, MD.   Karin Huth (GRC)—SBIR/STTR Procurement Manager; Procurement Oversight. • 105

106 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION selection, the evaluation of proposals, and selection of awards. The chapter then examines NASA’s commercialization effort that includes a detailed review of the regional dimension of NASA’s SBIR program. The chapter closes with an analysis of challenges for the future of SBIR at NASA, given the agency’s new organizational structure and mission focus. An annex to this chapter describes the SBIR program at each NASA center. 5.2  MANAGING SBIR AT NASA 5.2.1  Guiding Principles NASA has based the management of its SBIR program on the following four principles: • Aligning research topics to the highest technology priorities of the agency. • Focusing on program effectiveness as measured by Phase III commercialization. • Enhancing program efficiency by using advanced information technology. • Providing opportunity for a cross section of small U.S. business.  5.2.2  Program Administration The NASA SBIR program has varied over the years in terms of its degree of centralization. Currently, NASA’s SBIR program is managed at multiple levels. • Level 1—SBIR Program Executive (Headquarters; agency-wide). • Level 1—SBIR program Mission Directorate liaisons (Headquarters, Mission Directorates). • Level 2—SBIR Program and Procurement Policy Managers (agency wide). • Level 3—SBIR Field Center Program Managers (Centers). • Level 4—Contract Officer Technical Representative (COTR) (projects). Level 1. Program Executive.  Located at NASA Headquarters, and supported by a national office, the program executive focuses on overall program administra- tion. Overall program policy, effectiveness, and assessment are the responsibility of the Headquarters Program Executive. The national office, located until recently at Goddard, has now moved to Ames.   NASA Program Management Web site, accessed at <http://sbir.nasa.gov/>, July 2005.   The current program executive is Carl G. Ray. His title is Program Executive, Technology Infusion, Innovative Partnerships Program (IPP) Office.

PROGRAM MANAGEMENT 107 Level 1. Mission Directorates.  Each of four Mission Directorates (MDs) has assigned a senior staffer as liaison between the technology programs run by the MD’s and the SBIR program. Following the 2006 restructuring, the Mission Di- rectorates now dominate topic selection, and approve project selection.  Level 2. SBIR Program and Procurement Policy Managers.  The NASA SBIR Program Management Office, at the NASA Ames Research Center, runs SBIR in conjunction with NASA Mission Directorates and centers. The NASA Shared Ser- vices Center provides the overall procurement management for the programs. Level 3. Field Centers.  Prior to FY2006, program operations were managed at each of the ten NASA Field Centers. Following the FY2006 reorganization, the program will run through only four field centers (Ames, JPL, Glenn, and Langley) At each center, an SBIR Field Center Program Manager administers the program. Contracts are managed by NASA’s Contracting Officer at each center. Level 4. The COTR.  The Contract Officer Technical Representative (COTR) serves as the primary contact between the project and NASA on a contract’s technology focus and objectives, and handles assessment of project progress. The COTR is a staffer at one of the centers. 5.2.3  Administrative Budget NASA’s budget for administering the SBIR programs is approximately $3.8 million per year. This funding—3.2 percent of the $119 million SBIR budget— comes from separate agency funds (not SBIR). This budget does not include the substantial costs associated with employee time used on the program, including in particular the time needed to develop and approve topics and subtopics, and to evaluate proposals. NASA has not calculated the value of this time, although the recent change to a full-cost accounting approach will make it easier to estimate full administrative costs in the future. 5.2.4  FY2006 Reforms The impact of the 2006 reorganization has been to refocus SBIR on the NASA’s core mission objective, de-emphasizing the previous stress on outside commercialization. Interviews with all of the Mission Directorate liaisons indicate that the bal- ance of power between the centers and Headquarters changed substantially in FY2005-2006. The reorganization helps to address a dissatisfaction with the outcomes of the   The FY2006 reforms are described in Section 5.2.4.

108 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION previous approach. The 2002 Commercial Metrics report (covering 1983-1996) found that only about six percent of NASA’s 1,739 SBIR Phase II awards during this period supported technologies that were eventually infused into NASA or other federal programs via Phase III funding. The reorganization also reflects changing needs and priorities within NASA. The addition of new missions and the expansion of existing ones have placed additional demands on Mission Directorates, squeezing funding for basic research. As a result of the reorganization, Mission Directorates are now focusing on aligning research funded through SBIR with specific technologies that can be taken up (or in NASA-speak “infused”) into the their own technology de- velopment programs. Whereas commercialization was the primary priority of NASA’s SBIR program (or, at least, an priority equal to the support for the agency’s mission) the focus of the program since the 2006 reorganization is squarely on support for the NASA mission. For SBIR, this involves finding and developing technologies that can help NASA meet its very specific needs and requirements. Overall, this new clarity of focus appears to be a positive development. As described in some detail in Chapter 4 (Outcomes), the low volume and high de- gree of specificity (e.g. space-hardiness) required to meet NASA’s needs makes it less likely that SBIR funded technologies can spin off into commercial sales.  5.3  THE AWARDS PROCESS In this section, we discuss the details of the NASA awards process. This can be disaggregated into the following components: • Topic development, including efforts to align topics with the needs of the agency. • Outreach into the business and technology communities, to help ensure that the best possible proposals reach NASA. • Project selection, including an assessment of commercialization potential. NASA sets aside 2.5 percent of its extramural research and development budget for SBIR awards. Each year NASA identifies various R&D topics, rep- resenting scientific and technical problems that the agency needs to solve, for   “Phase III funding” comprises contractual or other monies awarded to a SBIR project for federal agency use of the subject technology after expiration of a SBIR Phase II award.   Of course, some companies have made this transition successfully, but overall, there are significant structural impediments standing against successful commercialization from NASA SBIR project—as opposed for example to DoD, where there may be a huge potential market within the agency, or NIH where the private-sector market for SBIR-funded technologies is also potentially enormous.

PROGRAM MANAGEMENT 109 pursuit by small businesses under the SBIR program. These topics are bundled together into annual NASA "solicitations," which are publicly announced re- quests for SBIR proposals from interested small businesses. A small business can identify an appropriate topic that it wants to pursue from these solicitations and, in response, propose a project for an SBIR award. The following sections describe the SBIR award process that was in place at NASA throughout much of the study period for this assessment. This structure, implemented in the mid-1990s, has since been substantially altered by reforms in 2005-2006, as noted above. 5.3.1  Selecting SBIR Topics 5.3.1.1  Aligning Topics with Agency Needs Topic and subtopic development and selection are the primary tools used to ensure that the SBIR program is closely aligned with agency needs. Beginning in 1995, NASA’s SBIR program initiated a series of steps to make the program more consistent with NASA’s mission. The goal was to make the SBIR program a strategic asset for NASA by integrating all aspects of these programs with NASA’s mission. The change involved implementing an organi- zational structure that better supported the technology goals of what were then NASA’s four Mission Directorates: Aeronautics, Exploration Systems, Science, and Space Operations. This set into motion several initiatives that included selecting topics integral to NASA’s overall mission, creating a system that closely tracks each contract for program management, leveraging non-SBIR agency funds to support un- funded high quality projects, and developing metrics for evaluating commercial outcomes. 5.3.1.2  Deciding on Topics A tension exists between encouraging proposals focused on solving very spe- cific problems facing NASA, and making the topics broad enough to encourage enough firms to apply, and the application of innovative solutions. NASA’s staff acknowledges this tension; they note that topic definitions at NASA are designed to walk the necessarily fine line between overly tight specifications and the devel- opment of exciting technologies that are not however needed by the agency. Program management issues guidelines to each of the centers. Centers may then propose subtopics. These are eventually prioritized at NASA Headquarters. To facilitate this decision-making process, NASA holds an annual Solicita- tion Development Workshop. This is attended by both Mission Directorate repre-   See NASA solicitation Web page. Accessed at <http://sbir.gsfc.nasa.gov/SBIR/solicit.htm>.

110 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION sentatives and center managers. Centers propose subtopics at the workshop, while the Mission Directorates note which subtopics support their goals. Program management then tries to design solicitations based on the agency’s highest priority technology needs, as reflected by center rankings and Mission Directorate needs. If a technology is needed by more than one center, the topic and subtopic are assigned to the center that ranked it highest. Subsequently, NASA’s SBIR/STTR Program Manager makes prioritizing recommendations about topics to the SBIR/STTR Executive Director, who makes the final decisions. 5.3.2  Agency Outreach 5.3.2.1  Agency Outreach Objectives and Methods Like other agencies, NASA staff note it is important to reach out to the small business community, in order to encourage higher quality proposals from a wide range of applicants. At NASA, this outreach is undertaken mainly through par- ticipation in national and regional SBIR conferences. NASA does not organize these events. Representatives from the ten NASA field centers regularly attend these conferences, where they set up briefing charts and displays, and hand out literature about the program. The NASA SBIR/STTR office also works with NASA’s Office of Small and Disadvantaged Business Utilization (known as “Code K” at NASA) to increase participation by small and disadvantaged businesses. As a key part of its outreach, NASA maintains an extensive SBIR Web site. NASA’s application process is entirely Web-based. NASA does not believe that lack of information about the program is a significant barrier for potential applicants. 5.3.2.2  Agency Outreach Benchmarks NASA receives a large number of applications—from 1,099 companies in 2003. NASA staff note that the agency receives far more high quality applications than it can fund, as evidenced by the scores garnered by applicants. Agency staff also note that about a third of Phase I awards go to firms that have not previously won awards from NASA. NASA has made awards to firms in 48 out of 50 states,10 NASA staff believe that the decentralized nature of NASA, with numerous centers located often in   Interview with Paul Mexcur, Program Manager, November 21, 2003.   Ibid. 10  North and South Dakota are the two states that have not received a NASA SBIR grant.

PROGRAM MANAGEMENT 111 areas outside the main U.S. research hubs, provides sufficient geographical di- versity for the program.11 While there are no formal benchmarks or metrics for agency outreach, NASA officials point to the influx of new firms and the continuing 8:1 applica- tions to award ratio as evidence that new companies have little difficulty finding out about the NASA SBIR program, and that a significant increase in outreach is not needed. 5.3.3  Submission, Evaluation, and Selection12 5.3.3.1  Proposal Submission Small businesses submit their proposals to NASA through a sophisticated multipurpose online system called Electronic Handbooks and E-Submission (EHB). NASA describes this Electronic Handbook as a “set of Internet-based tools that support the paperless documentation and management of complex distributed processes,” including the SBIR program.13 EHB helps guide users through the program and provides real-time, online, paperless documentation and process management. 5.3.3.2  Evaluation Criteria Once submitted electronically via the Electronic Handbook, NASA screens the proposals to ensure that they are complete before sending them to the NASA center that “owns” the relevant topic for technical review. Evaluation is based on: • Scientific/Technical Merit and Feasibility. • Experience, Qualifications and Facilities. • Effectiveness of the Proposed Work Plan. • Commercial Merit and Feasibility. 5.3.3.3  Peer Review Panels—Membership, Selection, Qualifications Peer review in the NASA SBIR program is done internally by NASA tech- nologists.14 NASA staff scientists are used as technical experts. External re- viewers are also used for many Phase II applications, primarily for reviewing 11  Interview with Paul Mexcur, Program Manager, November 21, 2003. 12  This is based on an interview with Carl Ray, Executive Director and Paul Mexcur, Program Manager of the NASA SBIR/STTR Program. The interview was on November 10, 2003. 13  National Aeronautics and Space Administration, The Paperless Solution, p.1, accessed <http:// ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20020062196_2002101422.pdf>. 14  Interview with Paul Mexcur, Program Manager, November 21, 2003.

112 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION commercial potential. These reviewers include the Research Triangle Institute (RTI), independent experts, retired senior executives, and others. The total cost of all Phase II external reviews is $80,000 to $100,000 annually. All Phase II applications are subjected to a peer review by at least one non-NASA person. It should be noted that some award recipients have questioned the value of the external review, on the grounds that these staff may not be sufficiently familiar with the NASA programs—alignment with which will in the end be decisive for any proposal. To enhance fairness in the selection process, NASA screens for multiple applications from a single company, as well as duplication of proposals or tech- nologies.15 NASA will not accept more than ten Phase I applications from the same firm in a given year, and will make not more than five Phase I awards to a single firm.16 Program management also makes sure that awards are spread ap- propriately across different technical areas and Mission Directorates. Problems are resolved by adjusting the rankings of proposals. 5.3.3.4  Phase I Evaluation and Selection Submitted Phase I proposals must be complete, as evaluators are not expected to seek additional information. Evaluations are performed by NASA scientists and engineers at the center(s) identified in the solicitation as responsible for the applicable subtopic. In some cases, qualified experts from outside NASA (from industry, academia, and other government agencies) may provide additional ad- vice. Applicants should not assume that evaluators are acquainted with the firm, its key individuals, or with any experiments or other information. Any pertinent references or publications should be noted in the technical proposal. NASA gives primary consideration to (a) the proposal’s scientific and tech- nical merit and feasibility and (b) the proposal’s benefit to NASA. According to NASA, each proposal is judged and scored on its own merits using the factors described below: • Factor 1. Scientific/Technical Merit and Feasibility. The proposal is evaluated on whether it offers an innovative and feasible technical approach to the described NASA problem area. Proposals must demonstrate relevance to the subtopic. Specific objectives, approaches, and plans for developing and verify- ing the innovation must demonstrate a clear understanding of the problem and the current state of the art. The applicant must also define risks involved in the proposal. • Factor 2. Experience, Qualifications and Facilities. The technical capabilities and experience of the Principal Investigator or project manager, key 15  Based on interview with Paul Mexcur, Program Manager, November 21, 2003. 16  NASA sdoliciation Web site. Accessed at <http://sbir.gsfc.nasa.gov/SBIR/solicit.htm>.

PROGRAM MANAGEMENT 113 personnel, staff, consultants and subcontractors (if any), are evaluated for consis- tency with the research effort and their degree of commitment and availability. The necessary instrumentation or facilities must be shown to be adequate. The proposal should specify if the project will rely on any external sources, such as government furnished equipment or facilities. • Factor 3. Effectiveness of the Proposed Work Plan. The work plan is reviewed for its comprehensiveness, effective use of available resources, cost management, and proposed schedule for meeting Phase I objectives. The methods proposed for achieving each objective or task must be described in detail. • Factor 4. Commercial Merit and Feasibility. The proposal is evalu- ated for any potential commercial applications in the private sector or for use by the federal government. Factors 1, 2, and 3 are scored numerically with Factor 1 worth 50 percent and Factors 2 and 3 each worth 25 percent. The sum of the scores for Factors 1, 2, and 3 comprise the Technical Merit score. The score for Commercial Merit is in the form of an adjectival rating (Excellent, Very Good, Average, Below Average, Poor). For Phase I proposals, Technical Merit carries more weight than Commercial Merit. Each center ranks the proposals recommended for award relative to all other proposals recommended by that center. Center rankings are then forwarded to the Program Management Office for analysis, and are then presented to the Source Selection Official and Mission Directorate Representatives. Final selection decisions take into consideration the center rankings as well as overall NASA priorities, program balance, and available funding. Recommen- dations and relative rankings developed by the centers do not guarantee selection for award. The Source Selection Official has the final authority for choosing the specific proposals for contract negotiation.17 5.3.3.5  Phase II Evaluation and Selection The Phase II evaluation process is similar to the Phase I process. NASA plans to select for award those proposals offering the best value to the agency. Each proposal is reviewed by NASA scientists and engineers, and by qualified experts outside of NASA as needed per the factors identified below. Those proposals with high technical merit are reviewed for commercial merit. NASA uses a peer review panel to evaluate commercial merit. Panel membership includes non-NASA experts in business development and technol- ogy commercialization. 17 The list of proposed selections is posted on the NASA SBIR/STTR Homepage. Accessed at <http://sbir.nasa.gov>. Additionally, all firms receive a formal notification letter.

114 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION • Factors 1-3.  The first three selection factors for Phase II are essentially identical to those for Phase I, except that Factor 1 (Scientific/Technical Merit and Feasibility) also addresses the extent to which Phase I objectives were achieved, and the impact of Phase I results on Phase II feasibility. • Factor 4. Commercial Potential and Feasibility.  NASA assesses the proposed commercialization plan in terms of its credibility, objectivity, reason- ableness of key assumptions and awareness of key risk areas and critical business vulnerabilities, as applicable to the following factors: Commercial potential of the technology. Commercial intent of the applicant. Capability of the applicant to realize commercialization. Factors 1, 2, and 3 are scored numerically with Factor 1 worth 50 percent and Factors 2 and 3 each worth 25 percent. The sum of the scores for Factors 1, 2, and 3 comprise the Technical Merit score. Proposals receiving numerical scores of 85 percent or higher are evaluated and rated for their commercial potential using the criteria listed in Factor 4, and by applying the same adjectival ratings described above for Phase I. For Phase II proposals, commercial merit is a critical factor. This sequential evaluation (of technical merit followed by commercialization) is an interesting innovation, allowing NASA to focus scarce resources for assessing commercial- ization only on the most promising application. Once again, each center makes recommendations for awards among those proposals that it evaluates. The center recommendations are forwarded to the Program Management Office for analysis and presented to the Source Selection Official and Mission Directorate Representatives. Final selection decisions con- sider the center recommendations, overall NASA priorities, program balance and available funding, as well as any other evaluations or assessments (particularly pertaining to commercial potential). Recommendations provided by the centers do not guarantee selection for award. The Source Selection Official has the final authority for choosing the specific proposals for contract negotiations. 5.3.4  Funding “Gaps” Funding gaps can be found between the end of financial support under Phase I and the start of support under Phase II. About two thirds of respondents to the NRC Phase II Survey reported a gap between Phase I and Phase II fund- ing at NASA; the average length of the gap was 6 months. Only 3 percent of respondents reported a gap of one or more years.18 NASA cites the efficiency of its Electronic Handbook in minimizing this gap. Nevertheless, several firms interviewed for case studies indicated that the funding 18  NRC Phase II Survey, Question 26.

PROGRAM MANAGEMENT 115 gap remains an important issue, especially for smaller, less developed firms. In addition, three-quarters of NRC Phase II Survey respondents who experienced a gap responded by stopping work on the project.19 It is unclear whether NASA has benchmarked its own funding gap against those of other agencies. NASA does not appear to have adopted any of the “gap-reducing” initiatives initiated at other agencies. These include: • Development of a Phase I “option” that provides bridge funding for selected projects at the end of Phase I (DoD). • Simultaneous application for Phase I and Phase II (the NIH Fast Track). • Work at risk (NIH).20 • Expedited processing of Phase II awards (DoE). Several of the small firms interviewed for this study suggested that it would make sense to find a more standardized way to operate contracts, given that both the funding amounts and time to delivery are essentially fixed. 5.3.5  Other Aspects of Award Selection 5.3.5.1  Reporting Phase II typically requires quarterly reports, submitted electronically. In ad- dition, NASA Phase II awards usually require a prototype as a deliverable. These reports are used by the COTR and the COTR’s management to help identify potential for Phase III.21 The NASA contracting officer is supported by the Contract Officer’s Tech- nical Representative (COTR). The COTR is the firm’s primary contact within NASA on the contract’s technology focus and objectives. Given that one primary goal of the SBIR program at NASA is the eventual infusion of the firm’s tech- nology into NASA’s programs and missions, the interaction with technologists within NASA is critically important, and the COTR is both the link and the fa- cilitator for such interaction. 5.3.5.2  Resubmission Procedures and Outcomes Resubmissions of rejected applications are not allowed.22 However, NASA can select an applicant at any point in time. Although rare, a previous application 19  NRC Phase II Survey, Question 28. 20  At NIH, firms scoring well inside the likely Payline may decide to continue work before a Phase II award is made. The Phase II award covers up to three months work of pre-contract expenses incurred for Phase II work. 21  Based on interview with Paul Mexcur, Program Manager, November 21, 2003. 22  Based on interview with Paul Mexcur, Program Manager, November 21, 2003.

116 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION BOX 5-1 Tracking SBIR Technology Progress NASA has developed mechanisms to track and make available information on the progress of the technology it funds. Previously, NASA simply made note of the technology at the beginning and at the end of the project. NASA now main- tains a database on each project and updates each project’s specifications on a continuing basis. As the technology in question matures through the phases of SBIR, the program manager regularly updates a progress chart, called a “Quad” chart (see Figure B-5-1). This chart includes items such as expected outcomes from the project, commercial potential and intended utilization. The data provided by the chart also serve internal marketing purposes, with the ultimate goal being to move projects from Phase II to Phase III. Technology Development for a Low-Cost Deployable Lidar Telescope Project No. 258-70-117 Dr. Lee Peterson, University of Colorado, Boulder Objectives Sub-System Develop and validate precision Experiment deployment technology for low-cost, optical UV Lidar telescopes. Develop new optical precision deployment technology. 50:1 improvement in structural performance. Minimize need for active optical figure control. System Concept Validate technology in a sub-system test. Single petal and mirror segment. Use integrated structural-optical models to extrapolate to full system flight behavior. Technical Elements Schedule and Deliverables Segmented mirror with a deployed 03-04 Component development depth reaction structure. and experiment design. New components with sub-micron 04-05 Component experiments deployment repeatability and and models complete. microdynamic stability. 05-06 Sub-system experiments Sub-system deployment and and models complete. microdynamic experiments on Final Report: March 2006. single-petal prototype hardware. Innovative virtual boundary condition Co-Is/Partners sub-system test methodology. Co-I: Dr. Syed Ismail, NASA LaRC. Component-, sub-system-, and system- Co-I: Dr. Mark Lake, Consultant. level models updated and validated Co-I: Dr. Jason Hinkle, CU. including uncertainty tolerances. Science Advisor: Dr. Ed Browell, NASA LaRC. Impact Technical Advisor: Tim Collins, NASA LaRC. 4-10 times improvement in sensitivity. Partner: Dr. Ed Friedman, Boeing-SVS. Delta-II diameter mirror in a Pegasus-size package. Enables UV, VIS, and IR Lidar/DIAL systems for O3, H2O, CO2, aerosol, and cloud measurements from space. Instruments Passive Optical FIGURE B-5-1  Technology Development for a low-cost deployable Lidar telescope. SOURCE: National Aeronautics and Space Administration.

PROGRAM MANAGEMENT 117 may be reconsidered because a particular technology became a higher priority for NASA.23 In these cases, the relevant center goes back to the firm to verify that it still wants to do the project and can do what was proposed. 5.3.5.3  Other Agency Funding In recent years, NASA has begun to encourage centers to match SBIR funding for recommended high-quality proposals that might otherwise remain unfunded. NASA SBIR’s program tells its centers that the program will put up half the funds for a project if a center can come up with the other half of the money. While only three or four proposals have been funded in this manner, this innovative effort to leverage resources and to ensure that SBIR projects are high priority for the agency deserves more attention. 5.3.5.4  Debriefing of Unsuccessful Applicants After Phase I and Phase II selection decisions have been announced, debrief- ings for unsuccessful proposals are available to the applicant’s corporate official or designee via email. Debriefings are not opportunities to reopen selection de- cisions. Instead, they are intended to acquaint the applicant with the perceived strengths and weaknesses of their proposal and to identify constructive options for the applicant. Debriefings do not disclose the identity of the proposal evaluators, proposal scores, or the content of, or comparisons with, other proposals. Perhaps as a re- sult, several interviewees from case studies indicated that they received little of value from the debriefings. It is unclear why NASA takes such a restrictive view of the information disclosed in its debriefings. For Phase I proposals, debriefings are automatically emailed to the desig- nated business official within 60 days of the selection announcement. Unsuccess- ful Phase II applicants are contacted by the appropriate field center for debriefing within 60 days of the selection announcement. 5.4  BEYOND PHASE II—THE TRANSITION TO PHASE III As NASA wrestles with the problem of increasing the take-up of SBIR tech- nologies within the agency, it will be necessary to address a number of different related issues. 23  Historically it has happened just two times in eight years.

118 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION 5.4.1  No Phase III Transition Support NASA says that it encourages Phase II awardees to continue on to Phase III.24 However, NASA does not appear to have programs in place to support this chal- lenging transition. Indeed, small businesses participating in the SBIR program across the federal government find the transition to Phase III difficult, not least because of NASA’s changing mission priorities.25 NASA does not have a Phase IIB incentive linked to third-party financing, like that of the NSF SBIR program.26 5.4.2  Training Programs for Agency Phase I and Phase II Awardees Other agencies have also tried to address the Phase II transition by improving the commercial aptitude of funded firms, through a range of training and support programs. Such programs include: • The Navy Technology Assistance Program, a training program run by a third party, and culminating in a widely attended forum connecting firms to pos- sible funders. • The NIH CAP program, also with third-party training (a different pro- vider), and a forum focused more on attracting funding from venture capital. • The DoE technology support programs, which focus on developing individual marketing and development plans for firms. NASA does not provide formal commercialization training such as that provided at other SBIR agencies. NASA cites constraining factors such as funding, avail- ability of personnel, and widely dispersed geographic locations. NASA has variously employed incubators, Technology Utilization Centers, a university grant program, and state science and technology centers to help SBIR entrepreneurs improve their business skills to prepare for commercialization. 27 However, there is little evidence that these projects have generated much in the way of positive returns, and a number are now apparently defunct. There are important structural obstacles to the development of commercially 24  Based on interview with Paul Mexcur, Program Manager, November 21, 2003. 25  The issue of changing mission priorities at was raised by NASA at the NRC 2005 conference on the Phase III in Transition Conference. See National Research Council, SBIR and the Phase III Challenge of Commercialization, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2007. 26  The NSF SBIR program adds Phase IIB grant supplements following a Phase II grant condi- tional on attraction of third-party financing. For a description of the NSF Phase IIB program, see National Research Council, An Assessment of the SBIR Program at the National Science Foundation, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2008. Unlike the NSF Phase IIB supplements, the Phase IIe program at NASA focuses on Phase II and not the transition to Phase III. 27  Based on interview with Carl Ray, Executive Director, November 14, 2003.

PROGRAM MANAGEMENT 119 successful products from the NASA SBIR program. A program that is focused on building specific one-off solutions to the unique challenges and rigors of space flight or aeronautical testing is not likely to discover technologies with wider commercial potential and appeal.28 5.4.3  Take-up Within the Agency Phase III can of course also mean take-up within the agency—the granting of continuation contracts, which under the rules governing SBIR have a privileged contracting status, in that a Phase II award satisfies government contracting rules that require competition before contracts can be awarded. Thus Phase III can be sole source. However, this is little evidence that this potential advantage for SBIR contractors has played much part at NASA—according to interviews with Mission Directorate staff.29 More significantly, NASA does not track Phase III awards across the agency on an ongoing basis. As a result, there is little evidence on which to support management initiatives in this area. 5.5  PROGRAM EVALUATION NASA appears committed to improving its evaluation and assessment pro- gram, and to utilizing results from that process to help guide program manage- ment. In interviews, both the Program Director and the Mission Directorate liaison officials stressed that this was a priority issue for NASA. 5.5.1  The Challenge of Evaluation Evaluating NASA’s SBIR program presents major challenges. The agency itself has utilized SBIR for different strategic objectives over time, and the rela- tive prioritization of commercialization and agency mission has also changed over time. Moreover, technology often unfolds in complex ways, taking different paths. As a result, available metrics in for tracking program successes may miss impor- tant outcomes.30 For example, measures that quantify knowledge effects, such as patents, trademarks, and licenses usually occur well after the end of the SBIR contract period, and may miss the transfer of knowledge through less formal mechanisms (e.g. a shift of principal investigator to a new company). Moreover, 28  The myth that Teflon technology, used to coat cooking utensils, is a spin-off from the space program is widespread and long standing. In fact, Teflon was invented by DuPont in 1938. Dr. M. Gregoire was granted a patent on the Teflon coating of a steel pan in 1954. 29  Interview with Jason Cruzer, December 7, 2007. 30  Interview with Carl Ray and Paul Mexcur, November 10, 2003.

120 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION for NASA, successful infusion and/or commercialization is expected to occur seven years or later after a Phase I award.31 Well aware that its metrics do not capture important benefits and, thus, un- derestimate SBIR benefits, NASA’s SBIR program managers recognize the need to improve assessment and data collection. Currently, for example, NASA’s Phase III metrics do not include subcontracts, bids for other procurement opportunities, use of prototypes through other mechanisms, or future developments of SBIR- supported technology. NASA does, however, produce an SBIR Management Report that invites feedback from managers.32 NASA also conducts an annual self-assessment on the year’s solicitation process and outcomes. The latter includes a statistical analysis of how each subtopic performed; about one-third of the solicitation subtopics are changed each year. Before each solicitation, NASA does a “lessons learned” exercise. 5.5.2  Resource Constraints What NASA spends on evaluation and assessment is not easily determined because there is no line item in the budget for evaluation. Only a small part of the program’s $3.8 million administrative budget is devoted to evaluation and assessment, according to the Program Executive. While NASA tracks the progress of each project during the contract phase using the quad chart and other reports (see Box 5-1), information on outcomes is less robust.33 This means that NASA’s SBIR metrics likely underestimate the program’s total benefits, both to NASA as well as to the private sector. Partly, this is because NASA has not implemented a tracking program such as that developed at DoD. Such a program requires that companies update out- comes data for all previous SBIR awards (at all agencies) whenever they apply for a new award from DoD. NASA may be able to piggy-back on the DoD pro- gram at minimal cost—and at relatively low resource cost to companies, as there is considerable overlap between DoD and NASA firms. In addition, all firms applying at DoD already provide the requested information about all projects, including those funded by other agencies such as NASA. While NASA might adopt the DoD model of long term data collection through the Company Commercialization Reports database (CCR), NASA’s SBIR program management has not had sufficient time or the resources nec- 31  Interview with Jack Yadvish, research team leader for Commercial Metrics, March 7, 2005. 32  Based on interview with Paul Mexcur, Program Manager, November 21, 2003. 33 NASA tracks changes in the technology over the course of the project, distinguishing between the times required for a supported technology to move from Phase II to Phase III. (Interview with Carl Ray and Paul Mexcur, November 10, 2003.) Currently, NASA data—unlike that of DoD—does not distinguish between Phase III research support and Phase III procurement.

PROGRAM MANAGEMENT 121 essary to follow the knowledge and economic effects of its SBIR-supported technology. 5.5.3  Phase III At the agency level, there appears to be no formal tracking of NASA pro- curement from SBIR winners. While some field center SBIR programs attempt to track Phase III activity by SBIR contractors, there appears to be no common, standard approach, and some centers perform no tracking. Also, there is no shared definition of the investment threshold for infusion activity or any agreement on the preferred data sources for such activity. Some field center SBIR program of- fices use agency procurement data. Others rely on self-declarations by the small businesses. The absence of reliable data on Phase III—especially on Phase III contracts within NASA, is a matter of concern both to the SBIR program office and the Mission Directorates. It is hard to manage a program effectively if there are no reliable indicators for the most important dimensions of success. Resource limitations have played a major role in this area, as have concerns about the imposition of additional reporting requirements on companies or in- deed within the agency. However, addressing this problem should be among the agency’s most pressing priorities as SBIR becomes a more central component in the agency’s overall research and operations strategy. 5.5.4  Assessing Outreach Outreach activities at NASA’s SBIR Level 1 are not formally evaluated. In- reach activities are carefully evaluated. For the NASBO program (described in Section 5.6.1) evaluation is informal at the Southern California pilot chapter, and at the SBIR Program Executive level. Information does exist on the distribution and frequency-of-use of NASA’s key outreach/inreach publications: • Spin-off (annual print and CD publication)—in 2004, 35,000 print or CD units were distributed at conferences, expos, and other events. • Technology Innovation (print quarterly)—15,000 copies were circulated in 2004 at conferences, expos and other events. Downloads from NASA Web sites are not tracked. • Technology Briefs (print and electronic monthly)—about 500,000 sub- scribers were mailed the briefs in 2004. Downloads from NASA Web sites are not tracked. • Success Stories (a NASA Web site record of 1,262 technologies)—Web site hits are not tracked. • TechFinder (electronic portal; a technology transfer database)—the most

122 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION recent data shows that during November, 2004 TechFinder averaged 4,000 hits daily, and 130 daily visits in which a specific technology description was viewed. However, this covers all NASA technology, not just SBIR. No other outcome metrics exist for outreach, and these data do not fully address the core question: are potential buyers of SBIR-funded NASA technologies aware of what is available? Do they use this information effectively? Could this outreach be improved? 5.5.5  Assessing Alignment with Agency Mission Given a lack of funding, NASA’s Commercial Technology Division does not anticipate an update to its 2000 publication, Commercial Metrics. Based on an extended survey of SBIR awardees, this 2000 study found that during 1983-1996 about 15 percent of the 1,739 NASA SBIR Phase II awardees who responded to the NASA survey34 generated technologies that infused into NASA or other federal programs via Phase III funding.35 Thirty-one percent commercialized in the private sector.36 NASA defines commercialization as the sale of NASA technology-derived products or services that resulted in actual revenues for the SBIR firm. The Com- mercial Metrics survey noted “a minimum of 612 products and services” com- mercialized from 1983-1996 that generated at least $2.28 billion of cumulative revenues in nonfederal markets. NASA’s total SBIR investment for that same period was $1.11 billion. Of course, these data cover a funding period that ended more than 10 years ago. Although agency budget constraints prevent a comprehensive update of this survey, more recent data shows that at least 267 NASA Phase III contracts have resulted from SBIR technology. Those contracts totaled $157,769,228. Assessing NASBO The NASA Alliance for Small Business Opportunity (NASBO) supports in- cubators to improve NASA SBIR commercialization and prepare potential firms as investor ready companies. The evaluation of NASBO has also become part of NASA’s strategic discussion about revitalization of its Innovative Technology Transfer Program, as “spin-in” activity (supporting Mission Directorate program 34  Access the NASA SBIR Commercial Metric Survey at <http:sbir.nasa.gov/SBIR/survey.html>. Eighty-four percent of eligible firms responded to the NASA survey. NASA claims that many SBIR firms that did not respond had multiple Phase III contracts and/or commercial success. 35  NASA uses the term “Phase III funding” here means contractual or other monies awarded to a SBIR project for federal agency use of the subject technology after expiration of a SBIR Phase II award. 36  The extent of infusion/commercialization overlap is unknown for these data.

PROGRAM MANAGEMENT 123 work) has been added to complement “spin-out” activity, and both SBIR and technology transfer work fall under NASA’s Innovative Partnership Program. This discussion was triggered by a 2003-2004 external review of NASA technology transfer practices, conducted by the National Academy of Public Administration (NAPA) at the request of the Federal Office of Management and Budget.37 That review cited NASA’s need for a comprehensive strategy for identifying technology needs and commercialization opportunities, and called for a reform of NASA technology transfer practice to better balance spin-in and spin-out activity. 5.5.6  SBIR Success Stories NASA publishes “success stories” to document and publicize information on positive SBIR outcomes. These success stories represent one way of detailing outcomes from SBIR-funded projects. As of 2006, NASA had documented 510 success stories and 287 Phase II contracts. Success is defined as contributing tech- nology that helps NASA accomplish its missions and/or producing commercial value in the private sector. Both forms of success are documented on the NASA SBIR Web site.38 The site allows users to search by project, state, NASA center, and year. Each success story contains specific information on the innovation, the company’s commercial- ization activities, the project’s key accomplishments, applications in government and science, and an image of the product. Table 5-1 shows the number of success stories produced by each NASA cen- ter, and provides one indication that “success” varies considerably by centers and regions. NASA centers with fewer success stories include Stennis, Dryden, Ames, Kennedy, and Langley. By comparison, Johnson, the Jet Propulsion Laboratory (JPL), Marshall and Goddard have a higher proportion of successes. The Glenn Research Center accounts for nearly 30 percent of all success stories. It should be noted that success stories are self-reported by firms and centers, and some centers are likely to be more aggressive in reporting success stories than others. This is not therefore a reliable quantitative metric for assessing center activities. Why do such differences exist? SBIR budget shares and differences in core areas of technology explain some of the variation. Differences in success rates may also be due to specialized regional infrastructure focused on commercial- ization—such as the Regional Technology Transfer Centers (RTTC), NASA incubators, and various state and local partnerships—though this remains to be studied. Although the “success stories” approach is illustrative, it is of limited use. For example it is not clear exactly who determines which projects qualify as a 37  Access information on the NAPA study at <http://www.napawash.org/pc_management_­studies/ ongoing_nasattf.html>. 38  Accessed at <http://sbir.gsfc.nasa.gov/sbirweb/successes/Success_Story_Search.jsp>.

124 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION TABLE 5-1  NASA Success Stories by Center Since the Program’s Beginning Center Number of Success Stories Percent of Success Stories Ames Research Center 19 3.90 Dryden Flight Research Center 8 1.7 Glenn Research Center 137 28.3 Goddard Space Flight Center 86 17.8 Jet Propulsion Laboratory 58 12 Johnson Space Center 56 11.6 Kennedy Space Center 20 4.1 Langley Research Center 29 6 Marshall Space Flight Center 64 13.2 Stennis Space Center 7 1.4 TOTAL 484 100 SOURCE: National Aeronautics and Space Administration. success story, and on what basis. The stories themselves are essentially anecdotes, though sometimes persuasive ones. And while the stories are searchable along several dimensions, “value to the agency” is not one of them. Thus the success stories should be viewed as a useful adjunct to more data driven assessments. They could also be improved by deploying a more systematic and transparent approach to their collection. 5.5.7  Evaluation and Assessment: Conclusions NASA is aware of the challenge it faces in developing the data sources and analytic tools needed to help manage the SBIR program effectively. The 2002 Commercial Metrics report began to lay the groundwork for more definitive eval- uation of the SBIR program, but little has been built on that basis since then. Discussions with NASA staff indicate that they are well aware of deficien- cies in this area, and are eager to correct them. Tools that will support program management effectively should therefore be a high priority. The survey underly- ing the 2000 Commercialization Metrics report can be viewed only as a very preliminary step toward the gathering of useful data.39 It is focused on company level outcomes rather than on projects. It is also important to remember that summary statistics on market sales and NASA funding do not capture the overall return—economic and otherwise—to the NASA SBIR investments. On the one hand, sales alone ignore other costs: the $2.3 billion in sales generated by SBIR firms also involved financial as well as opportunity costs for the firms and for NASA far beyond NASA’s $1.1 bil- lion investment in SBIR funding. On the other hand, the calculations did not 39  Access the NASA SBIR Commercial Metric Survey at <http:sbir.nasa.gov/SBIR/survey.html>.

PROGRAM MANAGEMENT 125 include the support SBIR projects provide to NASA missions, the technology spillover benefits produced, or the knowledge spillovers that augment NASA’s own ability to solve technology problems. So better data is important, but rigid application of conclusions drawn from what are inevitably limited data should also be avoided. 5.6  COMMERCIALIZATION SUPPORT This section examines the infusion and commercialization assistance avail- able through NASA’s SBIR program at the national and field center levels. 40 Unlike some other agencies (e.g., NIH, parts of DoD, DoE), NASA does not provide a formal training program. However, it does offer support for com- mercialization at different levels of SBIR management. NASA SBIR senior staffers understand that the technological excellence of SBIR contractors is not always matched by entrepreneurial skill. As a result, the SBIR Program Executive has focused on “stakeholdership”—increased SBIR program interface with prospective customers and investors. The three main agency-wide efforts underway are external outreach activities aimed at “marketing” SBIR funded technologies in the commercial marketplace and at other agencies, internal inreach activities (sometimes called “infusion” ac- tivities) focused on increasing the adoption of SBIR-funded technologies within NASA, and the NASA Alliance for Small Business Opportunity (NASBO), which leverages internal and external resources to mature SBIR technologies for cus- tomers. A fourth effort, focused on “incentivizing” Mission Directorate program offices to infuse SBIR technologies, is under development. Most recently, since 2006 the focus has shifted decisively away from commercialization and external marketing toward improving the uptake of SBIR-funded technologies within NASA. At the same time, the NASBO program appears to be of declining inter- est, and is currently under management review. NASA does not formally market its SBIR-derived technologies. However, because the agency understands the difficulties facing companies as they ap- proach Phase III, it has developed a range of mechanisms through which to publicize the technologies developed using SBIR (and other NASA R&D activi- ties). As noted earlier, these include Spin-off, an annual publication; Technology Innovation, a quarterly publication; Technology Briefs, released monthly; Success Stories; TechFinder, NASA’s electronic portal and database, and other materials such as a 2003 DVD portfolio of NASA SBIR projects. All materials are provided free to potential technology customers and end- users on a subscription basis, as handouts at selected technical and investment community events, and online at NASA’s Web site. Senior SBIR staffers see such 40  At NASA, the effort to link SBIR technology development with mission program utilization is termed “infusion,” defined as a Phase III occurrence at the end of the SBIR Phase I—Phase II program.

126 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION TABLE 5-2  Total NASA Phase I and Phase II Awards, 1983-2001 NASA Center Number of Awards Percent to Home State ARC, DFRC, JPL (California) 575 38 GRC (Ohio) 328 8 GSFC, LaRC (Maryland, Virginia, DC) 670 10 JSC (Texas) 310 17 KSC (Florida) 104 6 MSFC, SSC (Alabama, Mississippi) 402 19 TOTAL U.S. 2,389 19 SOURCE: National Aeronautics and Space Administration, Commercial Metrics database. external marketing as commercialization assistance, which brand technologies with the NASA imprimatur, as being of value to SBIR contractors. At the center level, NASA centers with specific technology needs put out solicitations and evaluate subsequent SBIR applications. Because NASA centers generally lack knowledge of business and commercialization, they have relied on external organizations—such as Regional Technology Transfer Centers (RTTC) or state advanced technology programs—to facilitate commercialization. Assisting commercialization effectively requires intimate knowledge of a region’s companies and industries and frequent contact between the intermedi- ary and the firm. Because roughly 80 percent of most centers’ awards go to firms in other states (see Table 5-2), the centers’ management of commercialization involves very little face-to-face contact and the centers often do not have a deep understanding of the regions in which their awardees operate. One result is that NASA centers are not heavily involved in SBIR commercialization. 5.6.1  NASBO and Technology Incubators NASA’s commitment to commercialization has been strong from the be- ginning, as evidenced by its support of an extensive regional technology trans- fer network. The network includes the National Technology Transfer Center (NTTC), Innovation Partnership Program offices in each of ten NASA Centers, six Regional Technology Transfer Centers (RTTC) (currently being phased out), seven regional NASA incubators, the Research Triangle Institute (RTI), and the NASA Alliance for Small Business Opportunities (NASBO). NASA has in the past co-invested with regional stakeholders in nine business incubators, each located near one of the agency’s ten field centers, in conjunc- tion with NASA’s Innovative Partnership Program. These incubators offer physi- cal resources and limited technical assistance from management consultants to resident small businesses to promote successful NASA infusion or private-sector commercialization of their technologies. NASBO’s first chapter, the “NASA Commercialization Center” (near the Jet

PROGRAM MANAGEMENT 127 Propulsion Laboratory) is one such incubator. It is a collaboration with the Cali- fornia State Polytechnic University—Pomona and Southern California Edison, a public utility that invests heavily in advanced technology entrepreneurship. A second NASBO chapter (near the Johnson Space Center in Texas) is a nonincubator collaboration with the Technology Tree Group (TTG), an angel investor network. It was created using a nonfunded Space Act Agreement with TTG. TTG invests directly in SBIR firms of its choosing to meet the SBIR firm’s strategic and operational needs, leveraging TTG’s close ties to the National Seed and Venture Capital Fund Association and the Angel Capital Association. Unlike the NASA Commercialization Center, with its regional information technology focus, the TTG chapter is national in scope and has a medical technology focus. Here, assistance to SBIR firms includes market and technology assessments, and help in readying SBIR client firms for presentations at investor events, including the “World’s Best Technology Show” (April, 2005). According to current TTG management, this effort is now largely independent of NASA, and NASBO has been of declining significance, largely because resources are lacking. A third chapter is planned in collaboration with the Georgia Institute of Technology (Georgia Tech), with which NASA has multiple mission-driven partnerships. This chapter could link seven incubators in southern five states to cull NASA and other SBIR Phase II inventories for technologies that respond to defined needs in NASA and DoD acquisition programs. At present, chapter partners are still examining the feasibility of a Southeastern NASBO chapter, due to the small number of SBIR awardees in these states. Each NASBO incubation chapter initiative is tasked with following NASA “ideation preparation guidelines.”41 These include two principal activities: a competing technologies contrast grid, and a market/customer availability as- sessment. In the first exercise, a grid is populated with significant technical- performance features on one axis, and direct plus indirect competing technologies on the other axis. In the second exercise, the grid is populated with top products under consideration in varied market segments on one axis, and dominant prod- uct features plus benefits on the other axis (these include product cost, market share, technology used, and strength/weakness points). Both “spin-in” and “spin- out” technologies are encouraged. These guidelines were designed to generate customer-focused discussions by SBIR firms, and to orient NASBO incubator tenants to the framework of an infusion/commercialization culture. Regarding outcome metrics, NASBO incubator chapters are asked to track the results of these ideation activities, as well as all post-SBIR contractual activity by incuba- tor tenants. NASBO’s pilot chapter, the NASA Commercialization Center, has helped eight incubated SBIR firms since 2003. Three of these firms have sold products or services into federal markets. Two firms have yet to report infusion/commer- 41  National Aeronautics and Space Administration, Technology Innovation 11(4):36, 2004.

128 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION BOX 5-2 NASA Sponsored Business Incubators NASA sponsors nine small business incubators in different regions of the coun- try. Their purpose is to provide assistance in creating new businesses based on NASA technology. • Business Technology Development Center. • Emerging Technology Center. • Florida/NASA Business Incubation Center. • Hampton Roads Technology Incubator. • Lewis Incubator for Technology. • Mississippi Enterprise for Technology. • NASA Commercialization Center/California State Polytechnic University. • University of Houston/NASA Technology Commercialization Incubator. • NASA Illinois Commercialization Center. cialization revenues but are pursuing Phase III strategies. Three firms are still maturing technologies. The NASA/Technology Tree Group collaboration, created in 2003, results in “about five investments a year” made in NASA SBIR Phase II projects.42 TTG does not reveal details of these investments. Its SBIR clients are not yet marketing products based on NASA-funded technology. The NASA/Georgia Tech collaboration has no outcome information because the initiative is still in a formative stage. According to NASA’s SBIR Program Executive, NASA incubators are significantly underfunded in comparison with physical plant and tenant support needs. 5.6.2  Center-level Activities and Practices SBIR contracts are supervised with Mission Directorate assistance to ensure that SBIR awards are aligned with Mission Directorate needs. SBIR represents about 500 new contracts a year for NASA, representing almost half of the agen- cy’s total for for-profit contractors. As described earlier, this alignment begins at the topic/subtopic development phase, when taxonomic descriptions of SBIR projects are designed to parallel the taxonomies of Mission Directorate technology roadmaps. The process continues through the awards process. Finally, during Phase II, the program emphasizes infusion opportunities for awardees. 42  Interview with TTG Chief Executive Officer Michael Fitzgerald, March 7, 2005.

PROGRAM MANAGEMENT 129 During SBIR Phase I, the interface between SBIR awardees and NASA SBIR project monitors, or COTRs, is limited and technical. During Phase II, the selection process includes outside peer evaluation of infusion/commercialization potential, and internal evaluation of the same by Mission Directorate personnel and other staff. SBIR firms also submit Phase II quarterly progress reports. These reports must include statements on mission program application. In the second year of Phase II, NASA expects its SBIR firms to develop infusion/commercial- ization plans with assistance from the project monitor. Administratively, NASA SBIR activities for Levels 1 and 2—including in- fusion/commercialization issues—are discussed in a weekly teleconference of Level 1 and 2 principals, deputies and associates, and a monthly video confer- ence that includes NASA field center SBIR Program Managers. A semi-annual meeting of SBIR Level 1 and 2 personnel with all ten NASA field center SBIR Program Managers is also held. At this meeting, infusion/commercialization prac- tices are discussed, and the field centers make recommendations on prospective improvements to the program. 5.6.3  Access of SBIR Firms to Prime Contractors One key to successful Phase III contract activity is careful management of the relationship between SBIR firms and prime contractors performing platform or system work for federal customers. NASA’s SBIR infusion strategy focuses on developing “market pull” from Mission Directorate project offices for SBIR technologies. Prime contractors who create mission hardware and software are generally not involved directly or indirectly in the NASA SBIR process. Nor are NASA SBIR firms trained to interact with the primes or other subtier suppliers who are logical customers for SBIR technologies when they have contracts with the same project offices that generated the relevant SBIR topics/subtopics. There is growing awareness of this gap, especially at the field center level, and at least one center—Langley Research Center—has supported a pilot pro- gram to address it. However, the issue has not yet been formally raised at higher management levels for resolution. Prime contractors, such as Raytheon, Lockheed Martin, and Alliant-Thiokol, have now begun to advocate for SBIR technologies in Department of Defense programs. Similar, “market pull” from NASA primes for SBIR technologies can be expected to increase the commercialization of NASA’s SBIR-funded technologies.43 43  See presentations by senior representatives of Boeing, Raytheon and other Prime Contractors on market pull in National Research Council, SBIR and the Phase III Challenge of Commercialization, op. cit. See pages 75-94.

130 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION 5.7  SUPPORT FOR AGENCY MISSION ALIGNMENT44 With the appointment of NASA Administrator Michael D. Griffin in 2005, NASA made “spin-in”—the use of SBIR technology by NASA for mission needs—the SBIR program’s main priority. This shift to mission-purpose uses is being accompanied by other fundamental changes underway at NASA: • A new Moon-Mars mission, which is altering the relative position of NASA’s ten Centers; • A redesign of NASA’s Innovation Partnership Program (IPP) to reduce fragmentation and to emphasize spin-in45; • A new budget model that employs full-cost accounting and incorporates competition for funding among NASA centers; and • Movement of the Innovation Partnership Program to NASA Headquar- ters, giving it higher priority status within the agency. Interviews with Mission Directorate staff stress that this shift in emphasis toward much improved alignment between Mission Directorate needs and SBIR program operations has been driven by the needs of the Directorates as much as by a re-evaluation of roles. Specifically, the shifts that have taken place in NASA as a result of new missions and reordered priorities have stretched NASA resources very tightly. They have also encouraged some Mission Directorates—especially Space Operations—to shift program research dollars up the TRL readiness level indicators, away from basic research. As a result, SBIR dollars—which are insulated from this shift—have become an increasingly important source of low-TRL research funding. In some cases, they are the only available funds. There is no formal agency-wide SBIR policy at NASA to link SBIR contrac- tors with prospective infusion customers in mission project offices. However, eight field centers have evolved processes that task the SBIR project monitor, or SBIR subtopic manager, with brokering such relationships, with varying degrees of specificity. This process, at its simplest, entails identifying prospective Phase III inves- tors among NASA mission programs and projects during Phase I and II proposal evaluation. In more complex processes, SBIR project monitors are asked to pres- ent Phase II proposals to ranking committees with a record of discussions and meetings with Phase III prospects on the subject SBIR technology. In the latter case, such infusion information is shared with the SBIR winner by the SBIR project monitor. Like DoD, NASA contracts out most of its mission hardware/software work 44  “Inreach” refers to technology alignment efforts between SBIR technologies and potential NASA technology users. 45  As noted above, the SBIR program is now part of the IPP.

PROGRAM MANAGEMENT 131 to prime contractors, systems integrators, and sub-tier suppliers. Although these companies are large potential customers for SBIR technology, the NASA SBIR program currently has no systematic interface with these prospective custom- ers (with the exception of Langley’s pilot program linking SBIR awardees to primes). 5.8  THE REGIONAL DIMENSION NASA, with its ten field centers, has considerable potential for a regional technology orientation. This section examines how NASA deals with the regional aspects of its SBIR program. 5.8.1  Geography and the Regional Distribution of Awards Three main factors influence the regional distribution of SBIR awards. First, SBIR awards are highly concentrated geographically in a few locations. Second, NASA centers operate in a national market, typically awarding four of five SBIR grants to firms located in states other than the state in which the center is located. Third, technology innovation is highly concentrated geographically. One conclusion from these observations might be that the geography of SBIR awards simply reflects an efficient matching of NASA needs (reflected in each center’s awards) with the best small, high-tech companies distributed through- out the country. If this is true, outreach programs, coupled with information made available on NASA’s Web site, have created a successful SBIR program. NASA’s extensive documentation of SBIR success stories lends credence to this conclusion. Of course, the same regional pattern could also emerge from a poorly func- tioning SBIR program. Here, the same uneven geography could result from all regions doing a relatively ineffective job of utilizing NASA’s early-stage funding to support promising companies. We would still observe the nation’s top high- tech regions performing proportionately better. In such a case, reform could improve commercialization of and infusion from SBIR companies by ensuring that all regions do a better job locating and supporting the most innovative small businesses. Economic benefits could be significantly less if, for example, a NASA technology problem was not solved because the best match was not made, an outstanding Phase I did not make it to Phase II, or a technology was developed too slowly to achieve the best result. 5.8.2  Complex Management Challenges NASA’s description of its technology transfer management goal—to manage NASA technology transfer from “top-to-bottom, coast-to-coast, and cradle-to- grave”—indicates the scale of the challenge. “Top-to-bottom” refers to all levels

132 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION TABLE 5-3  NASA Phase I and Phase II Awards by State and Region, 1983-2003 Number Number of Total Region and NASA of Phase I Phase I Phase II Phase II Number of Total State Center Awards Dollars Awards Dollars Awards Dollars Far West 1,433 83,404 596 301,863 2,029 385,268 Alaska 2 100 1 500 3 600 Arizona 109 6,530 37 19,250 146 25,781 California ARC, 1,116 64,822 460 231,629 1,576 296,450 JPL, DFRC Hawaii 10 528 7 3,710 17 4,239 Idaho 3 209 0 0 3 209 Nevada 15 821 5 2,357 20 3,178 Oregon 66 3,931 32 16,764 98 20,695 Washington 112 6,463 54 27,653 166 34,116 Mid-Continent 607 134,946 242 124,657 849 160,829 Arkansas 5 259 1 594 6 854 Colorado 248 14,817 93 49,068 341 63,885 Iowa 10 557 6 2,635 16 3,193 Kansas 5 324 2 1,195 7 1,519 Missouri 10 658 4 2,286 14 2,944 Montana 16 1,029 5 2,536 21 3,565 Nebraska 4 220 2 732 6 952 New Mexico 77 4,709 24 12,503 101 17,211 North Dakota 1 70 0 0 1 70 Oklahoma 2 98,874 1 311 3 410 South Dakota 1 69 0 0 1 69 Texas JSC 188 11,008 88 44,434 276 55,442 Utah 37 2,161 14 7,366 51 9,527 Wyoming 3 191 2 997 5 1,188 Midwest 315 18,857 117 60,993 432 79,851 Illinois 44 2,494 16 7,843 60 10,337 Indiana 34 1,912 15 8,167 49 10,079 Michigan 68 4,011 27 13,655 95 17,666 Minnesota 52 3,217 20 10,596 72 13,814 Ohio GRC 117 7,223 39 20,732 156 27,955 New England 1,256 73,318 529 276,133 1,785 349,451 Connecticut 125 7,312 50 26,168 175 33,481 Maine 6 336 2 1,071 8 1,406 Massachusetts 724 41,883 292 150,328 1,016 192,211 New 94 5,612 47 25,103 141 30,715 Hampshire New Jersey 118 6,942 49 25,995 167 32,937 New York 177 10,434 85 45,080 262 55,514 Vermont 12 799 4 2,388 16 3,187

PROGRAM MANAGEMENT 133 TABLE 5-3  Continued Number Number of Total Region and NASA of Phase I Phase I Phase II Phase II Number of Total State Center Awards Dollars Awards Dollars Awards Dollars Mid-Atlantic 565 33,649 216 109,508 781 143,155 Delaware 7 450 2 1,051 9 1501 Maryland GFSC 202 11,720 69 36,091 271 47,811 Pennsylvania 119 6,777 49 22,457 168 29,234 Virginia LaRC 234 14,512 95 49,419 329 63,930 West Virginia 3 190 1 490 4 679 Southeast 466 26,617 186 95,265 652 121,884 Alabama MSFC 209 12,127 93 49,304 302 61,431 Florida KSC 116 6,737 44 21,619 160 28,357 Georgia 29 1,650 11 5,396 40 7,046 Kentucky 1 50 0 0 1 50 Louisiana 6 239 1 231 7 470 Mississippi SSC 20 1,014 7 3,867 27 4,881 North Carolina 26 1,525 8 4,198 34 5,723 South Carolina 1 69 0 0 1 69 Tennessee 58 3,206 22 10,650 80 13,857 SOURCE: National Aeronautics and Space Administration. of management; “coast-to-coast” highlights the importance of the ten NASA centers and NASA’s regional infrastructure, and “cradle-to-grave” points out the difficulty of managing SBIR from solicitation through Phase III. The management challenge is even more complex because the SBIR program is a component of NASA’s Innovation Partnership Program (IPP).46 Although often overlooked, each of IPP’s components interacts with state and local sci- ence and technology infrastructures. As suggested by Table 5-4, the number of distinct technology development programs within and related to NASA raises an obvious question: How do these combine to determine innovation supported by the SBIR program? 5.8.3  The Limits of the Traditional External Network While the rationale for NASA’s traditional external network—geographic proximity linking NASA technology where firms using the technology increase the rate of innovation in the private sector—was clear, the performance of this network remains in doubt. In the traditional network, specialized intermediaries, like the Regional 46  Until recently, SBIR was able to operate somewhat independently of IPP’s other programs.

134 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION TABLE 5-4  NASA and Non-NASA Programs NASA Non-NASA NASA Headquarters Web site (e.g., EHB) *Local University NTTC (National Technology Transfer Program) *Corporate R&D Labs NASA Center Research Lab/Branch *Nonprofit R&D Centers (e.g., hospital) NASA Center SBIR/STTR Program Management *STTI (State Science & Technology Office Institute) NASA Center Patent Office *State S&T Program *RTTC Program (6 RTTCs) *State DoD (e.g., early-stage funding) *Regional Affiliates of RTTCs (each of six regions) Angel Funders *Rural State SBIR Outreach (RSSO) Program Venture Capital Firms *NASBO Prime Contractor *NASA Incubator *NASA Ames Research Park *FAST (Federal and State Technology Partnerships) NOTE: The items with an asterisk are regionally oriented. SOURCE: National Aeronautics and Space Administration. Technology Transfer Centers, provided an essential component to bridge the gap between NASA technology and adoption of the technology by private industry. The intermediary brought knowledge of business, the investment community, regional economies, and the ability to provide a base for sustained interactions between sources of NASA technology and firms.47 Although the rationale was clear, the evidence supporting the network’s per- formance was weak and mostly anecdotal. For instance, despite documentation by NASA of numerous individual success stories, there were no compelling stud- ies substantiating the contribution of the regional infrastructure to innovation and technological change—either to spin-out or spin-in uses. Without a new evalua- tion methodology and much better data it remains impossible to know whether, for example, an RTTC or NASA incubator helped to create a success, much less differentiate a spin-out success from a spin-in or dual-purpose success. This dis- appointing outcome largely stems from a fragmented NASA infrastructure that has been documented by a recent report from the National Academy of Public Administration (NAPA) and reinforced by NRC interviews.48 The NAPA report asserted that the numerous components of NASA’s tech- nology transfer program have not functioned in a coordinated way.49 For exam- 47  NASBO has the potential to fill these needs, but does not yet have the required scale of operation and the need for cross-regional cooperation to meet this challenge. 48  Access information on the NAPA study at <http://www.napawash.org/pc_management_­studies/ ongoing_nasattf.html>. 49  Recent steps are being taken to create more coordination among RTTCs and NASA centers. SBIR/STTR headquarters has implemented monthly conference calls with NASA Centers and RTTC directors. In addition, headquarters has held several national network meetings.

PROGRAM MANAGEMENT 135 ple, even though SBIR is a component of the Innovation Partnership Program and reports to the same person, NAPA found that there was very little coordination among technology transfer programs. Its findings have been accepted by NASA and some of the recommended changes are already taking place; others are being discussed.50 NAPA concluded, in general, that the Innovation Partnership Program has been successful with administrative functions. However, the network is frag- mented, in part because roles and responsibilities are unclear. Several other factors also contributed to fragmentation: there was little direction from NASA headquarters; RTTCs reported to centers rather than to headquarters (see below); and NTTC operated under cooperative agreements rather than arrangements based on performance. An RTTC focused on its region is not well-matched with each NASA center’s use of SBIR because their mission-based topics and subtopics should draw applicants from across the country. Moreover, this disconnect between the re- gional focus of the RTTCs and the broad mission needs of NASA also means that technologies located in another region potentially beneficial to the RTTC’s region will be missed or substantially delayed. RTTCs have primarily worked independently, not as a NASA-wide and nationwide system. This should not be a surprise because each RTTC reports to a specific NASA center in their region and the RTTCs’ contracts state that 85 percent of their activities are to be allocated to their region’s companies and technologies. Moreover, Innovation Partnership Program personnel’s professional back- grounds are often not well matched with the requirements of technology trans- fer. For example, because of skill mix called for by the Innovation Partnership Program, staff frequently has difficulty communicating with NASA research- ers and potential external partners (e.g., universities and companies) about technologies. Corroborating the NAPA report, NRC interviews also found that NASA personnel had strongly held views that the RTTCs have performed poorly. NASA personnel gave various reasons for this disappointing performance, including: poor management; lack of clear objectives and metrics; too little emphasis on SBIR; and a geography that does not reflect the economics of regions. As a re- sult, much of the Innovation Partnership Program’s infrastructure is now being reorganized to focus on its mission of leveraging technology for NASA’s Mission Directorates, programs, and projects. Whatever structure emerges from the rethinking and reorganization of the Innovation Partnership Program (IPP), NASA continues to have a major stake in regions. The nation’s major sources of advanced technology are highly geo- graphically concentrated and are derived from the constantly evolving network 50  See Leonard Yarbrough, “Initial Responses to the NAPA Recommendations” (preliminary), November 16, 2004.

136 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION of regional innovation systems that form the nation’s innovation system. These dominant regions are both the primary sources (spin-in) and users (spin-out) of innovation and new technology. Spin-in and spin-out are really two sides of the same coin, with the primary difference being the direction of technology flow. Because SBIR is dual-purpose, NASA has a stake in an infrastructure that sup- ports both spin-in and spin-out. Whatever reorganization takes place, both spin-in and spin-out need to be incorporated and a new relationship to regions clearly specified. 5.8.4  Spin-in Challenges NASA’s challenge in creating an effective regional infrastructure has be- come even more difficult with this increased emphasis on using SBIR for gaining technology for NASA’s mission needs. The dilemma NASA faces is that, with a dual-purpose program, both infrastructures are important and should be linked. A program focused primarily on spin-in requires an infrastructure that differs from the original regional infrastructure described above. There are two reasons why it may be difficult to create an orientation towards infusion. First, the pre-2005 program created an infrastructure that emphasized commercialization—and commercialization (not infusion) is what the regional infrastructure was designed to do. Second, organizing the infrastructure to ac- complish spin-in will require greater coordination and focus by NASA personnel. Most likely, more resources will be necessary because more planning and man- agement will be needed to ensure that SBIR projects research technology useful to NASA. Some at NASA think that SBIR will be marginalized by the shift to a spin-in approach to technology transfer. This was also the conclusion of NAPA in its recent report on NASA technology transfer. A spin-in focus, like a spin-out focus, requires specialized and frequent con- tact with awardee firms. Moreover, it requires a specialized knowledge of how the relevant technology meshes with NASA’s specific mission needs. Interestingly, a detailed knowledge of the region’s firms and industries is also necessary. It seems clear that NASA’s new emphasis on spin-in raises the stakes on the use of the SBIR program. With tight budgets, the restructuring of NASA centers’ technology capabilities, and a competitive approach to funding centers, centers that figure out how to best utilize SBIR to garner mission-use technology will have an advantage. Several steps in this direction may be necessary. • Greater Role for Centers. First, rather than turning to regional orga- nizations established to commercialize federally funded research for regional benefit, individual NASA centers (and their researchers) will need to take greater responsibility for identifying companies best matched to the Center’s specific technology needs. While the extent to which the pool of firms selected this way

PROGRAM MANAGEMENT 137 will overlap with the pool selected pre-2005 is unknown, significant differences will likely exist. For example, many firms that have sought NASA SBIR funds in the past had commercialization, not NASA mission needs, as their top priority. If NASA centers emphasize short-term NASA needs over more long-term commer- cialization potential, it may divert the nation’s most innovative small companies away from NASA-oriented research areas entirely. The emphasis on spin-in does not eliminate the need for a regional intermedi- ary with deep knowledge of a region’s companies and industries. It is not likely that a NASA center can fill this gap. However, NIST’s Manufacturing Extension Programs (MEPs) might meet the need for local industry information. The MEPs may be the appropriate entities with the most knowledge of each state and local area’s manufacturing companies. • New Resources and Approaches. Second, with spin-in, NASA centers will need to find new resources and new approaches to managing SBIR from solicitation to Phase III. Under the previous spin-out emphasis, there was insuf- ficient incentive for individual NASA centers to devote the resources to track an SBIR firm’s progress carefully while it develops a technology over several years. As NASA centers are evaluated more carefully and systematically based on their success in gaining mission-purpose technology from the SBIR program, new structures will need to emerge to improve transitions throughout the several phases and to increase NASA benefits over the long run. • Collaboration among NASA Centers and National Laboratories. Third, since the bulk of each Center’s awards are distributed throughout the coun- try (and not just in the local area), managing the spin-in approach will require far more collaboration across all NASA Centers and perhaps other federal labs. Whether the goal is spin-in or spin-out, getting the most out of the SBIR program will take much more frequent and direct contact between NASA and SBIR com- panies. All NASA Centers must work system-wide on their core technologies. For example, information on firms and technologies developed by NASA Ames should be shared with all NASA Centers and possibly a new, centralized RTTC (CRTTC) with responsibility for assuring the systemwide cooperation of RTTCs. The RTTCs and SBIR programs of each NASA Center should also exchange both commercialization and spin-in information with other federal agencies to maximize the joint (multiagency) benefits. For example, a NASA technology of potential use to DoD should be shared with DoD. The difference between a spin-in and spin-out systemwide approach is ba- sic. Spin-in requires much greater involvement of NASA centers while spin-out involves much greater involvement of a commercialization intermediary. A well functioning and scaled up system of NASBO chapters could potentially take on this role effectively. • Identify National Technology Capabilities. Fourth, spin-in will also create a need for a new NASA capability for identifying and analyzing the spe- cific R&D and technology capabilities of firms, corporate R&D labs, universities,

138 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION research hospitals, and other federal labs. Presumably this analytical function would take place at NASA headquarters, although it is also a logical candidate for shared responsibility across all federal agencies—not just all SBIR programs. In addition to accurately describing regional technology capabilities, NASA Head- quarters would also take responsibility for drawing on advisers inside and outside NASA with the greatest expertise on NASA’s high priority technology needs. Regarding SBIR, one outcome would be superb technology intelligence that can be used to identify the best firms, universities, etc., working on key technologies. This approach to developing and using technology intelligence has a parallel in private industry, which has increasingly turned to external sources of technology over the past ten years. • Partnerships. Fifth, implicit in NASA’s current efforts to use its tech- nology transfer programs to obtain external sources of technology is a more strategic, long-term approach to meeting NASA’s technology needs. The implica- tion would seem to be that each NASA center will be responsible for develop- ing and implementing a long-run strategy for creating critical partnerships with universities, other federal labs, small high-tech firms, etc., that build NASA’s high-priority technology capabilities. This requires deeper knowledge of regional innovation systems. • Evaluation. Finally, NASA centers would need to be responsible for working closely with each other (or the new CRTTC) in measuring long-term, indirect benefits and developing new metrics to support spin-in. The methodology would be developed to be consistent for both spin-in/infusion and commercializa- tion applications. The gains from restructuring NASA’s technology transfer network will take the form of better technology, lower costs, and a higher rate of innovation in support of NASA’s space missions. In specific terms, this implies: • A higher percentage of Phase I companies achieving Phase III; • A faster rate of development of high priority NASA technologies from R&D in Phase I to use of technology by NASA in Phase II and Phase III for space missions; and • A larger fraction of SBIR technology used by NASA to advance its own capabilities.

PROGRAM MANAGEMENT 139 TABLE 5-5  Dual Purpose Uses of SBIR Spin-in (Mission Purpose) Spin-out (Commercialization) Significance Because advanced technology is Because commercialization requires of Region highly-geographically concentrated, extensive face-to-face contact, regions are important as the primary regions are important as places sources of technology. with infrastructure supporting commercialization. Requirements/ Requires high level knowledge of Requires knowledge of regional Capabilities specific technologies as well as industries, business, early-stage funders, knowledge of NASA needs and and state & local programs. organization. Organizational NASA Centers takes primary States take primary responsibility Responsibility responsibility for 1) evaluating for funding a new RTTC-like proposals, 2) monitoring technology, intermediary with responsibility for 3) ensuring PI-PIII transitions with outreach to identify new PRINCIPAL spin-in potential, and ensuring INVESTIGATOR candidates, evaluate cross-center sharing. NASA centers proposals for commercialization could work closely with NIST’s potential, and evaluate the state’s Manufacturing Extension Program PII successes for commercialization (MEP) network. opportunities, and assisting businesses in commercialization. New NASA Headquarters will need to A new, centralized cross-agency Organizational create or augment its capabilities for intermediary may be required to Structures identifying external sources of its ensure that SBIR/STTR technology high priority technologies for mission available in one region is connected purposes. NASA may also want to with commercialization opportunities create a new structure for funding in other regions. Like the NTTC, the highly promising mission-purpose new organization could be funded by all technologies that develop in Phase II. federal agencies. The new intermediary NASA centers could also work with could work closely with NIST’s each state’s RTTC-like intermediary to Manufacturing Extension Program link successful PII firms with NASA’s (MEP) network. prime contractors. Evaluation NASA would fund the development Working across all federal agencies, and Metrics of an evaluation methodology NASA would also fund development and metrics following a model of an evaluation methodology similar to that used by the ATP. that incorporates private-sector The methodology would serve as a commercialization benefits using an guide to developing metrics that are approach that dovetails with evaluation consistent with SBIR/STTR’s long- of spin-in benefits and metrics. term mission objectives. SOURCE: National Aeronautics and Space Administration.

140 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION Annex to Chapter 5 SBIR at the NASA Centers Until the most recent reorganization, basic SBIR processes were common to all ten field centers. Other processes affecting infusion/commercialization op- portunity reflect the missions of NASA’s various field centers. As a result, the centers have had differing definitions of what constitutes successful infusion/ commercialization of SBIR Phase II technologies, making it difficult to compare infusion/commercialization outcomes with accuracy. Below, we examine how SBIR was administered at each of the NASA centers. 5.9  AMES RESEARCH CENTER (ARC)—SAN JOSE, CA The Ames SBIR Program Office primary interface is with the three research directorates at Ames, whose technology foci are not closely aligned with agency Mission Directorates. These research directorates govern the SBIR topic develop- ment process, although Exploration Systems Mission Directorate does participate in subtopic development. SBIR Phase I activity by the Ames Program Office includes a cursory evalu- ation of infusion/commercialization opportunity in the proposal review process, although the Program Office is planning a preparatory workshop for the 2006 Phase I proposals ranking committee that will have an infusion/commercializa- tion appraisal component. In both Phase I and Phase II proposal review processes, the research direc- torates make proposal ranking recommendations to a ranking committee. During Phase II proposal evaluation, outside peer reviews of infusion/commercialization opportunity are considered, but are not decisive in ranking committee funding recommendations. During a Phase II contract, little or no emphasis is placed by SBIR project monitors on identification of infusion/commercialization opportuni- ties by the SBIR awardees. The Ames SBIR Program Office does not monitor either infusion or commer- cialization outcomes of SBIR projects, but evidence of “success stories”—usu- ally obtained anecdotally, and then verified with the SBIR firm—is passed on to NASA SBIR Level 1 and 2 principals. Although the aggregate of such success stories is “. . . a small number,” according to Ames SBIR Program Manager Geoffrey Lee, “. . . some of NASA’s largest Phase III contracts come from Ames SBIR contracts, whose technology was infused into Space Life Sciences Payloads programs.”51 Among the 18 Phase III contracts from Ames documented since 1983, Ames claims the largest Phase III dollar aggregate from SBIR projects among the ten NASA field centers. Its Phase III contracts have totaled approxi- 51  Interview with Geoffrey Lee, February 10, 2005.

PROGRAM MANAGEMENT 141 mately $123,500,000. This includes NASA’s largest individual SBIR Phase III contract—$59,104,971 given to Orbital Technologies, Inc. in 1994 to develop a plant research unit. NASA’s new full-cost accounting environment for all agency functions (i.e., all funding is derived from mission programs) requires field center compliance and reorganization in many cases. Consequently, the Ames SBIR Program Of- fice has had to explore development of an infusion/commercialization assistance component. This component will be derived from best practices of other field centers and discussions with Ames’s Technology Partnerships Office. 5.10  DRYDEN FLIGHT RESEARCH CENTER (DFRC)—EDWARDS, CA Dryden is the sole “flight research Center” among the ten NASA field centers. Its SBIR Program Office is primarily responsible for proof-of-concept aeronautics flight-test support. SBIR Phase I activity at Dryden includes a cursory evaluation of infusion/commercialization opportunity in the Phase I proposal technical review process. Phase II proposal evaluation is limited to a formal out- side peer review of infusion/commercialization opportunity. In both Phase I and Phase II proposal review processes, the research director- ates make recommendations to the DFRC Ranking Committee. During a Phase II contract, SBIR program monitors (from research directorates) and the Program Office do work with awardees to help ensure that the small business aligns its SBIR technology with a NASA acquisition opportunity. This work includes as- sistance in assuring resonance between the SBIR firm’s choice of “key words” 52 and NASA’s base technology taxonomy, so that a search of the NASA SBIR Da- tabase for technologies that could be infused into NASA enterprise work would yield relevant SBIR project files. DFRC SBIR does not formally monitor either infusion or commercialization outcomes of SBIR projects, and evidence of “success stories”—usually obtained anecdotally, and then verified with the SBIR firm—is passed on to NASA SBIR Level 1 and 2 principals. Informally, notes Dryden SBIR Program Manager Rod Bogue, “. . . we try and call our SBIRs at least every two years to ascertain Phase III success, especially NASA infusion of their technologies. To me, the business case of a SBIR project has to focus on Phase III opportunity—but our administrative resources to help SBIR projects in this regard are very limited.”53 DFRC SBIR has no estimate of its Phase III results from SBIR contracts. Accord- ing to the NASA EHB Web site, DFRC has had two Phase III contracts. Two of Dryden’s current SBIR Phase II projects participate in the NASA 52  “Key words” are self-assigned by SBIR awardees to identify their technology or technology domain. The key word chain appears on the cover sheets of most agency SBIR records, including electronic databases, and are searchable by external inquiry. 53  Interview with Rod Bogue, February 16, 2005.

142 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION Commercialization Center incubator. Their performance as incubator tenants is not monitored by DFRC SBIR. 5.11  GLENN RESEARCH CENTER (GRC)—CLEVELAND, OH The GRC SBIR Program Office aggressively pursues infusion opportunities for SBIR technologies. As with all field centers, SBIR Phase I activity includes evaluation of infusion/commercialization opportunity in the Phase I proposal technical review process. Phase II proposal evaluation also includes a formal out- side peer review of infusion/commercialization opportunities. GRC adds unique resources to these processes: • In addition to strongly encouraging Mission Directorate program offices to participate in SBIR topic and subtopic development, the SBIR Program Office holds one-day infusion/commercialization workshops—just prior to the proposal review process—for reviewers and for the subtopic managers who are key to the review process. • The GRC SBIR Program Office has, since 1986, noticed relevant acqui- sition opportunities in the Army and Air Force, and maintains ties with appropri- ate program offices in both service agencies. • GRC markets its SBIR program to small business SBIR candidates at various national and state SBIR conferences, to identify candidates with infu- sion/commercialization potential. To track GRC SBIR “success stories,” GRC SBIR Program Manager Dean Bitler employs a five-column Excel table that notes the SBIR company and loca- tion, the NASA organizational code and COTR; the SBIR project title; the SBIR topic/subtopic number; the year of award; and finally, a succinct infusion/com- mercialization history. This table, updated regularly with SBIR firms, tallies 142 projects whose technologies were successfully infused into federal acquisitions, were success- fully commercialized, or were in transition to Phase III at the time of the update. GRC does not track aggregate revenues. According to the NASA Electronic Handbook Web site, there have been a cumulative 48 Phase III contracts at Glenn. For GRC SBIR Program Office, the key venues for evaluation discussion are the semi-annual meetings of SBIR Level 1, 2, and 3 personnel, where infusion/ commercialization issues are both formally and informally discussed. 5.12  GODDARD SPACE FLIGHT CENTER (GSFC)—GREENBELT, MD The “Center-wide investment strategy” of Goddard’s Technology Manage- ment Office—of which the GSFC SBIR Program Office is a component—ensures

PROGRAM MANAGEMENT 143 that applications needs of the resident Earth Science Enterprise Mission Director- ate (ESEMD) drive all Goddard technology development. The Center’s SBIR Program Office says its role is to assist in managing a balanced technology pipeline at Goddard. “We are part of the ESEMD program offices’ investment strategy, and have been for some time,” says SBIR PM Dr. James Chern. “Eight years ago, we aligned our SBIR program with Goddard’s needs . . . but still, it took at least three years to convince the mission program offices of SBIR benefits. In fact, SBIR technologies are essential to building mis- sion technology portfolios.”54 The GSFC SBIR program emphasizes maturing SBIR projects at high Tech- nology Readiness Level (TRL), as seen in an Astra project strategic technol- ogy/systems model. GSFC deploys the agency-wide SBIR process model. SBIR Phase I activity includes evaluation of infusion/commercialization opportunities in the Phase I proposal technical review process. Phase II proposal evalua- tion also includes a formal outside peer review of infusion/commercialization opportunities. At Goddard, there is also serial interface with ESEMD regarding topic/sub- topic development to ensure that SBIR awards help fill the Center’s technology pipeline. ESEMD group leaders and branch heads play key leading roles in the SBIR process—including aggressive roles as subtopic managers. In these roles, they advocate specific infusion opportunities for SBIR technologies in ESEMD programs. GSFC’s SBIR Program Office pairs subtopic managers with COTRs to promote infusion opportunities. The COTR takes the lead in Phase II, begin- ning with the presentation of infusion opportunities, cited in Phase II proposals, to the review committee. GSFC’s Earth Science Technology Office management strategy map il- lustrates Goddard’s approach to meeting the technology needs of what were the Earth Science Enterprise missions, including the SBIR contribution to those missions. GSFC SBIR Program Office also markets its resources to small business SBIR candidates at various national and state SBIR conferences, and participates in the Center’s annual “Small Business Day.” Due to staff and budget constraints, GSFC SBIR Program Office does not actively track the infusion/commercialization success of its SBIR contractors. GSFC was, however, the first Center to produce and disseminate a Success Sto- ries publication—a function now assumed by NASA SBIR Levels 1-2 through Spin-off and other publications. According to the NASA EHB Web site, Goddard has had 15 Phase III contracts over the years. While the process is informal, evaluation of SBIR practices is continuous at Goddard through discussions with the Technology Management Office. Recom- mendations from these discussions are presented at the semi-annual meetings of Level 1, 2, and 3 personnel. 54  Interview with Dr. James Chern, February 22, 2005.

144 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION Key infusion/commercialization assistance issues for GSFC SBIR include improved alignment of ESEMD element program themes and potential Phase III opportunities, improved identification of SBIR projects with high infusion poten- tial, increased work with COTRs to improve communication with potential tech- nology customers, and better use of the NASA technology transfer network. 5.13  JET PROPULSION LABORATORY (JPL)—PASADENA, CA Owned by NASA and operated by the California Institute of Technology (normally known as “Caltech”), JPL has five resident Directorates: Planetary Flight Projects, Solar System Exploration, Astronomy and Physics, Earth Science Enterprise, and Interplanetary Network. Of these, the Mars mission’s Technology Program Office is key to JPL’s SBIR infusion focus, and is the principal contribu- tor to SBIR topic/subtopic development. Most JPL SBIR awards directly support the field center’s Strategic Technol- ogy Plan, and the SBIR program is closely aligned with future mission needs. This alignment creates infusion opportunities for SBIR projects, but also triggers added JPL management and technical staff support for its SBIR program office. SBIR topic/subtopic managers have technical/management responsibilities at JPL and NASA Headquarters levels. As with all field Centers, SBIR Phase I activity includes evaluation of infu- sion/commercialization opportunities in the Phase I proposal technical review process. Phase II proposal evaluation also includes a formal outside peer review of infusion/commercialization opportunity. As a “national field center,” however, the JPL SBIR proposal review process is a cross-cutting effort in which techni- cal program office representatives from other field centers actively participate in proposal evaluation and ranking committees. Moreover, JPL SBIR awardees in Phase I are often required to go beyond the “proof of principle” work typical of Phase I activity and perform prototyp- ing work usually found in the final year of Phase II activity. “We want to avoid funding research that does not lead to infusion opportunity,” says JPL’s SBIR Program Manager Wayne Schober, “. . . in concert with our belief that the pro- gram offices should own SBIR.”55 At JPL, the lead for SBIR infusion/commercialization activity is the project Technical Monitor (often a technical program office representative) charged with advocating for SBIR awardees to project managers in mission program offices. The Technical Monitor is tasked with ensuring that an SBIR project observes NASA program requirements. During Phase II, the Technical Monitor also serves as a liaison between SBIR contractors and appropriate project managers, and ar- ranges for project representative participation in meetings with the SBIR contrac- tor. In turn, the small business is required to provide status reports to its Technical 55  Interview with Wayne Schober, February 28, 2005.

PROGRAM MANAGEMENT 145 Monitor on the interface with mission projects. This information is then pushed back up to center technology management personnel for evaluation. JPL SBIR Program Office both tracks and documents its Phase II awards for infusion/commercialization success, using a stringent dollar threshold of $100k for defining Phase III success.56 By its definition, of 228 JPL Phase II projects since 1983, 41 have been infused into NASA mission programs (technology and flight hardware), and 66 have found commercial success, with some projects having both infusion and commercialization success. Aggregate revenues are not available.57 While JPL SBIR Program Office uses the agency Success Stories Quad for- mat to record and market its SBIR projects, it has also created a mission-specific format to document infusion of SBIR technologies. Evaluation of SBIR practices is informal and continuous at JPL through dis- cussions within the Technology Management Office and various project offices. This evaluation is informed in part by status reports provided by the SBIR firms as well as input from Technical Monitors. Some recommendations, filtered from these discussions, are presented at the semi-annual meetings of SBIR Level 1, 2, and 3 personnel. 5.14  JOHNSON SPACE CENTER (JSC)—HOUSTON, TX JSC has the largest mission program-based research budget of the field centers. JSC’s SBIR program has a complex interaction with multiple Mission Directorates. ESMD and Space Operations are the principal customers and driv- ers of SBIR topic/subtopic development, with the Explorations and Aeronautics directorates also playing substantive roles. Discussions with directorate program leads and technology element managers align JSC’s SBIR program with the center’s mission priorities. SBIR Program Manager Dr. Kumar Krishen also reviews agency Broad Area Announcements (BAAs) and Intramural Calls for Proposals (ICPs) to parse them for information that can inform the topic/subtopic development process.58 As with other field centers, SBIR Phase I activity includes evaluation of in- fusion/commercialization opportunities in the Phase I proposal technical review process. Phase II proposal evaluation also includes a formal outside peer review of infusion/commercialization opportunities. JSC’s unique practice focuses on interaction between SBIR principals and mission program principals. When subtopic managers brief the ranking com- mittee on Phase II proposals, they must cite evidence of infusion opportunities, including summaries of specific discussions and meetings with directorate tech- nology element leads on SBIR technology viability. 56  Some other field centers reportedly use a lower dollar threshold to determine Phase III success. 57  Itshould be noted that there is only one Phase III contract cited on the NASA EHB Web site. 58  Interview with Dr. Kumar Krishen, February 24, 2005.

146 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION Dr. Krishen reaches beyond NASA in the greater Houston area to advocate for SBIR commercialization and investment in SBIR firms with federal program infusion opportunities. At the time of his interview with NAS program staff in early 2005, his most recent SBIR marketing pitch, for example, had been made to the Houston Area Economic Alliance. JSC SBIR Program Office actively promotes its Phase III successes through its annual Johnson Space Center Spin-Off Awards (which also reward “spin-in” technologies infused into NASA mission directorate programs) and Success Stories. In addition, it annually nominates SBIR candidates for NASA’s Space Technology Hall of Fame—with two inductees in both 2004 and 2005. JSC uses NASA’s Success Stories Quad format of one-page summaries of SBIR projects, for use in infusion/commercialization marketing and Phase III recordation work. Although JSC SBIR Program Office does actively monitor its SBIR portfo- lio of companies to assess Phase III success, tracking statistics are not available except to NASA SBIR Levels 1-2 principals for inclusion in the agency-wide Spin-off and other publications. According to the NASA Electronic Handbook Web site, JSC has had 44 Phase III contracts over the years. JSC SBIR’s main venues for evaluation are the semi-annual meetings of SBIR Level 1, 2, and 3 personnel, and the weekly/monthly electronic confer- ences. Infusion/commercialization issues—including Dr. Krishen’s innovations regarding infusion/commercialization opportunity consideration in Phase I and II proposal rankings—have been reviewed by the NASA SBIR community at the semi-annual events. Internally, informal evaluation discussions of SBIR program effectiveness are held between the SBIR Program Office and local Mission Di- rectorate personnel. In 2005, the principal emergent issue was the need for closer ESMD participation in the subtopic review process. 5.15  KENNEDY SPACE CENTER (KSC)—FLORIDA KSC has a small mission program research budget, and a correspondingly small SBIR program. Mission project offices, especially in Human and Robotic Technology work, drive SBIR infusion/commercialization activity. KSC SBIR shares the basic topic/subtopic development and Phase I-Phase II proposal review processes common to all NASA field centers, but includes a special emphasis on infusion opportunity and infusion planning by the SBIR contractor in both Phase I and Phase II proposal review. The responsible party is the SBIR project monitor, or COTR, who is tasked in the Phase II proposal presentation format with identifying NASA technology user community “gate- keepers.” Gatekeepers include project office program managers from the Expend- able Launch and Space Station mission groups who provide infusion leadership for SBIR technology once the COTR has obtained buy-in from that gatekeeper through discussions and meetings.

PROGRAM MANAGEMENT 147 The KSC SBIR Program Office also monitors the participation of one of its SBIR firms resident in the NASA Commercialization Center incubator. Citing budgetary and staff constraints, the KSC SBIR Program Office does not actively track and update information on its Phase II projects regarding Phase III activity, apart from its interest in tracking SBIR-derived intellectual property. According to the NASA EHB Web site, there have been nine Phase III contracts over the years at KSC. As KSC SBIR is aligned with the center’s Human and Robotic Technology Program and that program’s Element Plan, the SBIR Program Office holds pe- riodic evaluative discussions with the infusion “gatekeepers” from Expendable Launch and Space Station organizations. Priority issues arising from these discussions in 2004-2005 are: closer align- ment of SBIR subtopics with KSC’s Human and Robotic Technology Program Element Plan; availability of demonstration venues to Phase II SBIR projects to ensure higher TRL levels upon Phase II completion; more formal infusion plan- ning by SBIR firms; and infusion incentives for these firms. These incentives might include cost-plus, award-fee contracts where the award-fee is heavily de- pendent on progress with the technology infusion process. No formal KSC plans have yet been developed to deal with these issues. 5.16  LANGLEY RESEARCH CENTER (LARC)—HAMPTON, VA Langley was founded in 1920 at the genesis of the aviation industry. The nation’s first aeronautics lab, Langley is dedicated to aeronautics and engineering technologies, as well as atmospheric science, with Earth Sciences responsibilities. Its $700 million budget in 2005 marks it as the smallest field center. However, Langley is co-located with Langley Air Force Base, the Jefferson National Ac- celerator Laboratory, Northrop Grumman Newport News naval shipyard and the Virginia Institute of Marine Science. As these institutions have a history of partnering to pursue technologies of mutual interest, numerous small firms with advanced technology skills can be found on LaRC’s perimeter. Since 2000, Vir- ginia has averaged the nation’s second-highest number of SBIR awards by state, with the Hampton Roads region surrounding LaRC as the state’s second most SBIR-productive area.59 At LaRC, SBIR Phase I activity includes evaluation of infusion/commercial- ization opportunities in the Phase I proposal technical review process. Phase II proposal evaluation also includes a formal outside peer review of infusion/com- mercialization opportunities. With LaRC’s emphasis on aeronautics and engineer- ing technologies, and atmospheric science, NASA mission directorate program offices do not appear to be solicited for participation in SBIR topic and subtopic 59  Hampton Roads Market Alliance, Hampton Roads: The Advanced Technology Center (presenta- tion), 2004.

148 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION development as frequently as with other field Centers. STTR topic development is influenced by LaRC’s Center for Excellence for Structures and Materials. The LaRC SBIR Program Office markets its resources to small business SBIR candidates at state SBIR events, its own annual “Research Development and Technology Conference” (in collaboration with Virginia’s Center for Inno- vative Technologies), and periodic small business mentoring events. Beginning in 2004, the LaRC SBIR Program Office leveraged its ties with regional tech- nology partners noted above, co-producing SBIR events in July and September, 2004. These two events launched the LaRC SBIR PO’s Technology Partnership Initiative. The “Northrop Grumman SBIR Technology Transition Conference” on 28 July, 2004, introduced a pool of 30 SBIR Phase II awardees from the NASA LaRC, Navy SBIR, and Department of Energy SBIR programs to Northrop Grumman shipyard engineers through the yard’s VASCIC Technology Develop- ment Center. SBIRs were preselected according to their congruity with shipyard technology needs in 11 technology areas (e.g., data control, sensors and monitor- ing, power systems, etc.) for the Navy’s Virginia-class submarine and CVN-21 aircraft carrier. The College of William & Mary’s follow-on “SBIR Mentoring Day” on Sep- tember 1, 2004, gave five firms from the same SBIR Phase II pool the opportunity to present their business cases to an experienced group of technology investors, for identification and analysis of risk factors. LaRC SBIR Program Office plans to continue the Technology Partnership Initiative at other corporate venues. LaRC SBIR Program Office does not appear to make available information on its Phase II projects regarding Phase III activity. However, according to the NASA EHB Web site, there have been 29 Phase III contracts over the years at Langley. As regards the Technology Partnership Initiative, according to Northrop Grumman Newport News sources, in December, 2004 the shipyard was pursuing discussions about further collaboration with six of the SBIR projects that pre- sented at the July, 2004 conference. No further updates are available. Scrutiny of the NASA LaRC Web site does not reveal “Success Stories” or other awards accruing to LaRC SBIR awardees. For LaRCs SBIR PO, the principal venue for evaluation is the semi-annual meeting of SBIR Level 1 and 2 personnel with all ten NASA field center SBIR program managers. Infusion/commercialization issues can be both formally and informally discussed among the NASA SBIR peer community at these events. No other information is available about evaluation of the LaRC SBIR program or its Technology Partnership Initiative. 5.17  MARSHALL SPACE FLIGHT CENTER (MSFC)—HUNTSVILLE, AL MSFC SBIR practice observes the basic NASA field center formula. SBIR Phase I activity includes evaluation of infusion/commercialization opportunities

PROGRAM MANAGEMENT 149 in the Phase I proposal technical review process. Phase II proposal evaluation also includes a formal outside peer review of infusion/commercialization opportuni- ties, as well as internal data on the alignment of candidate SBIR technologies with ESMD and/or SMD directorate needs. At MSFC, however, these needs are expressed by MSFC’s Chief Tech- nologist and the Center Technology Council, from subtopic development prior to Phase I onwards. The SBIR proposal review Ranking Committee’s Phase I recommendations are pushed back up to the Center Technology Council, which then assigns project Technical Monitors from appropriate program offices—or, in some cases, research labs. Infusion opportunity is anticipated by this direct interface in SBIR Phases I and II between the SBIR firm and an appropriate program office. Subtopic managers continue to play a role in SBIR work through the Phase II review process. Phase III activity is tracked somewhat formally at MSFC using a six-column Excel template. The center’s current tracking file cites 200 Phase IIIs (from ap- proximately 300 SBIR Phase II projects since 1989) which have a non-SBIR rev- enue aggregate of $117,531,532.60 Another ten SBIR projects are in the process of being added to this list, for an added $10,494,000. “Although we are staff and budget constrained,” says MSFC PM Lynn Garrison, “. . . we have learned to rely on procurement records—in addition to direct follow-up with SBIR awardees—as an accurate and comprehensive way of getting Phase III data we otherwise might miss. This was an MSFC decision, that the SBIR Program Office needed expert support to accurately track infusion/commercialization success.”61 Among the ten NASA field centers, MSFC approach to recording Phase III results is unique. Like other field Centers, MSFC evaluates its SBIR infusion/commercializa- tion practices informally, and it participates in discussions at the semi-annual meetings of field center SBIR PMs with NASA SBIR Level 1 and 2 staff. In ad- dition, MSFC SBIR Program Office has used strategic shifts at the agency-wide level—such as the recent merger of SBIR activity with ESMD—as an opportu- nity to reevaluate core practices. The recent decision to align center SBIR work through MSFC’s Chief Technologist and the Center Technology Council, instead of directorate POCs, is one such reform, as is the decision to track Phase III oc- currences through procurement sources. 5.18  STENNIS SPACE CENTER (SSC)—MISSISSIPPI SCC’s primary role is that of a test facility, and it averages six or fewer SBIR Phase II projects annually. Apart from observing the basic SBIR topic development and proposal review process, no special emphasis is placed on infu- sion/commercialization opportunity in the SSC SBIR program. 60  According to the NASA EHB Web site, Marshall has had 113 (the largest by far of any of the 10 centers) Phase III contracts over the years. 61  Interview with Lynn Garrison, February 28, 2005.

150 SBIR AT THE NATIONAL AERONAUTICS AND SPACE ADMINISTRATION SSC has recorded four instances of Phase III infusion/commercialization. Although SBIR PM Jim Bryant believes that at least two of these four SBIR tech- nologies has achieved multiple Phase IIIs, no detailed tracking data is available. According to the NASA EHB Web site, SCC has had eight Phase III contracts over the years.

Next: Appendixes »
An Assessment of the SBIR Program at the National Aeronautics and Space Administration Get This Book
×
Buy Hardback | $110.00 Buy Ebook | $89.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The Small Business Innovation Research (SBIR) program is one of the largest examples of U.S. public-private partnerships. Founded in 1982, SBIR was designed to encourage small business to develop new processes and products and to provide quality research in support of the many missions of the U.S. government, including health, energy, the environment, and national defense. In response to a request from the U.S. Congress, the National Research Council assessed SBIR as administered by the five federal agencies that together make up 96 percent of program expenditures.

This book, one of six in the series, reports on the SBIR program at the National Aeronautics and Space Administration, and finds that the program is making significant progress in achieving the Congressional goals for the program. Keeping in mind NASA's unique mission and the recent significant changes to the program, the committee found the SBIR program to be sound in concept and effective in practice at NASA.. The book recommends programmatic changes that should make the SBIR program even more effective in achieving its legislative goals.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!