National Academies Press: OpenBook

Assessing NASA's University Leadership Initiative (2021)

Chapter: 1 Assessing ULI's Progress Toward Meeting Its Goals

« Previous: Summary
Suggested Citation:"1 Assessing ULI's Progress Toward Meeting Its Goals." National Academies of Sciences, Engineering, and Medicine. 2021. Assessing NASA's University Leadership Initiative. Washington, DC: The National Academies Press. doi: 10.17226/25996.
×
Page 6
Suggested Citation:"1 Assessing ULI's Progress Toward Meeting Its Goals." National Academies of Sciences, Engineering, and Medicine. 2021. Assessing NASA's University Leadership Initiative. Washington, DC: The National Academies Press. doi: 10.17226/25996.
×
Page 7
Suggested Citation:"1 Assessing ULI's Progress Toward Meeting Its Goals." National Academies of Sciences, Engineering, and Medicine. 2021. Assessing NASA's University Leadership Initiative. Washington, DC: The National Academies Press. doi: 10.17226/25996.
×
Page 8
Suggested Citation:"1 Assessing ULI's Progress Toward Meeting Its Goals." National Academies of Sciences, Engineering, and Medicine. 2021. Assessing NASA's University Leadership Initiative. Washington, DC: The National Academies Press. doi: 10.17226/25996.
×
Page 9
Suggested Citation:"1 Assessing ULI's Progress Toward Meeting Its Goals." National Academies of Sciences, Engineering, and Medicine. 2021. Assessing NASA's University Leadership Initiative. Washington, DC: The National Academies Press. doi: 10.17226/25996.
×
Page 10
Suggested Citation:"1 Assessing ULI's Progress Toward Meeting Its Goals." National Academies of Sciences, Engineering, and Medicine. 2021. Assessing NASA's University Leadership Initiative. Washington, DC: The National Academies Press. doi: 10.17226/25996.
×
Page 11

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

1 Assessing ULI’s Progress Toward Meeting Its Goals On its ULI website,1 NASA states that its ULI strategic goals are as follows:  Assist in achieving aviation outcomes defined in the Aeronautics Research Mission Directorate (ARMD) Strategic Implementation Plan2 through NASA-complementary research  Transition research results to an appropriate range of stakeholders that lead to a continuation of the research. Transition can occur in a number of ways, including the following: — Creates a new product line in U.S. industry or a new ARMD project — Whole ULI concept is transitioned to U.S. industry/ARMD project — Part of the ULI concept is transitioned to U.S. industry/ARMD project — ULI findings impact direction of U.S. industry/ARMD  Provide broad opportunities for students at different levels, including graduate and undergraduate, to participate in aeronautics research  Promote greater diversity in aeronautics through increased participation of minority-serving institutions and underrepresented university faculties in ULI activities The study committee evaluated each of these strategic goals through a series of briefings from representatives from NASA, universities, industry, and others, followed by requests for information for greater insights and clarity. While it is difficult to measure success for some of the goals, particularly as the first round of awards is still ongoing, this chapter summarizes the committee’s assessments. Overall, the committee commends ARMD for its development of ULI and applauds its broad goals. This report, including this chapter, provides a number of findings and recommendations for improvement for NASA to achieve its vision for ULI. In general, the committee saw strong evidence of freedom for the PIs, with no interference from NASA, particularly regarding the scopes and directions of the funded project. Most of the PIs that presented to the committee indicated that they liked this freedom. There was also strong evidence of PIs getting positive feedback from both NASA and peer reviewers that have enhanced the quality and 1 NASA Aeronautics Research Institute, “The University Leadership Initiative (ULI),” https://nari.arc.nasa.gov/uli, accessed September 22, 2020. 2 NASA Aeronautics Research Mission Directorate, “Strategic Implementation Plan,” https://www.nasa.gov/aeroresearch/strategy, accessed September 22, 2020. 6

ASSESSING ULI’S PROGRESS TOWARD MEETING ITS GOALS 7 direction of the funded projects. It also appears that the peer reviewers are highly qualified individuals from industry, academic institutions, and government agencies, and there is strong evidence that the multidisciplinary nature and teamwork aspects of the ULI’s projects are well appreciated by the PIs. Below, the committee discusses each of the ULI strategic goals and how the ULI program relates to them. Strategic Goal 1: Assist in achieving aviation outcomes defined in the ARMD Strategic Implementation Plan through NASA-complementary research. From a funding perspective, ULI is a small part of ARMD’s overall research, although one that NASA hopes will make an outsized impact to its six strategic thrusts: 1. Safe, efficient growth in global operations; 2. Innovation in commercial supersonic aircraft; 3. Ultra-efficient subsonic transports; 4. Safe, quiet, and affordable vertical lift air vehicles (“transition to low-carbon propulsion” for the first three rounds); 5. In-time system-wide safety assurance; and 6. Assured autonomy for aviation transformation. Over the first three rounds, the NASA ULI program has awarded programs as follows:  One for safe, efficient growth in global operations;  One for innovation in commercial supersonic aircraft;  One for ultra-efficient commercial vehicles;  Two for transition to low-carbon propulsion;  Two for real-time system-wide safety assurance; and  Two for assured autonomy for aviation transformation. All six strategic thrust areas have therefore achieved at least one award. The ULI program has also awarded two other important areas that were in the solicitations in addition to the strategic thrusts:  Two for advanced aviation manufacturing, and  Two for materials and structures for next-generation aerospace systems. FINDING: In the first three rounds of ULI awards, NASA selected proposals that were applicable to all of the original strategic thrusts. Whereas there were awards made to support autonomy, electric propulsion, and technologies such as fastener-free assemblies that may be relevant to urban air mobility (which NASA now calls “advanced air mobility,” AAM), the lack of awards on vertical lift air vehicle technology to date was noted by the committee. With the repurposing of Strategic Thrust 4 to specifically call out “safe, quiet, and affordable vertical lift air vehicles,” NASA can now select vertical lift proposals in Round 4 and beyond. There may also have been confusion (on the part of proposers and NASA reviewers) about whether proposals for AAM were duplicative with existing research under the Revolutionary Vertical Lift Technology Project. The committee received feedback from past proposers that it is difficult for potential PIs to differentiate between “complementary” and “duplicative” research. For instance, ARMD is obviously performing aeronautics research, so an absurdism would be that all aeronautics research is therefore duplicative. At a micro level, is a new technique for computational fluid dynamics (CFD) optimization

8 ASSESSING NASA’S UNIVERSITY LEADERSHIP INITIATIVE acceptable, despite the fact that NASA already supports CFD research? The committee heard that one ULI project is on hypersonics despite the fact that NASA has a Hypersonic Technology Project. When research is complementary and when it is duplicative is not always apparent to the proposer without knowledge of all other NASA research initiatives. The committee concluded that there is a knife-edge balance between complementary and duplicative work and that proposers are confused as to where the gap is. The phrase “complementary research” in this goal could be replaced with “cutting-edge or transformative research” that is complementary to ARMD’s ongoing research activities. Even with the funded projects, it was not clear to the committee how complementary the ongoing funded projects are, since the PIs are expected to set their own research paths independent of NASA’s researchers and perhaps with limited knowledge of the actual ongoing research activities under each of ARMD’s six strategic thrusts. It appears that potential PIs are only provided a broader view of the thrusts during the applicants’ workshop conducted online by NASA, perhaps once or twice a year. In addition, the applicants’ workshop contains primarily topical and programmatic information about the ULI application process, with little information on how to navigate the process of generating a compelling idea for a future proposal. One way to address this would be to also include some of the specific ongoing research activities under each thrust during NASA’s presentations, as well as the name and contact details of NASA’s research group leaders. The agenda of the workshop could also be expanded to include presentations from NASA researchers, industrial partners, and other funded PIs. It would be helpful if more effort could be put in to help potential applicants understand the anatomy of a good technical idea for formulation toward a compelling ULI proposal. The current approach of conducting the applicants’ workshop online is not only a necessity due to the meeting restrictions caused by the pandemic but may also have benefits in the post-COVID future. However, the agenda could be revised. At present, only three items are included on the agenda, with the program manager being perhaps the only speaker (i.e., ULI solicitation overview, topics, and question/answer). This could be expanded to include presentations from selected researchers from each of ARMD’s thrusts to provide a summary of specific current and future research activities, as well as presentations from potential industrial partners to avoid duplicative research proposals. At a minimum, NASA could provide a summary of ongoing research activities under each strategic thrust with contact details of group leaders so that potential PIs could interact with them prior to developing a full proposal if they have any doubts regarding whether or not their proposed projects are complementary to or duplicative of NASA’s ongoing research activities. RECOMMENDATION: NASA should better inform the community of what research ARMD is already supporting in each of the six strategic thrusts so that proposers can avoid proposing work that might duplicate existing initiatives. NASA should provide a clear indication of what research it would consider “duplicative.” To broaden the perspective, the committee also heard from several proposers from multiple universities who commented on feedback from unsuccessful proposals. They said that some feedback had been unhelpful and questioned whether the reviewers understood the proposed topic. Having the appropriate technical experts reviewing the proposals is key to the credibility of the ULI program. They commented that in some cases, it appeared that raw comments were not reviewed before being sent back to the proposer. One university proposer stated that they had received comments back with one aspect of the proposal listed as both a “strength” and as a “weakness.” Poor comments discourage subsequent proposals, particularly in light of the significant amount of work that the ULI proposals require of the university/industry teams. While unsuccessful proposers may be more forthcoming with criticism of ULI, the committee believed it important to mention this because the proposers stated that this had undermined the credibility of the program. The committee heard that it would be more helpful if NASA program officers provided consensus feedback based on the reviews, which is done for National Science Foundation (NSF) panels. Obviously, care must be taken to ensure that the original intents of the proposal

ASSESSING ULI’S PROGRESS TOWARD MEETING ITS GOALS 9 reviewers are not lost or distilled in the synthesized consensus feedback. Perhaps some hybrid model of raw feedback and combined consensus feedback could be considered. Of course, review processes are both imperfect and challenging, and the ULI review feedback may be on par with other programs. Again, this feedback is included for completeness. Proposers noted that better feedback on how to improve subsequent submissions would have also been useful in order to understand what caused the proposal not to be selected. More clarity about the criteria used to evaluate proposals would be beneficial. As noted above, a strategic goal of the ULI program is to transition proposed efforts into NASA’s portfolio or directly to industry at the end of the program. Proposers stated that it was not clear what NASA’s expectations were for how this handoff was supposed to work. This may also contribute to industry’s reluctance to participate, especially the new AAM start-ups, which are often funded by venture capital investments and very protective of their work. Many of these companies are not yet at the stage where they are thinking about the long-term payoff of precompetitive collaboration. Proposers also suggested that better clarity was needed regarding how much the expectation is that the effort will focus on research versus how much on broader educational involvement, graduate and undergraduate extensive curriculum additions, diversity, and K–12 outreach. Again, NSF was suggested as an example, stating that the NSF Engineering Research Center program also stresses these broader educational goals but provides a specific budget for them. Proposers noted that the ULI solicitation language focuses on creating new systems and is not particularly accommodating of proposing something that would instead address difficulty in making a safety case or certifying a new capability. Proposers also desired clarity on intellectual property (IP). For a company to be involved in a project with the intention of seeing a new technology that could be implemented widely, it would want IP protection, potentially conflicting with both academic practices and government sponsorship. Finally, proposers cited clarity on technical risk: the proposals require very specific milestones and metrics, which tends to steer proposals toward much lower program risk. FINDING: The committee found that there was significant confusion among proposers regarding the ULI program, with proposers citing concerns about feedback that was unclear and conflicting advice from reviewers. Proposers indicated that greater clarity and transparency in the proposal process was desired. RECOMMENDATION: NASA should examine the solicitation and review process and materials to improve clarity on  Research versus educational goals,  New systems versus other goals,  Intellectual property, and  Technical risk. This will help ensure that ARMD provides consistent and useful information to the proposers. NASA could also review the process used by NSF for guidance on how to do this. While it may be outside the control of the ULI program, NASA could work backward from the end of spring to schedule the review process steps so as to maximize the opportunities for recruitment of graduate students. Finally, some of the PIs that made presentations to the committee were also concerned about proposal submission deadlines. Specifically, the current ULI deadlines for proposal submission and awards could be aligned with a typical university calendar, which is similar for most U.S. research-based institutions. It appeared from the presentations that the current awards are made at a time that is too close to the resumption of an academic year (usually in August); thus, it is likely difficult to recruit sufficient graduate students during the first year. For example, the deadline for Step B submission for the current ULI solicitation is November 2020. By the time the award is made and announced (Spring 2021), it will be

10 ASSESSING NASA’S UNIVERSITY LEADERSHIP INITIATIVE more challenging for the PIs to recruit graduate students for the next academic year (Fall 2021) as the deadline for application submission may have already passed. Strategic Goal 2: Transition research results to an appropriate range of stakeholders that lead to a continuation of the research. Transition can occur in a number of ways, including the following:  Creates a new product line in U.S. industry or a new ARMD project,  Whole ULI concept is transitioned to U.S. industry/ARMD project,  Part of the ULI concept is transitioned to U.S. industry/ARMD project, and  ULI findings impact direction of U.S. industry/ARMD. FINDING: On this ULI goal, the committee concluded that it was premature to assess success. It is clear that the performers are making progress in their research, but none of the efforts have reached completion as of yet. The committee notes that another perspective on evaluating the success of these goals might be to look where the students go upon graduation. For example, how many of the students initiate a start-up or go to work for one? Industry partnerships on university research are useful as a talent pipeline for training and hiring students for the partnered company, which in many cases may be more useful for that company than the actual research that is done. NASA could track students after their graduation to see if they stay in that field or with that industry partner. Technology transfer can also occur when professors on ULI projects engage in start-up activities, which is discussed in Chapter 5. In addition, while it was not in the purview of the committee to recommend restructuring the ULI program, the committee notes that some of the funded projects are in the realm of basic research, while others are applied research. Some of the PIs who spoke to the committee suggested restructuring of the ULI program to a two-phase program, with the first phase dedicated to basic research and the second phase devoted to technology transfer and transition. Indeed, technology transfer may be a more valuable and more easily measurable goal. (Technology transfer is further discussed in Chapter 5 of this report.) The committee provides the following observations on this potential construct. Structuring the ULI program as a two-phase program potentially leads to a more realistic achievement of its goals of supporting cutting-edge basic research and technology transfer. The current one-phase award with a project duration ranging from 3 to 5 years may not be sufficient to achieve highly transformative research with the potential of technology transfer and/or commercialization. There are two major components of a ULI project: (1) basic research and education and (2) technology transfer and/or commercialization. The present timelines for research implementations, coupled with the challenges associated with COVID-19, may make it difficult to truly bridge the gap between the implementation of the basic research and technology transfer. Based on the presentations made by some of the PIs, it appears that having two phases of ULI may be superior. Phase 1 (initial award) would focus on basic research and education and explore the potential for technology transfer, while Phase 2 (a contract renewal with additional contributions from industrial partners) would focus on the technology development/transfer and/or commercialization. Phase 2 may also be very important in the long run because it appears that there is no specific plan for continuity after the completion of projects in the current ULI approach. The committee also noted that basic research (which can result in published papers) would be more beneficial to junior faculty members striving for promotion. Strategic Goal 3: Provide broad opportunities for students at different levels, including graduate and undergraduate, to participate in aeronautics research. The committee sought data from NASA on how many graduate and undergraduate students were supported by the 13 funded projects, as well as how many of these are minorities. NASA was unable to provide the requested data to the committee. Because of this, the committee asked some of the project PIs

ASSESSING ULI’S PROGRESS TOWARD MEETING ITS GOALS 11 to comment on these demographic issues, and they indicated that their projects are indeed reaching a large number of students at different levels, including minority students. However, it was not possible for the committee to draw specific conclusions about minority representation among all the projects. ULI has provided funding for dozens of universities to conduct aeronautics research, partnered with industry. ULI currently has five projects underway in Round 1, three projects in Round 2, and five projects in Round 3. The briefings provided to the committee indicated that large numbers of undergraduate, masters, and Ph.D. students and postdocs are involved in the ULI projects, which cover broad topic areas. These briefings also indicated that some projects offer more opportunities to graduates versus undergraduates than other projects. This is to be expected given the different nature of the research. Opportunities for students have many different dimensions, which are explored in different sections of this report. FINDING: The committee concluded that NASA is indeed achieving the goal of providing broad opportunities for students at different levels (graduate and undergraduate). However, the opportunities vary substantially on a project-to-project basis. Strategic Goal 4: Promote greater diversity in aeronautics through increased participation of Minority-Serving Institutions and underrepresented university faculties in ULI activities. The committee agrees that this is an important goal for the ULI program and for NASA. The committee applauds ARMD for initiating ULI with this as a top goal and recognizes that NASA must fit the selected projects into a research strategy. As a result, some of the four strategic goals may at times be at cross purposes, and NASA has to balance these. NASA stated that it has adjusted the solicitation language and the evaluation criteria after early rounds to improve diversity. NASA also informed the committee that it believes that these changes have improved the diversity of the ULI program and that NASA was “satisfied” with the results of these changes. The committee agrees that NASA’s changes have resulted in some limited improvement in diversity in some aspects. However, the committee concluded that there is still more that NASA can do to increase participation of MSIs and underrepresented university faculties in ULI activities. For example, no female researchers have been selected as PIs in the ULI program. The number of funded projects from minority schools is also very small. Only one PI is from an MSI. The first three rounds of ULI support included one PI from an MSI, representing less than 8 percent of funded teams. During this period, a total of five funded projects included co-PIs from MSIs. FINDING: In the committee’s assessment, NASA has not achieved a sufficient degree of diversity in the ULI program. This is further discussed in following chapters of this report.

Next: 2 Efforts to Increase the Participation of Women, Historically Black Colleges and Universities, and Underrepresented and Minority Participants in ULI »
Assessing NASA's University Leadership Initiative Get This Book
×
Buy Ebook | $9.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

NASA created the University Leadership Initiative (ULI) to engage creative and innovative minds in the academic arena to identify significant aeronautics and aviation research challenges and define their unique approach to their solution. The ULI was started in 2015 as part of the larger University Innovation Project, with the goal of seeking new, innovative ideas that can support the U.S. aviation community and NASA's long-term aeronautics research goals, as established by its Aeronautics Research Mission Directorate.

Assessing NASA's University Leadership Initiative reviews the ULI and makes recommendations to enhance program's impact to benefit students, faculty, industry, and the U.S. public.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!