Maximizing Program Effectiveness Through Strategic Management
This chapter discusses an approach that NASA officials should consider as they address concerns identified in the committee’s assessment of the mission-enabling programs in Chapter 2 and as they apply the principles and metrics discussed in Chapter 3. The chapter also discusses specific management issues for three key areas that were singled out in the committee’s charge—innovative research, interdisciplinary research, and workforce development.
TRACEABILITY OF MISSION-ENABLING ACTIVITIES FROM STRATEGIC GOALS
The committee believes that NASA mission-enabling program managers should have a clear and publicly stated set of strategic goals and priorities for each SMD mission-enabling program. Currently, these goals and priorities appear to be privately held rather than publicly stated. Each manager has a plan, but these plans are neither clear to outside researchers and policy makers nor openly articulated and advocated by SMD’s senior management. Open, explicit articulation of mission-enabling priorities will lead to a more relevant SMD program that can also be more readily understood, adjusted in response to community input or new technical developments, and defended. The conclusion of the 1998 NRC report on NASA research and data analysis programs1 that “[mission-enabling activities are] not always thoroughly and explicitly integrated into the NASA enterprise strategic plans and that not all decisions about the direction of [mission-enabling activities] are made with a view toward achieving the goals of the strategies” is still the case today.
However, it is not sufficient to determine that a mission-enabling activity supports a strategic goal of the agency. The committee provides an approach for using a traceability matrix as an example of how SMD could derive the range and scope of mission-enabling activities via a systematic flow-down analysis of all tasks and requirements needed to address SMD strategic goals. The level of detail can be taken to the program element level and even lower. Activities can be grouped together to define program elements. Each activity can be prioritized, budgeted, and related to assessment criteria. Because of differences in the strategic goals across SMD divisions, traceability matrices logically would be designed to the division level, and cross-discipline and cross-division activities can be identified by their multiple appearances. This traceability matrix provides a mechanism that allows balance to be
determined among different activities by weighting costs by duration and priority. Mission-enabling activities can then be consolidated into higher-level tasks and into program elements through which activities are funded. The generation of such a traceability matrix can be an important tool for the active management of mission-enabling activities. It provides a means by which missing activities can be identified, optimal funding levels can be determined and compared with current budgets, and performance can be evaluated.
Because SMD strategic goals are fundamentally scientific and wide-ranging, it is appropriate for this traceability matrix to be generated with active input from and ongoing evaluation by the science community in conjunction with NASA management. The process is also a dynamic one such that tasks and requirements in the mission-enabling portfolio flowing down from these goals will themselves be constantly refreshed by new information and understanding arising from the execution of mission-enabling activities and missions.
A traceability matrix begins with an articulation of individual strategic goals and then identifies the full range of high-level mission-enabling tasks required to address those goals. Each task is then further reduced to general subtasks. Each subtask is broken down into specific sub-subtasks as appropriate. Individual requirements are identified for the lowest-level subtasks to be executed. Mission-enabling activities are those needed to satisfy a requirement. Each activity can then be prioritized, budgeted over an anticipated duration or recurring period, and tagged for inclusion in (or across) specific program elements. Criteria by which the mission-enabling activity can be assessed can also be identified and used as part of management of the activity. These activities can then be rolled up within their subtasks and tasks for the purpose of understanding and tracking current and future efforts at different levels and their relative and total cost.
A detailed example, following a single hypothetical thread in the Planetary Science Division, is illustrated in Appendix C.
HIGH-RISK/HIGH-PAYOFF RESEARCH AND TECHNOLOGY DEVELOPMENT
The committee is convinced that, while most of the SMD mission-enabling research budget should be clearly directed at supporting specific goals of the various science divisions, NASA can benefit from separately funded and protected mission-enabling activities that pursue high-risk/high-payoff advanced technologies or other research activities that could produce game-changing results. This approach is consistent with the majority of successful R&D enterprises where some defensible fraction of resources is allocated to research activities for which there is not an immediate predictable benefit, but which can potentially be the basis for important solutions to future problems or which open up future opportunities.
In NASA’s case, where mission development times can run as long as 7 to 10 years, such an approach is especially important because the phasing-in of new technologies can be 5 to 10 times longer than time to market in the commercial and university research sectors. Thus, one can envision progressively falling behind the state of the art in many technical areas unless technology is purposefully captured and utilized through an active mission-enabling research program.
Management of such research activities poses unique challenges because NASA’s technical development activities typically are structured to limit risk. This is especially true in mission-focused technology development activities where the fundamental objective is risk mitigation through the maturation of technologies to TRL-5 or TRL-6 prior to mission confirmation. As opposed to a risk mitigation strategy, game-changing research should be viewed as a mission-enabling strategy that is focused on the TRL-1 to TRL-4 elements with the objective of identifying and nurturing ideas and technologies that could serve SMD’s future missions one to three generations ahead of existing technical approaches, system architectures, and design methods.
In general, successful outcomes in these types of activities are happy accidents often resulting from cross-cutting work in multiple disciplines. The key element to any incubation of new ideas is to ensure that a broad range of seeds are both protected and nurtured. Thus, funding stability and creative management capable of taking a 20- to 30-year perspective are essential components for a successful program. Benchmarking relevant practices in other organizations that perform R&D provides some insights for consideration by NASA.
Benchmarking Relevant Practices of Other Organizations
In industrial and military research organizations with which committee members are familiar, anywhere from 5 to 10 percent of their respective research and technology budgets are allocated to high-risk, high-payoff (i.e., potential breakthrough/game-changing) research activities. The DOD’s service research laboratories provide one example of a budget allocation philosophy that explicitly supports high-risk, high-payoff endeavors as defined above. More specifically, examining the science and technology (S&T) enterprises of DOD’s three services—Army, Navy, and Air Force—indicated the following:
Fifteen to 20 percent of the entire S&T budget for the Army and the Air Force (25 to 30 percent for the Navy) is devoted to basic research, whose purpose is to discover new phenomena and acquire fundamental knowledge that can enable new technologies and military capabilities. Within the basic research spectrum of activities, 33 percent (Navy) to 50 percent (Air Force) of the 6.1 budget is allocated for “high-risk research,” which represents approximately 5 to 10 percent of the total S&T budget of each of these services.
The specific amount of the overall research budget, and that portion to be invested in “high-risk” research activities, is typically a top-down decision. This budget is fenced and protected even as the overall S&T budgets rise and fall, and it cannot be used for focused technology development to enable specific mission capabilities.
Basic research programs are often managed by an organizational entity separate from those entities that manage the much larger technology-related budget elements (6.2 and 6.3) that collectively constitute the overall S&T budget. A separate organizational entity helps ensure that basic research funds do not migrate to nearer-term, higher-visibility projects. The assigned personnel are also uniquely qualified to understand the nuances of managing a portfolio of diverse, high-risk projects, which often do not lend themselves to precise definition or traditional project management methods.
A second opportunity for benchmarking comes from using the National Academies’ report Rising Above the Gathering Storm.2 Major ideas encompassed in the report’s recommendations include the following:
Approximately eight percent of a government entity’s S&T budget should be allocated to high-risk, high-payoff research.
Government agencies must take the lead in funding such research, since industry R&D is predominantly short-term in nature.
Additional government budget is probably not needed for high-risk research; rather, it is a matter of ensuring that about eight percent of the existing S&T budget is not committed to specific mission programs and is set aside for high-risk research, and that the barriers which discourage government program managers from pursuing high-risk research are eliminated for this portion of the S&T investment portfolio. Barriers to performing high-risk research include (1) a peer review system that tends to favor established investigators using well-known methods, (2) pressure from customers and management for short-term results, and (3) risk averseness, because high-risk projects are prone to failing and increased government and public scrutiny make “projects that fail” increasingly untenable.
Appendix D presents a more detailed discussion on benchmarking results with respect to allocation of resources for high-risk, high-payoff research endeavors.
Organization and Management
In many cases, innovative research activities that are interdisciplinary may not fit clearly into one specific SMD science division. For example, game-changing research related to micro-thrusters, formation flying, micro-electro-mechanical-system valve devices, nano-materials, micro-instrumentation, radiation-hardened components,
and many other technologies have applicability across all divisions such that there would be value to having a directorate-wide program for such activities. There are a number of possible options for SMD to consider for funding and incubating such very early technical developments.
Programs within each science division
High-risk/high-payoff projects could be supported within existing research and technology program elements. Program managers could allocate to activities of this type some fraction of the funds for individual programs and make awards on the basis of peer review of this aspect of the program in the context of the overall activity, or they could allocate some small fraction of program funds independent of the peer review of proposals for projects in the core program.
An alternative approach would be a program within each SMD division similar to the NSF Small Grants for Exploratory Research and Early Concept Grants for Exploratory Research (i.e., small, fast-track, precompetitive projects). Small grants could be awarded at the discretion of the program manager to individuals whose proposals are not externally peer reviewed. The advantage of this approach is that all high-risk/high-payoff projects for a science division would be consolidated into a single program for the division, thereby providing for more efficient management, a critical-mass-size funding pool, and the opportunity to look more broadly across the division’s activities for the most attractive opportunities for this kind of work.
Cross-disciplinary programs managed centrally outside the science divisions
As noted above, some high-risk/high-payoff research is inherently cross-disciplinary or will include technology developments that have the potential to benefit more than one SMD science division. An important consideration is whether critical mass on such research investments can be achieved with relatively limited funding in each discipline as opposed to an aggregated sum serving all disciplines under dedicated management through the SMD chief scientist or some other office. This approach was utilized by SMD’s predecessor organizations for an innovative research program in the 1980s and for cross-cutting technology development projects in the late 1990s. Given the current national focus on innovative science and technology research, a clearly identified high-risk/high payoff research budget might also be more likely to attract new funding compared to smaller amounts partitioned between multiple discipline areas.
Several universities and other research centers throughout the country have a substantial number of scientists involved in a variety of NASA-funded research activities, not all funded by the same SMD division. These centers of excellence can be fertile grounds for innovative research and technical ideas. Typically, such early-stage research and ideas require a modest amount of start-up money to bring them to a level where they can realistically compete for peer-reviewed funding from a research and analsis (R&A) program of one of the SMD divisions. An option for providing this necessary seed money would be an innovative research block grant to such a university center or research institute, giving the center director discretion to allocate funds from this grant to several internal subgrants for early-stage technical developments. Internal grants would be awarded based on relatively simple internal proposals, typically not as detailed or well supported as a proposal to NASA.
Finally, the committee calls attention to a significant difference between the flexibility that some large entities, such as industrial firms and federal laboratories, have to self-fund some innovative research and the lack of that flexibility at universities. The former can include funding for innovative research in their overhead rates, e.g., via internal research and development funds from government contracts in industry and via directors’ discretionary funds in the latter, and offer internally competed opportunities that satisfy this need. Although some universities return a fraction of their overhead to investigators to cover innovative research, in general such returns are very small because the overhead was collected to cover other legitimate university expenses.
There are important opportunities to promote interdisciplinary research activities in space and Earth sciences. In some areas, such as the astrobiology program,3 NASA deserves credit for having created an entirely new field of research and stimulated a vigorous new research community. SMD also has been active in fostering interdisciplinary programs within Earth science. Similarly, many areas of planetary science are intrinsically interdisciplinary. In other areas, such as Sun-climate studies, it is less clear that an adequate interdisciplinary program exists.
In a management structure based on discipline-oriented divisions with limited resources, it is not surprising that interdisciplinary activities may get inadequate attention. Program managers may look at cross-discipline efforts as diversions that are not appreciated or rewarded by their upper management. Thus, a change in strategic approach will be required before program management ownership will be encouraged and rewarded. The key problem with both cross-discipline and high-risk/high-payoff efforts is that the path to success can be unclear and the final result unknown. In fact, by some metrics the majority can be judged as failures even though there may be end-game spin-offs or spillovers that are extremely successful.
Real game-changing research breakthroughs are especially likely to come from cross-discipline activities, because the boundaries between disciplines are the most fertile place for important discoveries.4 Therefore, program managers should be encouraged to “mine along the boundaries.” Associated with the exploitation of disciplinary boundaries is the exploitation of generational boundaries. Teams that have an effective mix of both young and mature engineers and scientists are the most effective at making technological leaps or leaps in scientific understanding. Thus, it is critical that early-career technologists be recruited and retained as part of the process.
The National Science Foundation is focused on basic research; therefore, it is more straightforward for NSF to foster interdisciplinary research than for NASA SMD, which is organized around space missions. NSF often issues cross-division and even cross-directorate solicitations for proposals specifically for interdisciplinary research. For example, NSF’s initiative in cyber-enabled discovery is specifically designed to bring together researchers from computer science and “domain” science (e.g., oceanography, biology, materials science, and so on).
However, except for these inherently interdisciplinary solicitations, interdisciplinary NSF proposals that are submitted in competition with traditional, discipline-based proposals do not necessarily fare well in the peer-review process. Such interdisciplinary proposals are often reviewed by two or more panels, thereby increasing the likelihood of a less than favorable review. On the other hand, NSF leadership does encourage its program managers to move outside the discipline boundaries for innovative interdisciplinary proposals, and that does counterbalance the tendency for lower rankings by panels. Consequently, the overall success of interdisciplinary proposals is not significantly lower than that of discipline-based proposals.
This only serves to emphasize the need for established goals for the mission-enabling activities of SMD, which are endorsed by the senior management of NASA and for which there is adequate staff to initiate and manage new activities.
DEVELOPING AND SUSTAINING A HEALTHY TECHNICAL WORKFORCE
Every SMD division’s research community, both extramural and intramural, has some level of workforce deficiencies or rising demands. For example, there might be a lack of critical-mass expertise in spaceflight hardware design and development, or a need to shore up shortfalls in certain science skill areas, or to plan for succession in a field where the scientific or engineering leaders are all approaching retirement.5 Supporting development
and maintenance of a workforce able to execute the program is certainly an appropriate mission-enabling activity. In the committee’s view, workforce development and maintenance should be a priority for mission-enabling activities, and efforts should be made to assess workforce status, define workforce needs, and manage the mission-enabling activities to meet these needs. Furthermore, awareness of the impact of funding volatility on the pool of capable researchers necessary to conduct SMD research activities and how this might vary from division to division are important issues. In areas where NASA is the principal source of funding, funding volatility may have a disproportionately negative impact on the corresponding workforce. Newly funded areas may not be able to tap a sufficient population of scientists and engineers to make progress. Sudden decreases in funding in another area may cause scientists to exit that area or new graduates to not enter it, disrupting the workforce for years to come. Workforce awareness and management are necessary components essential to the effective management of a science enterprise.
The committee envisions a three-part approach that can help SMD better utilize its mission-enabling activities to anticipate and address critical workforce issues. All three elements of this approach can benefit from participation by the outside scientific community via the involvement of advisory committees.
Identification of workforce needs both in the short term and in the long term
The first step should be to identify needs for key skill areas based on the current situation and the situation expected over the next decade. For instance, when mission-enabling activities are defined through a traceability exercise, there should be an assessment of the workforce required to execute those activities over the anticipated duration of the activity. The likely necessary distribution of skill areas (e.g., key areas in science, engineering, technical support, and technical management), age diversity (based on expected future needs), and location diversity (i.e., NASA centers, academia, industry) are all important factors. These factors can also constitute later workforce development effectiveness metrics.
Consideration should be given to the stability and sustainment of capabilities. For example, if fewer new missions are currently being developed, there is a risk of losing highly skilled engineers and technicians. Mission-enabling activities can help provide some continuity of funding to retain people. This can be achieved through technology development programs and, to some extent, suborbital programs. Likewise, any planning for significant increases in the volume and diversity of the data obtained from missions must include an assessment of the sufficiency of scientists capable of generating benefit from the acquired data and whether to expand that workforce accordingly. Some stable level of the workforce also should be dedicated to working on fundamental research (both in technology and in basic science).
Shaping programs and strategies for meeting workforce needs
As opportunities and options are selected for applying mission-enabling programs to address workforce development needs, there are several factors to consider:
Steady funding for specific types of programs, including technology development, suborbital flights, and basic research (the “build it and they will come” approach),
Programs specifically aimed at training and retention of early-career people—internships at NASA centers, graduate fellowships, early-career awards, Jet Propulsion Laboratory strategic partnerships, and support of the suborbital program,
Strategies to enhance workforce diversity,
Selection criteria for proposals in some programs that could formally include workforce considerations (e.g., training of students), and
Encouraging development of critical-mass research groups in NASA centers or universities and other research centers—in many cases, these groups will have unique capabilities (building spacecraft instrumentation,
building numerical models). Incentives could include special programs or funding vehicles for research groups (e.g., block grants).
The effectiveness of programs and approaches implemented in the step immediately above must be assessed regularly (preferably every 2 or 3 years). Effectiveness should be determined by measuring against well-defined metrics, such as age distribution and evolution over time and the rate of students and young scientists being trained through the programs. The evaluation process could include a workforce review board or could be carried out by each division director. The needs and the effectiveness of programs should be considered at both the division and the agency level. The ability to forecast workforce needs is an important tool.