Since the publication of Vannevar Bush’s 1945 report Science: The Endless Frontier and the creation of the National Science Foundation in 1953, the federal government has supported scientific research for societal benefit far beyond the initial purposes of national defense (Bush, 1945). The benefits gained have manifested and today include computers, the Internet, wireless communication, the laser, the global positioning system, and modern medicine, among many others. These advances have enabled the United States to achieve unprecedented prosperity, security, and quality of life.
Now, however, the nation faces increased global competition for new technologies and other innovations, even as it confronts growing economic exigencies. In this context, Congress wants to enhance the benefits of science for the U.S. economy and the advancement of other national goals—in particular, keeping the nation at the forefront of the global competition for new technologies and other innovations—while maximizing the efficiency and effectiveness of federal research investments.
In seeking to increase the returns on federal investments in scientific research, Congress asked the National Academies to study measures of the impacts of research on society. Of particular interest were measures that could serve to increase the translation of research into commercial products and services.
The committee’s investigation revealed that measures can usefully quantify research outputs, such as publications, and technology transfer, such as patents and licenses. They can be used to assess the outcomes of
some research areas, particularly applied research focused on a specific goal, and the impact of a university’s research on the regional economy. Measures of these important activities can serve to increase the societal benefits of research. At the same time, we agree with a common finding that metrics of research impacts must be viewed with considerable caution and that assessments, therefore, require both metrics and professional judgment. But current measures are inadequate to guide national-level decisions about what research investments will expand the benefits of science.
The American research enterprise is indeed capable of producing increased benefits for U.S. society, as well as the global community. To reap those benefits, however, requires new measures to guide federal research investments. To develop those measures, it is necessary to understand what drives the American research enterprise and what has made it so productive.
A SYSTEMS PERSPECTIVE
To understand how federal investments in scientific research result in societal benefits, it is necessary to understand the American research enterprise as a system that must be viewed in relation to the innovation system in which the discoveries produced by research are used to develop new technologies and other innovations. Without this system-level understanding, policies focused on relatively narrow objectives—such as increasing university patenting and licensing of research discoveries or reducing the funding for certain disciplines or types of research—could have undesired consequences.
Such an understanding, however, is not easily achieved. Discoveries often emerge from the highly complex and dynamic research enterprise as a result of the system as a whole. They are not due to any individual component of the system and thus cannot be predicted from the nature of the components. Nor can one predict how the knowledge from a research discovery might eventually be taken up and used, by whom, and in what ways that will lead to a transformative innovation. Indeed, research discoveries and the innovations to which they lead often arise serendipitously. The complexity of the research and innovation systems is why attempts to trace major innovations back to their original supporting research have rarely if ever revealed a direct flow of money in, value out.
This complexity means that a desired effect of the research system (for example, increased output of research discoveries of commercial value) is unlikely to be achieved by changing one or even a few components without regard to the critical drivers of the system and their interrelationships. Because of the complex and continually changing interactions
among components of the research system, a change in one component will lead to changes in others, often in unpredictable ways, and may have untoward effects.
THREE PILLARS OF THE RESEARCH SYSTEM
Significant opportunities exist to increase the societal benefits of scientific research and to inform policies designed to keep the United States at the forefront of global competition for new technologies and innovations. Moreover, measures can be developed to guide the effective implementation of such policies.
To these ends, however, it is necessary to take a systems perspective on the research enterprise, identifying the critical drivers, or pillars, of the research system and understanding the relationships among them. The committee identified three pillars of the research system: a talented and interconnected workforce, adequate and dependable resources, and world-class basic research in all major areas of science. These pillars are supported by an active, nongovernmental entrepreneurial community that is willing to make significant investments in research. This community is one external factor that—along with antitrust, regulatory, and intellectual property policies; venture capital; and other factors—creates a unique environment for the U.S. research enterprise relative to those of other nations.
The committee concludes that societal benefits from federal research can be enhanced by focusing attention on the three crucial pillars of the research system: a talented and interconnected workforce, adequate and dependable resources, and world-class basic research in all major areas of science.
A systems perspective also reveals how these three pillars interact to produce research discoveries: how knowledge flows among networks of individuals and institutions; how research is influenced by the availability of scientific infrastructure, funds, and other resources; how world-class research and the usefulness of research discoveries are affected by management, research environments, institutions, and peer review; and how these and other aspects of the three pillars interrelate.
A strong interplay among talent, resources, and basic research is critical. Talent is at the center: people generate knowledge; distribute it through colleges, universities, publications, and other means; and transform it through networks of individuals with varying perspectives and creative ideas. Without such human capital, the products of research cannot be applied in ways that create value for society. Maintaining broad expertise among those who conduct research also sustains the innovation system, because technological problems often arise in the development
of an innovation that requires research for their solutions. Research and innovation are symbiotic in this way. Similarly, many aspects of manufacturing contribute to and draw on research.
Adequate and dependable federal funding can ensure balance and vitality within the system of research, ensuring the continual, competitive flow of discoveries in the near and distant future. Critical resources for research also include scientific infrastructure, or the tools that allow for research excellence, and world-class research universities, national laboratories, and other research institutions. All of these resources provide essential support for the research process.
Basic research supports the critical pillar of talent by promoting national and international networks of researchers, through which ideas flow and scientific resources are shared. Basic research is performed primarily in universities, which nurture talent and launch future researchers. World-class basic research in all major areas of science is important for three major reasons.
First, truly transformative scientific discoveries often depend on research in variety of fields. Mathematics, statistics, and computer sciences, for example, helped advance magnetic resonance imaging and other medical technologies, while the social sciences contributed to an effective allocation of the spectrum for wireless communications.
Second, in today’s rapidly connected world, a discovery made somewhere is soon known everywhere. The competitive advantage may go not to the nation in which the discovery was made but to the nation that can use it more effectively to develop new technologies and other innovations. Reaping the benefits from research discoveries throughout the world requires a highly sophisticated domestic research enterprise built on people, infrastructure, and funding. In particular, awareness of scientific discoveries may travel quickly, but sufficient understanding to extend them or to apply them for the development of new technologies or other innovations often requires that the nation’s researchers possess considerable fundamental knowledge derived from diverse basic research.
Third, world-class basic research attracts researchers from around the world. These researchers further excellence in research, thus creating a self-reinforcing cycle.
Moreover, another benefit of research—particularly basic research—is its contribution to scientific infrastructure. Basic research generates—and benefits from—new methods of observation, measurement, data collection, analysis, and experimentation, enabling the quality of research to improve continually and the extent of research to expand.
HIGH-RISK RESEARCH
Not all research achieves its intended goals; high-risk research inevitably results in some failures. But even failures can lead to unanticipated discoveries and steer research in new directions. The transformative innovations that eventually result from some high-risk research can more than justify the investment in other such research that may fail. Only government has the broad social purpose and long horizon to invest in high-risk research so that society can reap its ultimate benefits. In some cases, government also is called upon to fund proof-of-concept research aimed at determining the commercial viability of an invention, at least to the extent of reducing the risk to the point where private industry would invest.
EVALUATION OF RESEARCH PROGRAMS
The standard review mechanism for prospective evaluation of research grant and contract proposals is some form of peer review and assessment. Some have criticized peer review for discouraging the funding of high-risk research or radically new research approaches. More recently, others have criticized it for the dilution of expertise in the National Institutes of Health review process. Yet despite the need for improvements in the peer review process, and especially in light of the decreasing success rate for research proposals, experience with the widespread use of alternative mechanisms by public agencies is limited, and little existing evidence suggests that there is generally a better mechanism. Nonetheless, peer review is designed to assess not overall program effectiveness but investigator qualifications and the innovativeness of individual projects within a given research program, and typically is most appropriate as a means of awarding funding rather than assessing performance.
Evaluation of research programs also faces challenges of attribution of observed outputs, outcomes, or performance. Randomized experiments could play a role here, but may be feasible only for comparing individuals or groups within a research area. Even so, very little evaluation has been conducted through randomized experimentation, and we believe there are opportunities to expand the use of this method. We encourage continuing to experiment with modifications of this approach to evaluation for both prospective and retrospective assessments.
Regardless of what approach to prospective evaluation of a research funding program is explored, it is preferable to build evaluation into the program from the very beginning. Doing so helps clarify goals and expectations and allows for the collection of important data that might otherwise be missed.
In today’s global economy, the United States is challenged to maintain a competitive position in the development of new technologies and other
innovations to achieve national economic goals, especially for employment and income. The levels, composition, and efficiency of federally funded research need to be adjusted to meet today’s circumstances. Measures can inform policy decisions to effect such adjustments. The United States, however, lacks an institutionalized capability for systematically evaluating the nation’s research enterprise as a whole, assessing its performance, and developing policy options for federally funded research.
THE NEED FOR IMPROVED MEASURES
Measures of research activities, outputs, and technology transfer are important, and both the measures and the underlying data need to be improved. We see opportunities for improving ongoing data collection efforts. For example, Science and Technology for America’s Reinvestment: Measuring the Effect of Research on Innovation, Competitiveness and Science would be more valuable if its data had more complete coverage, were linked to other data sources, and were made more accessible to researchers. We also see new areas in which measures can be developed, such as in the analysis of networks. But greater benefits can be realized by focusing attention on the three pillars of the research enterprise detailed above: talent, resources, and basic research. Measures designed around these pillars would promote a better understanding not only of these critical components and how they relate to each other, but also of the research enterprise as a system. These measures might include, for example, indicators of human and knowledge capital, indicators of the flow of knowledge in specific fields of science, indicators that can be used to track the flow of foreign research talent, portfolio analyses of federal research investments by field of science, international benchmarking of research performance, and measures of research reproducibility. Further research and data are needed, but a major contribution to that effort has been made by the National Research Council (2014) report Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy.
CONCLUDING OBSERVATIONS
The U.S. research enterprise is a complex, dynamic system in part because it has evolved with many of the characteristics of free enterprise: it is decentralized, pluralistic, competitive, meritocratic, and entrepreneurial. In this complex system, it is impossible to predict what innovations may eventually result from research discoveries or which types of research would, in the absence of other types, lead to transformative innovations. Attention to the pillars of talent, resources, and basic research will ensure that discoveries and innovations continue to emerge from the
scientific enterprise. Measures designed around these three pillars would promote a better understanding of the U.S. research enterprise as a system. These measures could be used to guide federal research investments that would enable the system to yield more of the societal benefits that have made it the world’s premier scientific research enterprise.
