KEY POINTS IN THIS CHAPTER
- Scientific research furthers national goals in many arenas, including the economy, national security, energy, health, the workforce, the environment, technical infrastructure, and agriculture.
- The American research enterprise is a highly complex system. Given this complexity, a desired effect (for example, increased output of research discoveries with commercial value) is unlikely to be achieved by changing one or even a few components of the system without regard to its critical drivers and their interrelationships.
- Understanding the research enterprise as a system could inform policies designed to enhance the benefits of publicly funded research for U.S. society and the global community. Changes to individual components made without this understanding may result in unintended and undesirable effects.
- The societal benefits of federal research investments can best be enhanced by focusing attention on three fundamental drivers, or pillars, of the research system: (1) a talented and interconnected workforce, (2) adequate and dependable resources, and (3) world-class basic research in all major areas of science.
For more than 65 years, the United States has led the world in science and technology. Discoveries from scientific research have extended understanding of the physical and natural world, of the cosmos, of society, and of humans—their minds, bodies, and economic and other social interactions. Through these discoveries, science has yielded a broad range of benefits (see Box 1-1). For example, scientific advances have enabled longer and healthier lives, provided for better education through the science of learning, enhanced the national economy, and strengthened America’s position in the global economy. These advances also have furthered national security and energy independence and, of particular note, have enabled the United States to remain at the forefront of global competition for commercially viable technologies and other innovations. Today, in the face of increasing competition from other nations, a highly effective and productive research enterprise is more essential than ever.
BOX 1-1
Some Benefits of Scientific Research
Economy—Provides high-tech, high-paying jobs, as well as hundreds of thousands of related jobs.
Contributions include sustained job creation by universities and other research institutions, marketable technologies and innovations, and increased economic competitiveness.
Energy—Produces efficient sources of energy and decreases the nation’s dependence on foreign fuels.
Contributions include nuclear power, biofuels, and hybrid powertrain technology.
National security—Improves national security by enhancing the nation’s ability to defend its shores and its cyberspace.
Contributions include better weapons systems, mechanisms for the rapid deployment of troops, compact explosive detectors, the Global Positioning System, and encryption technologies.
Environment and natural resources—Safeguards the nation’s food and water supplies and protects its air quality; ensures abundant natural resources for present and future generations.
Contributions include cool roofing materials; shale oil recovery; and devices for detecting contaminants in water, food, and air.
Health—Enhances health care and therapies to improve health, lengthen lives, and reduce disabilities.
Contributions include computational chemistry in pharmaceutical development, personalized medicine, and nuclear magnetic resonance imaging.
In the current environment of budgetary austerity, many question why the federal government should continue to invest heavily in scientific training, capacity, and research given countless other pressing priorities. Congress and others have shown increased interest in measuring the economic and societal returns on federal research investments. These returns are reflected in part in the value of new goods and services and in the ability of new technologies to drive the American economy, increase the nation’s standard of living, and create jobs. Because of the importance of scientific research and the new technologies and other innovations to which it leads, the Bureau of Economic Analysis and others have argued that research spending should be considered an investment rather than an expense (see Box 1-2).
While the contributions of scientific research are vast and multifaceted, influencing almost every aspect of daily life, the question arises of
Training and workforce—Develops and attracts people of talent, imagination, and intellect who enrich U.S. society and advance the nation’s science enterprise. Contributions include international research partnerships and collaborations, and greater sharing of research methods, techniques, and technical resources such as stock cultures.
Agriculture—Develops hybrid foods that enhance yields, improve nutrition, and alleviate global food shortages.
Contributions include genetically modified plants and remote sensing technology for site-specific fertilization and irrigation.
Infrastructure—Improves communications and interconnectedness among diverse peoples; improves human productivity and creates more leisure time as machines and technologies ease human workloads.
Contributions include the Internet, information retrieval and search technology, bioinformatics, and the Hubble space telescope.
Social innovation—Increases the speed of communication and collaboration among geographically distant individuals.
Contributions include crowdsourcing, social entrepreneurship, and social media informatics.
Policy—Enables analyses to support formulating, implementing, and evaluating public policies.
Contributions include methods for policy analysis and evaluation.
BOX 1-2
Treating Research as an Investment
Since July 2013, the U.S. Bureau of Economic Analysis, in its national economic accounts, has been treating business spending on research and development and the creation of intellectual property related to some creative and artistic works as investments rather than expenses. Calculations using the new definition showed that U.S. gross domestic product—the official measure of the market value of all goods and services produced in the United States—would have been an average of 2.7 percent larger each year during 1998 to 2007 than previously reported (U.S. Bureau of Economic Analysis, 2010).
whether these contributions can be measured in a meaningful and accurate way to guide future federal investments in the research enterprise. Is it possible to provide a clear picture of the returns on those investments? And how can analyses of such returns guide the development and implementation of policies designed to maintain the nation’s global competitiveness?
PURPOSE AND SCOPE OF THIS STUDY
This study was requested by Congress in the America COMPETES Act (P.L. 111-358), which became law on January 4, 2011. Seeking to increase the returns on federal investments in scientific research, Congress asked the National Academies to study metrics that could be used to gauge the impacts of scientific research on society. Of particular interest were metrics that could serve the goal of increasing the translation of research into commercial products and services. Interest in measuring the benefits of federally funded research arose also in part from a desire to enhance accountability and to provide guidance for federal research investments in stringent budgetary times.
To carry out this study, the National Research Council of the National Academies formed the Committee on Assessing the Value of Research in Advancing National Goals. The committee’s statement of task (see Box 1-3) reflects recognition of the limitations of metrics for the specific purposes set forth in the America COMPETES Act (as discussed further below). Accordingly, the committee was tasked with investigating some of the many pathways through which research contributes to the nation’s economy and well-being, as well as other national goals. In particular, the committee was asked to address how research contributes to human and knowledge capital in government and private business through the training of a research workforce. In exploring these questions, the committee
BOX 1-3
Statement of Task
A diverse panel of distinguished experts would be appointed by the NRC to conduct the study. Its membership would reflect experience in science, engineering, business, education, and the public sphere. The panel would investigate some of the many pathways through which research contributes to our economy and well-being and serves other national goals. A particular pathway of interest is how research contributes to human and knowledge capital in government and private business through the training of a research workforce. For its investigations of these pathways, the panel may commission case studies, including one or more that trace a successful innovation back to the basic and other research discoveries and ideas that enabled its development. The panel would address the technical and other measurement issues for assessing (1) the quality of research output of universities and other research institutions receiving federal government support and (2) the potential societal impact of research in advancing national goals. For this purpose, the study would:
- Review and synthesize the broad variety of efforts to assess research output and impacts in industry and government, including those of the National Science Foundation (NSF) National Center for Science and Engineering Statistics and the NSF-National Institutes of Health STAR METRICS Program.
- Review the experiences of other countries, in particular the United Kingdom, in assessing the impacts of research.
- Draw upon relevant current and past NRC studies, including those of the Committee on Science, Engineering, and Public Policy.
- Explore various methodological approaches to measurement of research quality, productivity, and impact, including those that help to quantify the return on investment of research.
- Examine the explanatory power, predictive ability, and incentive effects of various measures of the potential impact of research on society, in particular to better understand the contributions and limits of these measures in science and innovation policy decisions.
The panel would also identify the research and data needed to develop measures or other means to assess the impact of research in advancing national goals and recommend priorities. The purview of the panel would be all federally supported research, but the panel may need to be selective in examining some research areas in depth. If so, the panel’s choices would range from basic research, which contributes to fundamental understanding, to use-inspired basic research, as described in “Pasteur’s Quadrant.” The panel may also recommend how research could be better organized and supported to better contribute to national goals. For this purpose, the panel would be asked to think creatively about how the U.S. research enterprise can further adapt to a competitive and highly linked world with a globalized economy.
was charged with identifying the technical and other measurement issues entailed in assessing the quality of the research output of universities and other research institutions (e.g., laboratories operated by the federal government and private industry) that receive federal support.
In the America COMPETES Act, Congress explicitly asked how metrics could be improved or developed to measure the potential impact of research on society. In this regard, the wording of the committee’s statement of task captures an important distinction. Whereas the legislative request for this study used the term “metrics,” the statement of task uses the term “measures.” The committee understands and uses the former term as a quantitative value, which may capture the performance of a system or of its outputs and outcomes at a point in time or over a period of time. We understand and use the term “measures” more broadly, encompassing not only quantitative but also qualitative values that, for example, provide a basis for rankings or comparisons, such as an assessment of the caliber of research performed in one country versus another.
As discussed in detail in Chapter 4, metrics have many uses, as well as significant limitations. Whereas metrics are commonly used to measure numbers of patents, publications, and other easy-to-count items, this report describes the broader, more useful applications of measures in assessing research portfolios—a research program, a collection of grants, or scientific research managed by a federal agency or private entity (see Chapter 5)—and the progression from idea to product in various phases of research. Although currently available metrics for research inputs and outputs are of some use in measuring aspects of the American research enterprise, they are not sufficient to answer broad questions about the enterprise on a national level.
Implicit in the request from Congress was a broader charge: assessing how well the United States is achieving the benefits of science. A holistic understanding of how these benefits arise can provide greater insight into the drivers of the system, and reveal how best to sustain and reap further benefits from the research enterprise. Specifically, it is necessary to understand how the discovery, dissemination, and application of knowledge gleaned from scientific research generate new technologies and other innovations, and how the ultimate value of these innovations depends on their widespread adoption and use. This understanding can be gained by taking a systems perspective—that is, by considering research and innovation as inextricably connected, yet distinct, systems of components that interact in unpredictable and often difficult-to-specify ways to shape the research enterprise and the nation’s economy as a whole.
HOW A SYSTEMS PERSPECTIVE CAN HELP
From the scientific study of systems theory, researchers know that “emergent” phenomena1—effects sparked by interactions among components of a system—are enabled by the actions of the system as a whole; they cannot be explained by the behaviors or properties of its components. Therefore, all components of a system must be treated carefully to stimulate desirable effects while avoiding unintended and undesirable consequences that can arise from modifying one component without considering how the system might react (Jervis, 1997; Meadows, 2008).
The systems of research and innovation are no different. The research system is defined by the breakthrough discoveries that occur when many talented researchers and institutions generate knowledge in all scientific fields through basic and applied research. The innovation system produces advances—some of which may be revolutionary—both within the research system and beyond, relying on networks of institutions and researchers to integrate, transform, and disseminate discoveries in diverse fields (OECD, 1997). The innovation system also encompasses aspects of development, which enables the production of new technologies, products, processes, and other innovations of economic value. (See Box 1-4 for definitions of research, innovation, and development. See Chapter 3 for a more detailed description of the pathways from research to development.)
This report focuses on the benefits of federally funded research to society and on the various handoffs to and from the innovation system, including spin-offs, start-ups, technology transfer activities, proof-of-concept research, and public-private or regional partnerships. Chapter 2, for example, includes a discussion of how proof-of-concept research spans the gap between research and innovation. The report focuses primarily on basic and applied research, not on development as it may relate, for example, to the acquisition of venture capital, marketing, manufacturing, or other factors critical to the success of an innovation. However, some figures include both research and development in an effort to draw certain comparisons, and in some cases, the available data do not distinguish between research and development or are not broken down by type of research.
Intuitively one might think that basic science research should proceed directly to applied research, to an invention, and to an innovation that is developed and adopted to produce economic or other societal benefits. However, the progression from research to ultimate benefits encompasses
_____________
1An example of an emergent phenomenon is superconductivity, the property of being able to carry electrical currents with no dissipation of energy, which is exhibited by some metals below a critical temperature. Its discovery ultimately enabled magnetic resonance imaging that revolutionized modern medicine (National Research Council, 2007a).
BOX 1-4
Definitions of Research, Innovation, and Development
“Research is defined as systematic study directed toward fuller scientific knowledge or understanding of the subject studied. Research is classified as either basic or applied according to the objectives of the sponsoring agency” (National Science Foundation, 2007).a
- “Basic research is experimental or theoretical work undertaken primarily to acquire new knowledge of the underlying foundations of phenomena and observable facts, without any particular application or use in view” (OECD, 2002a).b
- “Applied research is original investigation undertaken to acquire new knowledge. It is, however, directed primarily toward a specific practical aim or objective” (OECD, 2002a).c
“An innovation is the implementation of a new or significantly improved product (good or service) or process, a new marketing method, or a new organisational method in business practices, workplace organisation, or external relations” (OECD, 2002a).d
The national innovation system has been defined slightly different by various groups, yet all of the definitions listed below share an emphasis on the interactions and relationships among multiple, divers, organizations or institutions (OECD, 1997):
- “. . . the network of institutions in the public and private sectors whose activities and interactions initiate, import, modify and diffuse new technologies” (Freeman, 1987, p. 1).
several interim steps and the development of multiple technology elements that combine to produce the eventual innovation. This process rarely, if ever, follows a linear progression. In fact, quite the opposite is frequently true.
For some research, it may be relatively straightforward to predict the near-term outcomes, yet it is extremely difficult to predict how research knowledge might be taken up and used, by whom, and in what ways, on the path to societal and economic benefits. One would have difficulty predicting, for example, when a feedback loop might carry findings from proof-of-concept research2 back to the basic science laboratory bench
_____________
2As we define and use the term later, proof-of-concept research is research conducted to establish the commercial viability of an invention.
- “. . . the elements and relationships which interact in the production, diffusion and use of new, and economically useful, knowledge . . . and are either located within or rooted inside the borders of a nation state” (Lundvall, 1992, p. 2).
- “. . . a set of institutions whose interactions determine the innovative performance . . . of national firms” (Nelson, 1993, p. 4).
- “…the national institutions, their incentive structures and their competencies, that determine the rate and direction of technological learning (or the volume and composition of change generating activities) in a country” (Patel and Pavitt, 1994, p. 12).
- “. . . that set of distinct institutions which jointly and individually contribute to the development and diffusion of new technologies and which provides the framework within which governments form and implement policies to influence the innovation process. As such it is a system of interconnected institutions to create, store and transfer the knowledge, skills and artefacts which define new technologies” (Metcalfe, 1995, pp. 462-463).
“Experimental development is systematic work, drawing on existing knowledge gained from research and/or practical experience, that is directed to producing new materials, products, or devices; to installing new processes, systems, and services; or improving substantially those already produced or installed” (OECD, 2002a).e
_____________
aAvailable: http://www.nsf.gov/statistics/nsb1003/definitions.htm [June 2014].
bAvailable: http://stats.oecd.org/glossary/detail.asp?ID=192 [June 2014].
cAvailable: http://stats.oecd.org/glossary/detail.asp?ID=120 [June 2014].
dAvailable: http://stats.oecd.org/glossary/detail.asp?ID=6865 [June 2014].
eAvailable: http://stats.oecd.org/glossary/detail.asp?ID=908 [June 2014].
to overcome an unexpected hurdle, which might then spark a new line of basic research, as well as require applied or further proof-of-concept research. Equally difficult is predicting whether that research will eventually lead to development, and even then, whether it will be successfully commercialized or embraced by society. And it is very difficult as well to predict how long this journey will take.
Clearly, the systems of research and innovation—and the many unpredictable pathways that connect them—are exceedingly complex. Yet while these pathways appear to be unpredictable almost to the point of being chaotic, one truth has been demonstrated time and again: from this complexity springs possibility; from this unpredictability, the systems give rise to transformative innovations.
Over time, then, the American research enterprise has evolved into a highly complex and dynamic system and it has adopted many of the characteristics of free enterprise. It is decentralized. It is pluralistic, with a diverse array of researchers, companies, institutions, and funding agencies. It is competitive, requiring researchers and organizations to compete for funding, for talent, for positions, for publications, and for other rewards. It is meritocratic, bestowing more significant rewards on those with highly competitive ideas and abilities through a built-in quality control system of peer review. And finally, it is entrepreneurial: it allows for risk taking, for facing the prospect of failure head on to reap potentially great rewards. This complexity and dynamism make the American research enterprise successful, but the poorly understood relationships among its many components mean that well-intentioned reforms (for example, favoring some scientific disciplines to increase the output of research discoveries of commercial value) could lead to unintended and undesirable effects.
THREE PILLARS OF THE AMERICAN RESEARCH ENTERPRISE
Understanding the American research enterprise requires knowledge of the fundamental drivers, or pillars, of the research system and their interrelationships. The committee identified three pillars on which we focus throughout this report: (1) a talented and interconnected workforce, (2) adequate and dependable resources, and (3) world-class basic research in all major areas of science. To understand how these pillars interact to produce research discoveries, one must understand how knowledge flows among networks of individuals and institutions; how research is influenced by the availability of funds and the methods, instrumentation, and other means used to conduct research; how accomplishing world-class basic research is affected by management, research environments, institutions, and peer review; and how these and other aspects of the pillars interrelate.
In using the term “pillars,” we refer to broad components of a complex system, each of which encompasses many aspects. The pillars should not be mistaken for a model of research or innovation because relationships among them are constantly changing and because there are no dependent variables, as each pillar feeds back into the others. Since the pillars are critical drivers of the system of research and provide important preconditions for innovation, each proposed policy or other change to
one pillar or any aspect thereof should account for the likely downstream effects on other pillars and on the system as a whole.3
The committee concluded that societal benefits from federal research can be enhanced by focusing attention on the above three crucial pillars of the research system: a talented and interconnected workforce, adequate and dependable resources, and world-class basic research in all major areas of science. We argue in this report for creating or adapting measures that can be used for this purpose. The pillars are described briefly here and in detail in Chapter 6.
A Talented and Interconnected Workforce
Talent encompasses not only science, technology, engineering, and mathematics (STEM) education and research training, but also many other aspects of the system, including inspiring young men and women to pursue STEM careers; attracting immigrants with technical skills; developing professional networks and partnerships; and supporting research environments that nurture the creativity, ingenuity, and passion of talented researchers. Highly trained talent is essential to sustain the American research enterprise. People amplify and expedite the nation’s capacity for innovation by generating knowledge; distributing it through colleges, universities, publications, and other means; and transforming it through networks of individuals with varying perspectives and creative ideas. Absent a strong pool of scientists and engineers familiar with research at the cutting edge, scientific research and its products are unlikely to be developed and applied in ways that create value for society.
Adequate and Dependable Resources
Certainly research depends on adequate and dependable funds. But resources encompass much more—in particular, access to scientific infrastructure, or the tools and organizations that allow for research excellence, including national and other laboratories, major research instruments such as the Hubble telescope and the General Social Survey, world-class research universities, and other research organizations. Adequate and dependable resources provide critical support for the process of research, encourage students to pursue STEM careers, encourage estab-
_____________
3As discussed in Chapter 6, the “health” of different fields of the research enterprise is sensitive to trends in federal research funding, especially fluctuations that produce a research “feast and famine” cycle. Federal research funding influences the career choices of prospective future scientists, their employment prospects, and the decisions of university and other research administrators on complementary investments in research facilities and personnel. See, for example, Alberts et al. (2014).
lished researchers to continue in their careers, and attract foreign talent. Thoughtful public and private investment enables the United States to maintain cutting-edge information technology and other scientific infrastructure, to maintain the best possible pool of talent, and to sustain world-class scientific institutions and means of communication.
In short, the historically robust American research enterprise requires a stable, reliable stream of investment to sustain the continued flow of discoveries necessary to ensure the nation’s welfare in both the near and distant future—a flow that can take years, often decades to prime and pump. And those investments must be broadly distributed across all major fields of science, not just those for which direct and near-term economic benefits are foreseen.
World-Class Basic Research
Basic research, in which investigators pursue their ideas primarily for fundamental understanding and not necessarily for a technological objective or other application, may advance national goals by leading directly to new technologies and other innovations, just as applied research occasionally leads to fundamental advances in scientific understanding. More often, however, basic research lays the foundation for economically significant future innovations. In addition, public investment in basic research contributes to the growth of a trained research workforce by developing their talent, abilities, knowledge, skills, experience, and professional connections, and enables American researchers and would-be innovators to exploit the worldwide networks of researchers who advance the scientific enterprise and open access to a vast stock of knowledge and technological approaches offering opportunities for commercialization.
ORGANIZATION OF THIS REPORT
This report presents the committee’s argument for enhancing the societal benefits of federal research investments by focusing on the three pillars of talent, resources, and basic research:
- Chapter 2 presents an overview of the evolution of the American research enterprise.
- Chapter 3 describes some of the many ways in which research contributes to the nation’s economy and societal well-being; examines the complex, lengthy, and often unpredictable pathways from research to innovation; and reveals how basic research plays a key role in the ultimate realization of societal benefits.
- Chapter 4 explores the usefulness and limitations of metrics in measuring the returns on research investments.
- Chapter 5 describes several studies that have attempted to measure research impacts and quality.
- Chapter 6 presents the committee’s argument for cultivating a holistic understanding of the research system through a focus on talent, resources, and basic research.
- Chapter 7 presents the committee’s overarching conclusion in the context of the key points of the report.
