Committee on Assessing the Value of Research
in Advancing National Goals
Richard F. Celeste, Ann Griswold, and Miron L. Straf, Editors
Division of Behavioral and Social Sciences and Education
NATIONAL RESEARCH COUNCIL
OF THE NATIONAL ACADEMIES
THE NATIONAL ACADEMIES PRESS
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001
NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.
This study was supported by Grant No. SMA-1215040 between the National Academy of Sciences and the National Science Foundation. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the organizations or agencies that provided support for the project.
International Standard Book Number-13: 978-0-309-30522-8
International Standard Book Number-10: 0-309-30522-5
Library of Congress Control Number: 2014952746
Additional copies of this report are available from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.
Copyright 2014 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
Suggested citation: National Research Council. (2014). Furthering America’s Research Enterprise. R.F. Celeste, A. Griswold, and M.L. Straf (Eds.). Committee on Assessing the Value of Research in Advancing National Goals, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.
THE NATIONAL ACADEMIES
Advisers to the Nation on Science, Engineering, and Medicine
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. C. D. Mote, Jr., is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Victor J. Dzau is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. C. D. Mote, Jr., are chair and vice chair, respectively, of the National Research Council.
This page intentionally left blank.
COMMITTEE ON ASSESSING THE VALUE OF
RESEARCH IN ADVANCING NATIONAL GOALS
Richard F. Celeste (Chair), President, Colorado College (Emeritus)
Rodney A. Brooks, Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology (Emeritus)
Alicia Carriquiry, Department of Statistics, Iowa State University
Christopher M. Coburn, Partners Healthcare, Boston, MA
Stephen E. Fienberg, Department of Statistics, Heinz College, and Machine Learning Department, Carnegie Mellon University
Bronwyn H. Hall, Department of Economics, University of California, Berkeley, and University of Maastricht, the Netherlands
John E. Kelly, III, IBM Research, IBM, New York
Josh Lerner, Harvard Business School
David C. Mowery, Walter A. Haas School of Business, University of California, Berkeley
Jason Owen-Smith, Department of Sociology, Organizational Studies Program, Institute for Social Research, and Barger Leadership Institute, University of Michigan
John Edward Porter, Hogan Lovells, Washington, DC
Stephanie S. Shipp, Virginia Bioinformatics Institute, Virginia Tech
Gregory Tassey, Economic Policy Research Center, University of Washington
Jeffrey Wadsworth, Battelle Memorial Institute
David Ward, Chancellor, University of Wisconsin–Madison (Emeritus)
Miron L. Straf, Study Director
Steven Ceulemans, Consultant
Ann Griswold, Science Writer
Viola Horek, Manager of Operations
Mary Ann Kasper, Senior Program Assistant
This page intentionally left blank.
The America COMPETES Act,1 which reauthorized the National Science Foundation (NSF), became law on January 4, 2011. The act required NSF to contract with the National Academies for a study to evaluate, develop, or improve metrics for measuring the potential impact of research on society. The language of the act is as follows:
“SEC. 521. STUDY TO DEVELOP IMPROVED IMPACT-ON-SOCIETY METRICS.
(a) IN GENERAL.—Within 180 days after the date of enactment of this Act, the Director of the National Science Foundation shall contract with the National Academy of Sciences to initiate a study to evaluate, develop, or improve metrics for measuring the potential impact-on-society, including—
(1) the potential for commercial applications of research studies funded in whole or in part by grants of financial assistance from the Foundation or other Federal agencies;
(2) the manner in which research conducted at, and individuals graduating from, an institution of higher education contribute to the development of new intellectual property and the success of commercial activities;
1H.R. 5116, P.L. 111-358.
(3) the quality of relevant scientific and international publications; and
(4) the ability of such institutions to attract external research funding.
(b) REPORT.—Within 1 year after initiating the study required by subsection (a), the Director shall submit a report to the Senate Committee on Commerce, Science, and Transportation and the House of Representatives Committee on Science and Technology setting forth the Director’s findings, conclusions, and recommendations.
The origin of the study was an amendment introduced by Senator Amy Klobuchar of Minnesota, then chair of the Subcommittee on Competitiveness, Innovation, and Export Promotion of the Senate Committee on Commerce, Science, and Transportation, and Senator George LeMieux of Florida, then subcommittee ranking member.
In discussions with the leadership of NSF, the National Academy of Sciences, the National Academy of Engineering, and Senator Klobuchar’s office, the limitations of metrics for the specific purposes of the legislation were noted. Accordingly, the decision was made to broaden the study while still addressing the intent of the legislation. The agreed-upon statement of task appears in Chapter 1.
With funding from NSF, the National Research Council (NRC) convened a committee to conduct a study responding to this charge. This report is the result of that study.
The committee met four times during 2013, sometimes calling on other experts to address specific topics. We also had the benefit of many contemporaneous conferences, workshops, and meetings involving committee members or staff. In addition, we benefited from myriad studies focused on quantifying the impacts of research, in particular in Australia, Canada, and the United Kingdom, which were summarized for and reviewed by the committee. A number of these studies are recent (see Appendix C). Moreover, previous NRC studies have addressed a similar charge, and we benefited from those studies as well.
The committee was fortunate to have a diverse and knowledgeable membership. The members brought many different perspectives to this study. Their expertise encompassed federal and state government policy making in research and innovation, research administration in academia and industry, and entrepreneurship in engineering and the life sciences. Members also represented a variety of academic research and expertise, including metrics, measurement, and statistics; the economics of technological innovation; the translation of university sciences into commercial technologies; and networks and the organization of research.
Despite the plethora of studies on the impacts of research, we believe this study brings to bear a fresh approach informed by a more holistic understanding of the research enterprise as a complex, dynamic system. As documented in this report, this understanding illuminates why America’s research expertise has historically been so successful; where attention should be focused in examining the societal benefits of research investments; and how those who make decisions on the allocation of funds for scientific research will best carry out that task by understanding the many pathways by which those benefits are generated, the extent to which the potential to yield those benefits can be characterized if not quantified, and the usefulness and limitations of metrics for this purpose.
This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the NRC’s Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process.
We thank the following individuals for their review of this report: William F. Brinkman, Physics Department, Princeton University; E. William Colglazier, U.S. Department of State; Rita R. Colwell, Center for Bioinformatics and Computational Biology, University of Maryland; Gordon R. England, President’s Office, E6 Partners LLC, Fort Worth, Texas; Donna K. Ginther, Center for Science Technology and Economic Policy, Institute for Policy and Social Research, University of Kansas; Robert L. Jervis, Department of Political Science, Columbia University; Paul L. Joskow, President’s Office, Alfred P. Sloan Foundation; John A. Montgomery, U.S. Naval Research Laboratory; Arogyaswami J. Paulraj, Department of Electrical Engineering, Stanford University; Barbara A. Schaal, Department of Biology, Washington University in St. Louis; Jeannette M. Wing, Research International, Microsoft Research; Andrew W. Wyckoff, Economic Analysis and Statistics Division, OECD; and Richard N. Zare, Department of Chemistry, Stanford University.
Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the report’s conclusions, nor did they see the final draft of the report before its release. The review of this report was overseen by Lawrence D. Brown, Department of Statistics, The Wharton School, University of Pennsylvania, and Susan Hanson, Department of Geography, Clark University. Appointed
by the NRC, they were responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authoring committee and the institution.
Richard F. Celeste, Chair
Miron L. Straf, Study Director
Committee on Assessing the Value of Research
in Advancing National Goals
Many individuals and organizations contributed to this study. Special recognition goes to Charles M. (“Chuck”) Vest, who, as president of the National Academy of Engineering, played a critical role in the development of this study and the formation of the committee. He met with us and followed our work until his death in December 2013. A former president of the Massachusetts Institute of Technology, he understood the importance of research and of research universities.
We thank our sponsor, the National Science Foundation (NSF), and many on its staff for their support, as well as their assistance in the development and conduct of this study. We are particularly grateful for the leadership of Subra Suresh, former NSF director. Others on the NSF staff who contributed include acting NSF director Cora Marrett, David Croson, Joshua Rosenbloom, Jennifer Thornhill, and Joanne Tornow.
Robert D. Atkinson, president of the Information Technology and Innovation Foundation (ITIF), provided valuable advice in the development of the study. Together with Stephen Ezell and Kathryn Angstadt on the ITIF staff, he provided much helpful information.
A number of people directing National Academies boards or studies were especially helpful. Kevin Finneran, director of the Committee on Science, Engineering, and Public Policy (COSEPUP), and Stephen
A. Merrill, director of the Board on Science, Technology, and Economic Policy, worked with us from the formative stages of the study to the development of this report. Chair Richard N. Zare and other members of COSEPUP provided helpful advice, as did Constance F. Citro, director of the Committee on National Statistics.
The National Academies’ Government-University-Industry Research Roundtable convened many conferences and webinars on topics in our purview, and its director, Susan Sauer Sloan, saw that we were represented at them. Anthony Boccanfuso, executive director of the University-Industry Demonstration Partnership, provided many helpful contacts and saw that we were informed about his organization’s studies on university-industry partnerships. Kaye Husbands Fealing, director of the Committee on National Statistics’ Panel on Developing Science, Technology, and Innovation Indicators for the Future, presented findings of that panel’s study and provided other helpful information for this report. Others at the National Academies who helped us in our work include Maria Lund Dahlberg, COSEPUP research associate; Nancy F. Huddleston, communications director of the Division on Earth and Life Studies; and James C. Lancaster, director of the Board on Physics and Astronomy.
In addition to Chuck Vest, other leaders of the National Academies were helpful in developing and guiding this study. We thank in particular Ralph C. Cicerone, president of the National Academy of Sciences; C. Dan Mote, Jr., president of the National Academy of Engineering; and Robert M. Hauser, executive director of the Division of Behavioral and Social Sciences and Education.
Many others also contributed to this study. Those who made formal presentations to the committee are listed in Appendix D on the study process, which also lists the organizations of others who contributed. To these we add Richard C. Atkinson, president emeritus of the University of California; William B. Bonvillian, director of the Massachusetts Institute of Technology Washington Office; Laura Hillier, director, evaluation and outcome assessment, Canada Foundation for Innovation; Marina Volkov, acting director of the National Institutes of Health’s Office of Science Management and Reporting; Andrew W. Wyckoff, director of the OECD Directorate for Science, Technology and Industry, and Yu Xie, Otis Dudley Duncan distinguished university professor of sociology, statistics, and public policy, University of Michigan.
We had the benefit of an exceptional staff led by study director Miron L. Straf. Our science writer, Ann R. Griswold, worked with us at all our meetings; summarized myriad studies for our review; and prepared drafts of this report based on presentations, discussions, text prepared by our members and staff, and her own research. Others on our staff
who provided assistance were Steven Ceulemans, initially as Christine Mirzayan science and technology policy graduate fellow and later as a consultant; Meredith B. Blake, consultant; Viola Horek, manager of operations; and Mary Ann Kasper, senior program assistant. Kirsten Sampson Snyder marshaled our report through National Academies review, and the report was edited by Rona Briere.
Those involved in the review of our report are acknowledged in the preface. They made many constructive comments.
Finally, we acknowledge the service and contributions of our committee members, who freely gave of their time and effort as a public service.
We are grateful to all who contributed to this study and this report.
This page intentionally left blank.