C
Annotated Bibliography of Selected Studies
1. American Academy of Arts and Sciences. (2012). Advancing Research in Science and Engineering (ARISE) II: Unleashing America’s Research & Innovation Enterprise. Initiative on Science, Engineering, and Technology Oversight Committee. Cambridge, MA: Author. Available: http://www.amacad.org/arise2.pdf [August 2014].
This report explores avenues in which changes in U.S. policies and practice might have the greatest impact on research innovations. Because transdisciplinary and transsector research play particularly important roles in advancing scientific discovery, the report recommends pursuing a deep conceptual and functional connectedness across scientific disciplines (particularly the physical and life sciences) and fostering cooperative, synergistic interactions among academia, government, and the private sector.
2. Council of Canadian Academies Expert Panel on Science Performance and Research Funding. (2012). Informing Research Choices: Indicators and Judgment. Ottawa: Council of Canadian Academies. Available: http://www.scienceadvice.ca/uploads/eng/assessments%20and%20publications%20and%20news%20releases/science%20performance/scienceperformance_fullreport_en_web.pdf [August 2014].
This report reviews the development in other countries of performance indicators of scientific research and their use in allocating funds. The panel concludes that several performance metrics are available; however, no single indicator, set of indicators, or assessment strategy offers an ideal solution in research assessment contexts for natural sciences and engineering discovery research. In light of this observation, the panel recommends four guiding principles to support research funding agencies undertaking science assessments in support of budget allocation: context matters, do no harm, transparency is critical, and expert judgment remains invaluable.
3. Group of Eight. (2011). Measuring the Impact of Research: The Context for Metric Development. Go8 Backgrounder 23. L. Rymer, principal author. Turner, Australia: Author. Available: http://www.go8.edu.au/__documents/go8-policy-analysis/2011/go8backgrounder23_measimpactresearch.pdf [August 2014].
This report presents a careful and critical review of the different approaches to measuring research impacts. Measuring return on investment can be tricky because the impact of research depends on a complex web of factors, such as how quickly the scientific community becomes aware of the findings, the success of follow-up research, how quickly the findings are put to practical use, the likelihood of success (high-impact research often entails greater risk), and how “positive impact” is defined. Having issued that warning, the report describes how the impacts of scientific research can be grouped into eight broad categories: effective teaching; advances in knowledge; encouraging additional investment by other parties; financial returns; and economic, social, environmental, and intangible (e.g., national reputation) outcomes. Impact in these categories can be assessed using the following methods, although the authors emphasize that none of the current measures can provide definitive results: input measures, output measures and benchmarking (e.g., bibliometric measures), peer review by expert panels, researchers’ anecdotes about the benefits of their work, detailed case studies of research outcomes, cost-benefit analyses, hindsight studies, surveys (e.g., stakeholder surveys to assess the perceived significance of a project; commercialization surveys to quantify staff, spin-off companies, and patents), economic models, and econometric analyses.
4. Guthrie, S., Wamae, W., Diepeveen, S., Wooding, S., and Grant, J. (2013). Measuring Research: A Guide to Research Evaluation Frameworks and Tools. Santa Monica, CA: RAND Corporation. Available: http://www.rand.org/pubs/monographs/MG1217.html [August 2014].
This study analyzes six research evaluation frameworks in various countries, also providing a brief overview of eight additional frameworks. It presents a guide to the key considerations entailed in developing approaches to research evaluation. The report also describes several tools used in research evaluation. The report emphasizes that perennial challenges to research evaluation (e.g., attribution, contribution, time lag between research and outcome, level of performance) need to be addressed in the development of evaluative methods. Furthermore, frameworks and tools should be tailored to the purpose of the evaluation and the type of material being evaluated. Research evaluation tools typically fall into one of two groups: formative tools, which are flexible and able to deal with cross-disciplinary and multidisciplinary assessment; and summative tools, which do not require judgment or interpretation and are quantitative, scalable, transparent, comparable, and suitable for high-frequency longitudinal use. These two types of evaluation tools serve different needs; multiple methods are required if researchers’ needs span both groups. The report notes further that research evaluation approaches should suit their wider context. Different approaches may be acceptable and credible in different environments, and it is important to consider this when developing a framework.
5. Hughes, A., and Martin, B. (2012). Enhancing Impact: The Value of Public Sector R&D. Cambridge, UK: CIHE-UK~IRC Task Force on Enhancing Value: Getting the Most Out of UK research. Available: http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&ved=0CDEQFjAA&url=http%3A%2F%2Fukirc.ac.uk%2Fdocs%2FCIHE_enhancing_Impact_Report.pdf&ei=qg7KUJCWGPSG0QGwtYCgBg&usg=AFQjCNE4YWYfGKTwm93yuTO8XRBv2fSGtQ&bvm=bv.1355272958,d.dmQ [August 2014].
This is the second in a series of reports exploring how the United Kingdom can gain the most value from publicly funded research. The report stresses the importance of moving from simple measures of impact, such as university spin-offs and patents, to a more nuanced understanding of the connections between the
public and private sectors in a system of knowledge production and innovation. The report concludes that narratives, rather than economic values, may be the most effective way to assess the impacts of research and the pathways to these impacts. The narrative format would allow for a description of the various factors influencing impact and therefore avoid many of the challenges inherent in developing useful metrics.
6. Mazzucato, M. (2011). The Entrepreneurial State. Demos. London, UK: Magdalen House. Available: http://marianamazzucato.com/projects/entrepreneurial-state/ [August 2014].
In this paper, Mazzucato argues that opportunities are being missed if recent developments in the innovation literature, economic theory in general, and experience from elsewhere in the world are not considered in setting UK policy. The paper aims to provoke a radical change in the understanding of the government’s role in economic policy. The author hopes to spark a conversation about how the state can use its power to specify the problems it wishes to solve through technological advances and innovation, thereby ensuring that those advances are able to take place. The paper concludes that a more entrepreneurial economy would be beneficial to the United Kingdom, and that such an economy would not necessarily require the British government to withdraw but to lead. The paper provides 10 recommendations for increasing innovation through various efforts, including policy changes, tax incentives, and elimination of existing roadblocks.
7. Martin, B.R., and Puay, T. (2007). The Benefits from Publicly Funded Research. Brighton, UK: Science and Technology Policy Research Unit, University of Sussex. Available: http://www.erawatchnetwork.eu/reports/sewp161.pdf [August 2014].
This paper concludes that there is no simple answer to the question, “What are the economic and social benefits of basic research?” The authors note that the benefits of publicly funded research come in a variety of forms, flowing through a variety of channels and over differing time scales. Seven relatively distinct mechanisms or “channels” are described through which benefits from research flow into the economy and society. The findings reported show that the benefits are substantial, certainly sufficient to justify considerable government investment in basic
research. The findings reveal seven main mechanisms or “exploitation channels” through which the benefits of basic research may flow to the economy or to society at large: (1) increase in the stock of useful knowledge, (2) supply of skilled graduates and researchers, (3) creation of new scientific instrumentation and methodologies, (4) development of networks and stimulation of social interactions, (5) enhancement of problem-solving capacity, (6) creation of new firms, and (7) provision of social knowledge.
8. National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. (1993). Science, Technology, and the Federal Government: National Goals for a New Era. Committee on Science, Engineering, and Public Policy. Washington, DC: National Academy Press. Available: http://www.nap.edu/catalog.php?record_id=9481 [August 2014].
This report recommends that the United States be among the leaders in all major areas of science, and notes that the nation’s ability to achieve world-class basic research could be tracked with the qualitative metric of international benchmarking. In particular, the report suggests that maintaining a world standard of excellence in all fields will help ensure that the United States can “apply and extend scientific advances quickly no matter when or where in the world they occur.” To this end, the federal investment must be vigorous enough to support research across the full spectrum of scientific and technological investigation.
9. National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. (2000). Experiments in International Benchmarking of U.S. Research Fields. Committee on Science, Engineering, and Public Policy. Washington, DC: National Academy Press. Available: http://www.nap.edu/catalog.php?record_id=9784 [August 2014].
This is a follow-up to the 1993 National Research Council report Science, Technology, and the Federal Government: National Goals for a New Era, summarized above. In this report, international benchmarking is used to assess U.S. performance in the fields of immunology, materials science and engineering, and mathematics. The report identifies eight factors predicted to have the greatest influence on the quality of future U.S. research performance relative to other nations: (1) the intellectual quality of researchers and the ability to attract talented researchers; (2) the
ability to strengthen interdisciplinary research; (3) the ability to maintain strong, research-based graduate education; (4) the ability to maintain a strong technological infrastructure; (5) cooperation among the governmental, industrial, and academic sectors; (6) increased competition from Europe and other countries; (7) a shift in emphasis toward health maintenance organizations in clinical research; and (8) adequate funding and other resources. Metrics focused on these eight factors could help sustain the world-class quality of basic research as an essential pillar of the research system.
10. National Institutes of Health. (2013). Draft Report on Approaches to Assess the Value of Biomedical Research Supported by NIH. Working Group on Approaches to Assess the Value of Biomedical Research Supported by NIH, Scientific Management Review Board. Bethesda, MD: Author.
A draft of the working group’s report notes that the tools, techniques, and data needed to develop comprehensive measures of value are still in the early phases of development, and therefore it is not possible to assess the value of NIH-funded biomedical research at this time. However, the draft report notes six strategies that could enhance assessment efforts: (1) a sustained investment in NIH’s data infrastructure, and dedicated funds and a mechanism to support assessment projects; (2) a focus on clear connections between the generation and impact of scientific knowledge; (3) a movement toward “credible, interpretable, and useful assessments of the value of NIH” that “attribute outcomes to all contributors and adopt a timeframe that is broad enough to include sufficient time for discovery to be applied”; (4) partnerships with stakeholders to complete the assessments; (5) establishment of a trans-NIH Committee on Assessments that would develop a strategy and process for assessing the value of NIH-sponsored research; and (6) beginning assessment activities with a clear statement of purpose for the exercise and a strong strategy for communicating and disseminating the assessment results.
11. National Research Council. (2011). Measuring the Impacts of Federal Investments in Research: A Workshop Summary. S. Olson and S. Merrill, Rapporteurs, Committee on Measuring Economic and Other Returns on Federal Research Investments. Washington, DC: The National Academies Press. Available: http://www.nap.edu/catalog.php?record_id=13208 [August 2014].
The workshop participants noted the myriad challenges to developing a universal measure of research impact that spans all scientific fields (e.g., the returns of research occur on an unpredictable timeline and depend on further efforts by individuals, society, or other organizations; the definition of “positive impact” can be variable; intangible outcomes such as knowledge, national reputation, and failure are crucial to success but difficult to measure). With that caveat, the participants identified six target areas in which the short- and long-term economic and noneconomic impacts of federal research funding can be assessed: (1) economic growth, (2) productivity, (3) employment, (4) social values (e.g., environmental protection and food security), (5) public goods (e.g., national security), and (6) the behavior of decision makers and the public.
12. National Research Council. (2012a). Continuing Innovation in Information Technology. Computer Science and Telecommunications Board, Division on Engineering and Physical Sciences. Washington, DC: The National Academies Press. Available: http://www.nap.edu/catalog.php?record_id=13427 [August 2014].
This report, referred to as the “tire-tracks” report for its famous diagram, shows how investments in academic and industry research are linked to the creation of new information technology (IT) industries with more than $1 billion in annual revenue. It describes how industry builds on government-funded university research and illustrates the interdependencies among subfields of computing and communications research. The report concludes that properly managed, publicly funded research in IT will continue to create important new technologies and industries, with an unpredictable timeline from the discovery of a new idea to the creation of a highly profitable industry. The complex partnerships among government, industry, and universities—and the federal government’s support of basic research—are critical to the success of IT, and consequently to national security and economic and societal well-being.
13. National Research Council. (2012c). Research Universities and the Future of America: Ten Breakthrough Actions Vital to Our Nation’s Prosperity and Security. Committee on Research Universities, Board on Higher Education and Workforce. Washington, DC:
The National Academies Press. Available: http://www.nap.edu/catalog.php?record_id=13396 [August 2014].
This report recommends the 10 most important actions that Congress, state governments, research universities, and others can take to maintain U.S. excellence in research that will help achieve national goals. For each recommendation, the report outlines an implementation strategy, budget considerations, and expected outcomes. Specifically, the report recommends that the federal government (1) adopt stable and effective policies, practices, and funding for university research and development (R&D) and graduate education; (2) provide greater autonomy for public research universities so they can leverage local and regional strengths to compete strategically and respond with agility to new opportunities; (3) strengthen the business role in the research partnership, facilitating the transfer of knowledge, ideas, and technology to society and accelerating “time to innovation” to achieve national goals; (4) increase university cost-effectiveness and productivity to provide a greater return on investment for taxpayers, philanthropists, corporations, foundations, and other research sponsors; (5) create a “Strategic Investment Program” that funds initiatives at research universities critical to advancing education and research in areas of key national priority; (6) the federal government and other research sponsors should strive to cover the full costs of research projects and other activities they procure from research universities in a consistent and transparent manner; (7) reduce or eliminate regulations that increase administrative costs, impede research productivity, and deflect creative energy without substantially improving the research environment; (8) improve the capacity of graduate programs to attract talented students by addressing issues such as attrition rates, time to degree, funding, and alignment with both student career opportunities and national interests; (9) secure for the United States the full benefits of education for all Americans, including women and underrepresented minorities, in science, mathematics, engineering, and technology (STEM); and (10) ensure that the United States will continue to benefit strongly from the participation of international students and scholars in the nation’s research enterprise.
14. National Research Council. (2012d). Rising to the Challenge: U.S. Innovation Policy for Global Economy. C.W. Wessner and A.W. Wolff, (Eds.), Committee on Comparative National Innovation Policies:
Best Practice for the 21st Century, Board on Science, Technology, and Economic Policy. Washington, DC: The National Academies Press. Available: http://www.nap.edu/catalog.php?record_id=13386 [August 2014].
This report emphasizes the importance of sustaining global leadership in the commercialization of innovation, which is vital to America’s security, its role as a world power, and the welfare of its people. Both advanced and emerging nations are pursuing policies and programs that appear to be less constrained than those in the United States. This report argues that more attention should be paid to achieving and benefiting from the outputs of innovation—the commercial products, the industries, and particularly high-quality jobs to restore full employment. America’s economic and national security future depends on success in this endeavor.
15. National Research Council. (2013a). Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. R.E. Litan, A.W. Wyckoff, and K.H. Fealing (Eds.), Panel on Developing Science, Technology, and Innovation Indicators for the Future, Committee on National Statistics, Division of Behavioral and Social Sciences and Education, Board on Science, Technology, and Economic Policy, Division of Policy and Global Affairs. Washington, DC: The National Academies Press.
This report examines science and technology indicators from a number of nations in North America, Europe, Asia, and Australia and offers recommendations on improving the U.S. National Science Foundation’s science and technology indicators to better enable the agency to respond to changing policy concerns. In particular, the report examines the use of specific metrics for measuring networks, as well as human and knowledge capital, and notes that indicators of human and knowledge capital could be created by linking existing longitudinal data from agencies and organizations such as the U.S. Census Bureau, the National Center for Science and Engineering Statistics (NCSES), and the Bureau of Labor Statistics. In addition, indicators could be generated to track the flow of knowledge in specific fields of science, which could potentially help answer questions about STEM labor mobility and provide the information needed to better match STEM training to the demand for particular skills. The report offers a number of recommendations for NCSES, including mak-
ing data quality a top priority, and working with other government agencies and departments to make existing data available and linkable between agencies.
16. OECD. (1996). The Knowledge-Based Economy. Paris: Author. Available: http://www.oecd.org/dataoecd/51/8/1913021.pdf [August 2014].
This OECD report describes the knowledge-based economy and explains why current understanding of this economy is constrained by the extent and quality of the available knowledge-related indicators. The report emphasizes the need to produce and disseminate the specific genres of knowledge that are needed at the time. It distinguishes the “know-what” (knowledge of facts and information) from the “know-why” (knowledge of the laws and principles of nature), the “know-how” (capabilities and practical skills), and the “know-who” (knowing who has each type of knowledge). A well-functioning system cannot simply rely on the knowledge of information and underlying principles gained in school and through basic research (i.e., the know-what and know-why). The system also depends on workers with practical skills (i.e., the know-how) and the invaluable awareness of other workers and their expertise, which is gained through networks, partnerships, and other professional relationships. The report notes four areas for indicator development (knowledge stocks and flows, knowledge rates of return, knowledge networks, and knowledge and learning) and makes recommendations on the development of indicators of the knowledge-based economy, noting that such indicators must start with improvements to more traditional input indicators of R&D expenditures and research personnel. Better indicators also are needed of knowledge stocks and flows, particularly relating to the diffusion of information technologies, in both the manufacturing and service sectors; social and private rates of return on knowledge investments to better gauge the impact of technology on productivity and growth; the functioning of knowledge networks and national innovation systems; and the development of human capital.
17. OECD. (1997). National Innovation Systems. Paris: Author. Available: http://www.oecd.org/dataoecd/35/56/2101733.pdf [August 2014].
This report explores the web of interactions among institutions, researchers, and private firms that make up national innovation systems and identifies best practices. It notes that an effective innovation system produces revolutionary advances both within the research system and beyond, relying on networks of institutions and researchers to integrate, transform, and disseminate discoveries in diverse fields. The report reflects the first phase of a two-phase OECD project to map knowledge flows and develop indicators for assessing national innovation systems.
18. OECD. (2000). The Impact of Public R&D Expenditure on Business R&D. D. Guellec and B. Van Pottelsberghe, Directorate for Science, Technology and Industry (Eds.). Paris: Author. Available: http://dx.doi.org/10.1787/670385851815 [August 2014].
This report concludes that among the major instruments of government policy, both fiscal incentives and direct funding stimulate business-funded R&D, whereas government- and university-performed research appear to have a crowding-out effect. In short, when the purpose is to increase business-funded R&D, it is apparently better to give money than knowledge to business. However, it must be kept in mind that publicly produced knowledge may result in technology that is used by business while not inducing it to increase its research expenditure. Moreover, it is not the major purpose of government laboratories to produce knowledge for the business sector. For university research, barriers to the transfer of knowledge to business can be mitigated by government (targeted) funding of business R&D. And whereas the crowding-out effect is immediate (contemporaneous with the research spending), spillovers may take time to reach industry, beyond the horizon of the assessment.
19. OECD. (2002). The Global Competition for Talent: Mobility of the Highly Skilled. Paris: Author. Available: http://www.oecd.org/sti/stpolicy/talent [August 2014].
This report analyzes international flows of human resources in science and technology, relying on the most recent data on policies and research performance assessments from OECD member and observer countries. The findings reported suggest that global innovation has increased as the international mobility of highly skilled workers has become more complex and frequent, and as more economies have come to participate in R&D and innovation
activity. Consequently, competition for talent is now influencing innovation policy initiatives across the globe. The report recommends addressing shortcomings in national policies that may limit the domestic supply of skilled workers, and ensuring that the wider environment for innovation and scientific endeavor is sound.
20. President’s Council of Advisors on Science and Technology. (2012). Transformation and Opportunity: The Future of the U.S. Research Enterprise. Washington, DC: Executive Office of the President. Available: http://www.whitehouse.gov/sites/default/files/microsites/ostp/pcast_future_research_enterprise_20121130.pdf [August 2014].
This report documents the importance of research and recommends a number of steps to strengthen the U.S. research enterprise. It explains why, according to the classic public good argument, the federal government must fund basic research.
21. U.S. Congress. (1986). Research Funding as an Investment: Can We Measure the Returns? A Technical Memorandum. Office of Technology Assessment. Washington, DC: U.S. Government Printing Office. Available: http://www.princeton.edu/~ota/disk2/1986/8622_n.html.
This report, requested by the Task Force on Science Policy of the House Committee on Science and Technology, explores whether the use of quantitative mechanisms associated with the concept of “investment” might allow for the meaningful prediction and measurement of research returns. The report concludes that while some quantitative techniques might prove useful to Congress in evaluating specific areas of research, basic science is not amenable to the type of economic analysis that might prove useful for applied research or product development. Even in the business community, the report concludes, decisions about research are much more the result of open communication followed by judgment than of quantification. The American research system endures and succeeds because it is complex and pluralistic, depending on various players (e.g., scientists, citizens, administrators, Congress) to reach final decisions on funding. Expert analysis, openness, experience, and expert judgment are better tools than economic quantitative methods, according to the report.
22. U.S. Department of Commerce. (2012). The Competitiveness and Innovative Capacity of the United States. Washington, DC: Author. Available: http://www.commerce.gov/sites/default/files/documents/2012/january/competes_010511_0.pdf [August 2014].
This report, prompted by the America COMPETES Act, finds that the competitiveness of the United States can be improved by focusing on three pillars that historically helped unleash the innovative potential of the private sector: federal support for basic research; education; and competitive, cutting-edge technological infrastructure (e.g., helping rural areas gain broadband Internet access). All three pillars are areas in which the federal government has made, and should continue to make, significant investments.
