Appendix B
Agendas of Open Committee Meetings
First Meeting
DECEMBER 12-13, 2017
TUESDAY, DECEMBER 12, 2017
Welcome and Introductions
Mary Ellen O’Connell, Executive Director, Division of Behavioral and Social Science and Education
Harvey Fineberg, Committee Chair; President, Gordon and Betty Moore Foundation
The Scientific Enterprise
Edward (Ned) Hall, Committee Member; Chair, Department of Philosophy, Harvard University
National Science Foundation’s Interests and Goals for the Study
Joan Ferrini-Mundy, Chief Operating Officer, National Science Foundation
Perspectives on Reproducibility and Replication: Scientific Societies, Part I
Panelists, primarily leaders from U.S. scientific societies and organizations, have been asked to focus on the following topics:
- Within your field of science, what is the level of awareness, interest, concern, and involvement in reproducibility and replicability of research results?
- Are there specific areas within your field of science that are more likely to have issues with reproducing scientific results?
- What reproducibility challenges does your field of science face with cross disciplinary research?
Behavioral and Social Sciences
William G. Jacoby, Department of Political Science, Michigan State University; Editor, American Journal of Political Science
Howard S. Kurtzman, Acting Executive Director, Science Directorate, American Psychological Association
Felice J. Levine, Executive Director and Ethics Officer, American Educational Research Association
Physical Sciences
Kate Kirby, Chief Executive Officer, American Physical Society
David Sholl, John F. Brock III School Chair, School of Chemical and Bio-molecular Engineering, Georgia Institute of Technology
Statistics
Ron Wasserstein, Executive Director, American Statistical Association
Earth Sciences
Brooks Hanson, Senior Vice President Publications, American Geophysical Union
Engineering
Philip DiVietro, Managing Director of Publishing, American Society of Mechanical Engineers
John Baillieul, Distinguished Professor, Department of Mechanical Engineering, Boston University
Public Comments
WEDNESDAY, DECEMBER 13, 2017
Welcome, Day One and Day Two Overviews
Harvey Fineberg, Committee Chair
Perspectives on Reproducibility and Replication: Scientific Societies and Agencies, Part II
- Panelists, primarily leaders from U.S. scientific societies and organizations, have been asked to focus on the following topics:
- What reproducibility challenges does your field of science face with cross-disciplinary research?
- Within your field of science, what is the level of awareness, interest, concern, and involvement in reproducibility and replicability of research results?
- Are there specific areas within your field of science that are more likely to have issues with reproducing scientific results?
Life Sciences
Yvette Seger, Director of Science Policy, Federation of American Societies for Experimental Biology
Reproducibility of Scientific Research within the Agencies
Patricia Valdez, Extramural Research Integrity Officer, National Institutes of Health
Anne Plant, Biosystems and Biomaterials Division, National Institute of Standards and Technology
- International panelists have been asked to focus on the following topics:
- What is the level of awareness, interest, concern, and involvement in reproducibility and replicability of research results within your national scientific societies?
- Are there specific areas of science that are more likely to have issues with reproducing scientific results?
- What reproducibility and replicability issues exist for cross-disciplinary research?
Eric-Jan Wagenmakers, Professor of Psychology, University of Amsterdam
Jean Phillipe de Jong, The Dutch Royal Society of Sciences
- The editor of a major cross-disciplinary journal was asked to focus on the following questions:
- Can journals assess levels of R&R across science?
- What R&R challenges does cross-disciplinary research pose that can be addressed by journals?
- Are cross-disciplinary papers handled differently from “pure” science papers in terms of peer review or publishing decisions?
- What R&R challenges does cross-disciplinary research pose that can be addressed by journals?
Veronique Kiermer, Executive Editor, Public Library of Science
Reporting of Reproducibility Issues in Science
Richard Harris, Science Correspondent, National Public Radio
Public Comments
Second Meeting
FEBRUARY 22-23, 2018
THURSDAY, FEBRUARY 22, 2018
Welcome and Introductions
Harvey Fineberg , Committee Chair; President, Gordon and Betty Moore Foundation
Perspectives on Reproducibility and Replication: American Economic Association
The speaker has been asked to focus on the following questions:
- Within economics, what is the level of awareness, interest, concern, and involvement in reproducibility and replicability of research results?
- Are there specific areas within economics that are more likely to have issues with reproducing scientific results?
- What reproducibility challenges does economics face with cross-disciplinary research?
Margaret Levenstein, Professor of Economics and Director, Inter-university Consortium for Political and Social Research, University of Michigan
Panel 1: Overview of Extent of Reproducibility Issues in Scientific and Engineering Research
The panelists have been asked to focus on the following session questions:
- How extensive is the lack of reproducibility in research results in science and engineering, in general?
- At what level does a lack of reproducibility become a problem for the wellbeing of science or engineering?
- Does the lack to reproduce scientific results impact the public perception of specific scientific fields and/or science and engineering in general?
John Ioannidis, C.F. Rehnborg Chair in Disease Prevention and Co-Director, Meta-Research Innovation Center, Stanford University
Brian Nosek, Director, Center for Open Science and Professor of Psychology, University of Virginia
Daniel Sarewitz, Co-Director, Consortium for Science, Policy & Outcomes, and Professor of Science and Society, Arizona State University
Panel 2, Part 1: Reproducibility Issues in Computational Sciences and Statistics
The panelists in this session have been asked to address the session questions (above) with a focus on the management of computational code and data.
David Madigan, Executive Vice President and Dean of the Faculty of Arts and Sciences, Columbia University
Arjun Kumar Manrai, Department of Biomedical Informatics, Harvard University
Panel 2, Part 2: Reproducibility Issues in Computational Sciences and Statistics
The panelists in this session have been asked to address the session questions (above) with a focus on the impact of the misuse of statistics in research.
Giovanni Parmigiani, Harvard T.H. Chan School of Public Health and Dana Farber Cancer Institute
Steven Goodman, Professor of Medicine and Epidemiology, Associate Dean for Clinical and Translational Research, and Co-Director, Meta-Research Innovation Center at Stanford (METRICS), Stanford University
Panel 2, Part 3: Reproducibility Issues in Economics and Social Science
The panelists in this session have been asked to address the session questions (above) as they relate to economics, social sciences, and psychology.
Paul L. Joskow, Elizabeth and James Killian Professor of Economics, Emeritus Department of Economics, Massachusetts Institute of Technology
Arthur (Skip) Lupia, Hal R. Varian Collegiate Professor of Political Science, University of Michigan
Joseph Simmons, Professor of Operations, Information, and Decisions, Wharton School, University of Pennsylvania
Public Comments
FRIDAY, FEBRUARY 23, 2018
Welcome and Introductions
Harvey Fineberg, Committee Chair
Panel 3: Reproducibility Issues in Engineering
Gianluca Setti, Politecnico di Torino, Italy, and IEEE editor
Panel 4: Reporting of Reproducibility Issues in Science
Christie Aschwanden, Lead Science Editor, FiveThirtyEight
Laura Helmuth, Science Editor, The Washington Post
Public Comments
Third Meeting
WEDNESDAY, APRIL 18, 2018
Welcome and Introductions
Harvey Fineberg, Committee Chair; President, Gordon and Betty Moore Foundation
Perspectives on Scientific Progress and Irreproducibility
Richard Shiffrin, Department of Psychological and Brain Sciences, Indiana University Bloomington
Panel 1: Reproducibility in the Physical and Earth Sciences
Joan Brennecke, Cockrell Family Chair in Engineering #16, McKetta Department of Chemical Engineering, The University of Texas at Austin
Peter Mohr, Atomic Physics Division, National Institute of Standards and Technology
Panel 2: Reproducibility in Industry and Industrial Engineering
Carl Ascoli, Chief Science Officer, Rockland Immunochemicals
William Lyons, Director, Global Research and Development Strategy for the Global Technology Organization, Boeing Research and Technology
Introduction to Economics and Reproducibility
Daniel L. Goroff, Vice President and Program Director, Alfred P. Sloan Foundation
Panel 3: The Economics of Addressing Reproducibility Issues in Science
Heidi Williams, Department of Economics, Massachusetts Institute of Technology
Myron P. Gutmann, Professor of History and Director of the Institute of Behavioral Science, University of Colorado Boulder
Richard Freeman, Herbert Ascherman Chair in Economics, Harvard University
Brent Goldfarb, Management and Organizations Department and Dingman Center for Entrepreneurship, University of Maryland
Public Comments
Fourth Meeting
MAY 9, 2018
Welcome and Introductions
Harvey Fineberg, Committee Chair; President, Gordon and Betty Moore Foundation
Panel: Perspectives on Reproducibility and Replication of Results in Climate Science
The panelists have been asked to focus on the following session questions:
- How has the awareness and understanding about reproducibility and replication in climate science evolved over recent years?
- Are there specific challenges regarding reproducibility that you have encountered or are aware of? Identify specific steps that are being taken, either by you or by others, to ameliorate these issues.
- Highlight historical and potential new approaches to reproducing and replicating climate science research using examples such as paleoclimate data to test models and estimate uncertainties.
Michael Evans, Department of Geology, University of Maryland
Gavin Schmidt, Director, Goddard Institute for Space Studies, National Aeronautics and Space Administration
Rich Loft, Director, Technology Development Division, National Center for Atmospheric Research
Andrea Dutton, Department of Geological Sciences, University of Florida
Wrap-Up
Harvey Fineberg, Committee Chair
Fifth Meeting
MAY 31, 2018
Welcome and Call to Order
Harvey Fineberg, Committee Chair; President, Gordon and Betty Moore Foundation
Panel: International Perspectives on Reproducibility and Replication in Science and Engineering
The panelists have been asked to focus on the following session questions:
- Are there specific examples in your country/region where a lack of reproducibility and replicability in research results has led to doubt about reported results more broadly? Are reproducibility and replication of research results a global concern or is it a concern focused within specific countries?
- Are there particular scientific fields in which lack of reproducibility and replicability is more/less of a concern?
- Are there any concrete actions that organizations (e.g., funders, publishers, societies) in your country or region have taken to address concerns about reproducibility and replicability? What actions should they take?
- Should the research community work regionally and/or globally to address concerns about reproducibility and replicability? If so, what should be the priorities?
Laura Fierce, Environmental and Climate Sciences Department, Brookhaven National Laboratory, and member, Executive Committee, Global Young Academy [in person]
Koen Vermeir, French National Centre for Scientific Research, former Co-Chair, Scientific Excellence and Open Science Programs, and Member, Executive Committee, Global Young Academy [via Zoom]
Harry Xia, President, Alliance for Scientific Editing in China [in person]
Suman Chakraborty, Department of Mechanical Engineering, Indian Institute of Technology Kharagpur, India [via Zoom]