National Academies Press: OpenBook
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2016. Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/21915.
×

Statistical Challenges in Assessing
and Fostering the Reproducibility
of Scientific Results

Summary of a Workshop

Michelle Schwalbe, Rapporteur


Committee on Applied and Theoretical Statistics

Board on Mathematical Sciences and Their Applications

Division on Engineering and Physical Sciences

images

THE NATIONAL ACADEMIES PRESS

Washington, DC

www.nap.edu

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2016. Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/21915.
×

THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001

This workshop was supported by Grant No. DMS-1351163 between the National Academies of Sciences and the National Science Foundation. Any opinions, findings, or conclusions expressed in this publication do not necessarily reflect the views of any organization or agency that provided support for the project.

International Standard Book Number-13: 978-0-309-39202-0
International Standard Book Number-10: 0-309-39202-0
Digital Object Identifier: 10.17226/21915

Cover: The photo “Peas in a Pod” by Andrew_Writer is used with Creative Commons Attribution 2.0 Generic license (https://www.flickr.com/photos/dragontomato/3698617520/in/gallery-106928713@N04-72157638971870524/).

This report is available in limited quantities from:

Board on Mathematical Sciences and Their Applications
National Academies of Sciences, Engineering, and Medicine
500 Fifth Street NW
Washington, DC 20001
bmsa@nas.edu
http://www.nas.edu/bmsa

Additional copies of this workshop summary are available for sale from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu/.

Copyright 2016 by the National Academy of Sciences. All rights reserved.

Printed in the United States of America

Suggested citation: National Academies of Sciences, Engineering, and Medicine. 2016. Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/21915.

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2016. Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/21915.
×

images

The National Academy of Sciences was established in 1863 by an Act of Congress, signed by President Lincoln, as a private, nongovernmental institution to advise the nation on issues related to science and technology. Members are elected by their peers for outstanding contributions to research. Dr. Ralph J. Cicerone is president.

The National Academy of Engineering was established in 1964 under the charter of the National Academy of Sciences to bring the practices of engineering to advising the nation. Members are elected by their peers for extraordinary contributions to engineering. Dr. C. D. Mote, Jr., is president.

The National Academy of Medicine (formerly the Institute of Medicine) was established in 1970 under the charter of the National Academy of Sciences to advise the nation on medical and health issues. Members are elected by their peers for distinguished contributions to medicine and health. Dr. Victor J. Dzau is president.

The three Academies work together as the National Academies of Sciences, Engineering, and Medicine to provide independent, objective analysis and advice to the nation and conduct other activities to solve complex problems and inform public policy decisions. The Academies also encourage education and research, recognize outstanding contributions to knowledge, and increase public understanding in matters of science, engineering, and medicine.

Learn more about the National Academies of Sciences, Engineering, and Medicine at www.national-academies.org.

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2016. Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/21915.
×

This page intentionally left blank.

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2016. Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/21915.
×

PLANNING COMMITTEE ON STATISTICAL CHALLENGES IN ASSESSING AND FOSTERING THE REPRODUCIBILITY OF SCIENTIFIC RESULTS

CONSTANTINE GATSONIS, Brown University, Co-Chair

GIOVANNI PARMIGIANI, Dana-Farber Cancer Institute, Co-Chair

STEPHEN FIENBERG, Carnegie Mellon University

STEVEN N. GOODMAN, Stanford University School of Medicine

JOHN H. HOLMES, University of Pennsylvania Perelman School of Medicine

ALAN F. KARR, RTI International

JELENA KOVAČEVIĆ, Carnegie Mellon University

XIHONG LIN, Harvard University

ROGER PENG, Johns Hopkins Bloomberg School of Public Health

VICTORIA STODDEN, University of Illinois, Urbana-Champaign

Staff

MICHELLE K. SCHWALBE, Senior Program Officer

RODNEY N. HOWARD, Administrative Assistant

LINDA CASOLA, Senior Program Assistant and Staff Editor

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2016. Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/21915.
×

COMMITTEE ON APPLIED AND THEORETICAL STATISTICS

CONSTANTINE GATSONIS, Brown University, Chair

DEEPAK AGARWAL, LinkedIn

KATHERINE BENNETT ENSOR, Rice University

MONTSERRAT (MONTSE) FUENTES, North Carolina State University

ALFRED O. HERO III, University of Michigan

DAVID M. HIGDON, Virginia Bioinformatics Institute

ROBERT E. KASS, Carnegie Mellon University

JOHN LAFFERTY, University of Chicago

XIHONG LIN, Harvard University

JOSÉ M. F. MOURA, Carnegie Mellon University

SHARON-LISE T. NORMAND, Harvard University

GIOVANNI PARMIGIANI, Dana-Farber Cancer Institute

ADRIAN RAFTERY, University of Washington

LANCE WALLER, Emory University

EUGENE WONG, University of California, Berkeley

Staff

MICHELLE K. SCHWALBE, Director

RODNEY N. HOWARD, Administrative Assistant

LINDA CASOLA, Senior Program Assistant

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2016. Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/21915.
×

BOARD ON MATHEMATICAL SCIENCES AND THEIR APPLICATIONS

DONALD SAARI, University of California, Irvine, Chair

DOUGLAS N. ARNOLD, University of Minnesota

JOHN B. BELL, E. O. Lawrence Berkeley National Laboratory

VICKI M. BIER, University of Wisconsin, Madison

JOHN R. BIRGE, University of Chicago

L. ANTHONY COX, JR., Cox Associates, Inc.

MARK L. GREEN, University of California, Los Angeles

BRYNA KRA, Northwestern University

JOSEPH A. LANGSAM, Morgan Stanley (retired)

ANDREW W. LO, Massachusetts Institute of Technology

DAVID MAIER, Portland State University

WILLIAM A. MASSEY, Princeton University

JUAN C. MEZA, University of California, Merced

CLAUDIA NEUHAUSER, University of Minnesota

FRED S. ROBERTS, Rutgers University

GUILLERMO R. SAPIRO, Duke University

CARL P. SIMON, University of Michigan

KATEPALLI SREENIVASAN, New York University

ELIZABETH A. THOMPSON, University of Washington

Staff

SCOTT T. WEIDMAN, Director

NEAL GLASSMAN, Senior Program Officer

MICHELLE K. SCHWALBE, Senior Program Officer

RODNEY N. HOWARD, Administrative Assistant

BETH DOLAN, Financial Associate

Page viii Cite
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2016. Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/21915.
×

This page intentionally left blank.

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2016. Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/21915.
×

Acknowledgment of Reviewers

This workshop summary has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published workshop summary as sound as possible and to ensure that the workshop summary meets institutional standards for objectivity, evidence, and responsiveness to the project’s charge. The review comments and draft manuscript remain confidential to protect the integrity of the study process. We wish to thank the following individuals for their review of this workshop summary:

C. Glenn Begley, TetraLogic Pharmaceuticals,

Joel Greenhouse, Carnegie Mellon University,

Joelle Lomax, Science Exchange, and

Randy Schekman, University of California, Berkeley.

Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the views presented at the workshop, nor did they see the final draft of the workshop summary before its release. The review of this workshop summary was overseen by Lawrence D. Brown, University of Pennsylvania, who was responsible for making certain that an independent examination of this workshop summary was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this summary rests entirely with the author and the institution.

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2016. Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/21915.
×

This page intentionally left blank.

Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2016. Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/21915.
×
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2016. Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/21915.
×
Page R1
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2016. Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/21915.
×
Page R2
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2016. Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/21915.
×
Page R3
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2016. Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/21915.
×
Page R4
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2016. Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/21915.
×
Page R5
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2016. Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/21915.
×
Page R6
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2016. Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/21915.
×
Page R7
Page viii Cite
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2016. Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/21915.
×
Page R8
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2016. Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/21915.
×
Page R9
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2016. Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/21915.
×
Page R10
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2016. Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/21915.
×
Page R11
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2016. Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/21915.
×
Page R12
Next: 1 Introduction »
Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop Get This Book
×
Buy Paperback | $42.00 Buy Ebook | $33.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Questions about the reproducibility of scientific research have been raised in numerous settings and have gained visibility through several high-profile journal and popular press articles. Quantitative issues contributing to reproducibility challenges have been considered (including improper data measurement and analysis, inadequate statistical expertise, and incomplete data, among others), but there is no clear consensus on how best to approach or to minimize these problems.

A lack of reproducibility of scientific results has created some distrust in scientific findings among the general public, scientists, funding agencies, and industries. While studies fail for a variety of reasons, many factors contribute to the lack of perfect reproducibility, including insufficient training in experimental design, misaligned incentives for publication and the implications for university tenure, intentional manipulation, poor data management and analysis, and inadequate instances of statistical inference.

The workshop summarized in this report was designed not to address the social and experimental challenges but instead to focus on the latter issues of improper data management and analysis, inadequate statistical expertise, incomplete data, and difficulties applying sound statistic inference to the available data. Many efforts have emerged over recent years to draw attention to and improve reproducibility of scientific work. This report uniquely focuses on the statistical perspective of three issues: the extent of reproducibility, the causes of reproducibility failures, and the potential remedies for these failures.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!