Enhancing Scientific
Reproducibility in
Biomedical Research
Through
Transparent Reporting
PROCEEDINGS OF A WORKSHOP
Theresa Wizemann, Sylvia Ncha, Amanda Wagner Gee, and
Carolyn Shore, Rapporteurs
Forum on Drug Discovery, Development, and Translation
Forum on Neuroscience and Nervous System Disorders
National Cancer Policy Forum
Roundtable on Genomics and Precision Health
Board on Health Sciences Policy
Board on Health Care Services
Health and Medicine Division
THE NATIONAL ACADEMIES PRESS
Washington, DC
www.nap.edu
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001
This activity was supported by contracts between the National Academy of Sciences and AbbVie Inc.; Amgen Inc.*; Association of American Medical Colleges; AstraZeneca; Burroughs Wellcome Fund (Contract No. 1020264); Cell Press; Critical Path Institute; Eli Lilly and Company (Contract No. 4900709231)*; FasterCures, Milken Institute*; Foundation for the National Institutes of Health; Friends of Cancer Research; GlaxoSmithKline (Contract No. OTH-PPL-32245*); Johnson & Johnson; The Lancet; Merck & Co., Inc. (Contract No. MRLCPO-19-106723*); National Institutes of Health (Contract No. HHSN2632018000029I; Task Order Nos. HHSN26300007 and HHSN26300009): National Cancer Institute, National Center for Advancing Translational Sciences, National Institute of Allergy and Infectious Diseases, National Institute of Mental Health, National Institute of Neurological Disorders and Stroke, Office of Extramural Research, Office of Science Policy; Nature Research; New England Journal of Medicine; Sanofi (Contract No. 57505685*); Takeda Pharmaceuticals (Contract No. 53108*); and U.S. Food and Drug Administration (Grant Nos. 5R13FD005496-04 and 5R13FD005496-05)*. Any opinions, findings, conclusions, or recommendations expressed in this publication do not necessarily reflect the views of any organization or agency that provided support for the project.
International Standard Book Number-13: 978-0-309-66349-6
International Standard Book Number-10: 0-309-66349-0
Digital Object Identifier: https://doi.org/10.17226/25627
Additional copies of this publication are available from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.
Copyright 2020 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
Suggested citation: National Academies of Sciences, Engineering, and Medicine. 2020. Enhancing scientific reproducibility in biomedical research through transparent reporting: Proceedings of a workshop. Washington, DC: The National Academies Press. https://doi.org/10.17226/25627.
__________________
* This text was revised after final publication to correct this contract information.
The National Academy of Sciences was established in 1863 by an Act of Congress, signed by President Lincoln, as a private, nongovernmental institution to advise the nation on issues related to science and technology. Members are elected by their peers for outstanding contributions to research. Dr. Marcia McNutt is president.
The National Academy of Engineering was established in 1964 under the charter of the National Academy of Sciences to bring the practices of engineering to advising the nation. Members are elected by their peers for extraordinary contributions to engineering. Dr. John L. Anderson is president.
The National Academy of Medicine (formerly the Institute of Medicine) was established in 1970 under the charter of the National Academy of Sciences to advise the nation on medical and health issues. Members are elected by their peers for distinguished contributions to medicine and health. Dr. Victor J. Dzau is president.
The three Academies work together as the National Academies of Sciences, Engineering, and Medicine to provide independent, objective analysis and advice to the nation and conduct other activities to solve complex problems and inform public policy decisions. The National Academies also encourage education and research, recognize outstanding contributions to knowledge, and increase public understanding in matters of science, engineering, and medicine.
Learn more about the National Academies of Sciences, Engineering, and Medicine at www.nationalacademies.org.
Consensus Study Reports published by the National Academies of Sciences, Engineering, and Medicine document the evidence-based consensus on the study’s statement of task by an authoring committee of experts. Reports typically include findings, conclusions, and recommendations based on information gathered by the committee and the committee’s deliberations. Each report has been subjected to a rigorous and independent peer-review process and it represents the position of the National Academies on the statement of task.
Proceedings published by the National Academies of Sciences, Engineering, and Medicine chronicle the presentations and discussions at a workshop, symposium, or other event convened by the National Academies. The statements and opinions contained in proceedings are those of the participants and are not endorsed by other participants, the planning committee, or the National Academies.
For information about other products and activities of the National Academies, please visit www.nationalacademies.org/about/whatwedo.
PLANNING COMMITTEE ON ENHANCING SCIENTIFIC REPRODUCIBILITY IN BIOMEDICAL RESEARCH THROUGH TRANSPARENT REPORTING1
HARVEY FINEBERG (Chair), Gordon and Betty Moore Foundation
OTIS BRAWLEY, Johns Hopkins University
BARRY COLLER, The Rockefeller University
STUART HOFFMAN, Department of Veterans Affairs
VERONIQUE KIERMER, PLOS
BENEDICT KOLBER, Duquesne University
ALEXA MCCRAY, Harvard Medical School
JILL MESIROV, University of California, San Diego School of Medicine
MARTIN MURPHY, CEO Roundtable on Cancer
RICHARD NAKAMURA, Center for Scientific Review, National Institutes of Health (Retired)
FRANKLIN SAYRE, Thompson Rivers University
IDA SIM, University of California, San Francisco
VALDA VINSON, Science
Health and Medicine Division Staff
CAROLYN SHORE, Director, Forum on Drug Discovery, Development, and Translation
AMANDA WAGNER GEE, Program Officer
SYLVIA NCHA, Associate Program Officer
MELVIN JOPPY, Senior Program Assistant
ANDREW M. POPE, Senior Director, Board on Health Sciences Policy
Consultant
THERESA WIZEMANN, Science Writer
___________________
1 The National Academies of Sciences, Engineering, and Medicine’s planning committees are solely responsible for organizing the workshop, identifying topics, and choosing speakers. The responsibility for the published Proceedings of a Workshop rests with the workshop rapporteurs and the institution.
FORUM ON DRUG DISCOVERY, DEVELOPMENT, AND TRANSLATION1
ROBERT M. CALIFF (Co-Chair), Duke University and Verily Life Sciences
GREGORY SIMON (Co-Chair), Kaiser Permanente Washington Health Research Institute and University of Washington
CHRISTOPHER P. AUSTIN, National Center for Advancing Translational Sciences
LINDA BRADY, National Institute of Mental Health
TANISHA CARINO, FasterCures, Milken Institute
BARRY COLLER, The Rockefeller University
THOMAS CURRAN, Children’s Mercy, Kansas City
RICHARD DAVEY, National Institute of Allergy and Infectious Diseases
JAMES H. DOROSHOW, National Cancer Institute
JEFFREY M. DRAZEN, New England Journal of Medicine
STEVEN K. GALSON, Amgen Inc.
CARLOS GARNER, Eli Lilly and Company
JULIE L. GERBERDING, Merck & Co., Inc.
LYNN HUDSON, Critical Path Institute
DEBORAH HUNG, Harvard Medical School
ROSS MCKINNEY, JR., Association of American Medical Colleges
JOSEPH P. MENETSKI, Foundation for the National Institutes of Health
BERNARD H. MUNOS, InnoThink Center for Research in Biomedical Innovation
KELLY ROSE, Burroughs Wellcome Fund
ROB SCOTT, AbbVie Inc.
ANANTHA SHEKHAR, Indiana University School of Medicine
ELLEN V. SIGAL, Friends of Cancer Research
LANA R. SKIRBOLL, Sanofi
AMIR TAMIZ, National Institute of Neurological Disorders and Stroke
PAMELA TENAERTS, Clinical Trials Transformation Initiative, Duke University
JOANNE WALDSTREICHER, Johnson & Johnson
CARRIE WOLINETZ, Office of Science Policy, National Institutes of Health
ALASTAIR WOOD, Vanderbilt University
JANET WOODCOCK, U.S. Food and Drug Administration
___________________
1 The National Academies of Sciences, Engineering, and Medicine’s forums and roundtables do not issue, review, or approve individual documents. The responsibility for the published Proceedings of a Workshop rests with the workshop rapporteurs and the institution.
Forum Staff
CAROLYN SHORE, Forum Director
AMANDA WAGNER GEE, Program Officer
JENNIFER HINNERS, Program Officer
SYLVIA NCHA, Associate Program Officer
EESHAN KHANDEKAR, Associate Program Officer
MELVIN JOPPY, Senior Program Assistant
ANDREW M. POPE, Senior Director, Board on Health Sciences Policy
FORUM ON NEUROSCIENCE AND NERVOUS SYSTEM DISORDERS
FRANCES JENSEN (Co-Chair), University of Pennsylvania
JOHN KRYSTAL (Co-Chair), Yale University
SUSAN AMARA, Society for Neuroscience
RITA BALICE-GORDON, Sanofi
KATJA BROSE, Chan Zuckerberg Initiative
EMERY BROWN, Harvard Medical School and Massachusetts Institute of Technology
DANIEL BURCH, Pharmaceutical Product Development, LLC
JOSEPH BUXBAUM, Icahn School of Medicine at Mount Sinai
SARAH CADDICK, Gatsby Charitable Foundation
ROSA CANET-AVILÉS, Foundation for the National Institutes of Health
MARIA CARRILLO, Alzheimer’s Association
EDWARD CHANG, University of California, San Francisco
TIMOTHY COETZEE, National Multiple Sclerosis Society
JONATHAN COHEN, Princeton University
ROBERT CONLEY, Eli Lilly and Company
JAMES DESHLER, National Science Foundation
BILLY DUNN, U.S. Food and Drug Administration
MICHAEL EGAN, Merck Research Laboratories
NITA FARAHANY, Duke University School of Law
JOSHUA GORDON, National Institute of Mental Health
RAQUEL GUR, University of Pennsylvania
MAGALI HAAS, Cohen Veterans Bioscience
RAMONA HICKS, One Mind
RICHARD HODES, National Institute on Aging
STUART HOFFMAN, Department of Veterans Affairs
JONATHAN HORSFORD, National Institute of Dental and Craniofacial Research
YASMIN HURD, Icahn School of Medicine at Mount Sinai
STEVEN HYMAN, Broad Institute of Massachusetts Institute of Technology and Harvard University
MICHAEL IRIZARRY, Eisai Inc.
GEORGE KOOB, National Institute on Alcohol Abuse and Alcoholism
WALTER KOROSHETZ, National Institute of Neurological Disorders and Stroke
STORY LANDIS, National Institute of Neurological Disorders and Stroke (Emeritus)
ALAN LESHNER, American Association for the Advancement of Science (Emeritus)
HUSSEINI MANJI, Janssen Research & Development, LLC
CAROLINE MONTOJO, The Kavli Foundation
STEVEN PAUL, Voyager Therapeutics
EMILIANGELO RATTI, Takeda Pharmaceuticals International
TODD SHERER, The Michael J. Fox Foundation for Parkinson’s Research
DAVID SHURTLEFF, National Center for Complementary and Integrative Health
SANTA TUMMINIA, National Eye Institute
NORA VOLKOW, National Institute on Drug Abuse
ANDREW WELCHMAN, Wellcome Trust
DOUG WILLIAMSON, Lundbeck
STEVIN ZORN, MindImmune Therapeutics, Inc.
Forum Staff
CLARE STROUD, Forum Director
SHEENA M. POSEY NORRIS, Program Officer
PHOENIX WILSON, Senior Program Assistant
ANDREW M. POPE, Senior Director, Board on Health Sciences Policy
NATIONAL CANCER POLICY FORUM
EDWARD J. BENZ, JR. (Chair), Dana-Farber Cancer Institute and Harvard Medical School
GARNET L. ANDERSON, Fred Hutchinson Cancer Research Center and University of Washington
KENNETH ANDERSON, American Cancer Society and Jerome Lipper Multiple Myeloma Center
KAREN BASEN-ENGQUIST, University of Texas MD Anderson Cancer Center
CHRIS BOSHOFF, Pfizer Inc.
CATHY J. BRADLEY, Colorado School of Public Health and University of Colorado Cancer Center
OTIS W. BRAWLEY, Johns Hopkins University
ROBERT W. CARLSON, National Comprehensive Cancer Network
GWEN DARIEN, National Patient Advocate Foundation
NANCY E. DAVIDSON, Seattle Cancer Care Alliance; Fred Hutchinson Cancer Research Center; and University of Washington
GEORGE D. DEMETRI, Harvard Medical School; Dana-Farber Cancer Institute; and Dana-Farber/Harvard Cancer Center
JAMES H. DOROSHOW, National Cancer Institute
NICOLE F. DOWLING, Centers for Disease Control and Prevention
SCOT W. EBBINGHAUS, Merck Research Laboratories
KOJO S. J. ELENITOBA-JOHNSON, University of Pennsylvania
AWNY FARAJALLAH, Bristol-Myers Squibb Company
STANTON L. GERSON, Case Western Reserve University and University Hospitals Seidman Cancer Center
LORI HOFFMAN HŌGG, Department of Veterans Affairs
LINDA HOUSE, Cancer Support Community
HEDVIG HRICAK, Memorial Sloan Kettering Cancer Center
ROY A. JENSEN, Association of American Cancer Institutes; The University of Kansas Cancer Center; and Kansas Masonic Cancer Research Institute
LISA KENNEDY SHELDON, Oncology Nursing Society
SAMIR N. KHLEIF, Georgetown University Medical Center
MICHELLE M. LE BEAU, University of Chicago Medicine Comprehensive Cancer Center
MIA LEVY, Rush University Cancer Center and Rush System for Health
J. LEONARD LICHTENFELD, American Cancer Society
NEAL J. MEROPOL, Flatiron Health
MARTIN J. MURPHY, CEO Roundtable on Cancer
RANDALL A. OYER, Penn Medicine Lancaster General Health
RICHARD PAZDUR, U.S. Food and Drug Administration
RICHARD L. SCHILSKY, American Society of Clinical Oncology
DEBORAH SCHRAG, Harvard Medical School and Dana-Farber Cancer Institute
LAWRENCE N. SHULMAN, University of Pennsylvania
DAN THEODORESCU, Cedars-Sinai Medical Center
GEORGE J. WEINER, The University of Iowa
ROBERT A. WINN, Virginia Commonwealth University
Forum Staff
ERIN BALOGH, Senior Program Officer
EMILY ZEVON, Associate Program Officer
KATE HAWTHORNE, Senior Program Assistant
ANNALEE GONZALES, Administrative Assistant
SHARYL NASS, Director, National Cancer Policy Forum, and Director, Board on Health Care Services
ROUNDTABLE ON GENOMICS AND PRECISION HEALTH
GEOFFREY GINSBURG (Co-Chair), Duke University
MICHELLE PENNY (Co-Chair), Biogen
NAOMI ARONSON, Blue Cross Blue Shield Association
ARIS BARIS, Regeneron Pharmaceuticals
KARINA BIENFAIT, Merck & Co., Inc.
VENCE BONHAM, National Human Genome Research Institute
ANN CASHION, National Institute of Nursing Research
ROBERT DARNELL, The Rockefeller University
KATHERINE DONIGAN, U.S. Food and Drug Administration
W. GREGORY FEERO, Journal of the American Medical Association
JESSICA GILL, National Institute of Nursing Research
MARC GRODMAN, Genosity
JILL M. HAGENKORD, Color Genomics
EMILY HARRIS, National Cancer Institute
RICHARD HODES, National Institute on Aging
PRADUMAN JAIN, Vibrent Health
SEKAR KATHIRESAN, Massachusetts General Hospital
MUIN KHOURY, Centers for Disease Control and Prevention
CHARLES LEE, The Jackson Laboratory for Genomic Medicine
THOMAS LEHNER, National Institute of Mental Health
PATRICK LOERCH, Johnson & Johnson
JAMES LU, Helix
SEAN MCCONNELL, American Medical Association
JENNIFER MOSER, Department of Veterans Affairs
ANNA PETTERSSON, Pfizer Inc.
VICTORIA PRATT, Association for Molecular Pathology
NADEEM SARWAR, Eisai Inc.
SHERI SCHULLY, Office of Disease Prevention, National Institutes of Health
JOAN SCOTT, Health Resources and Services Administration
SAM SHEKAR, Northrop Grumman Information Systems
NIKOLETTA SIDIROPOULOS, University of Vermont Health Network Medical Group
KATHERINE JOHANSEN TABER, Myriad Women’s Health
RYAN TAFT, Illumina
JACQUELYN TAYLOR, New York University
SHARON TERRY, Genetic Alliance
JOYCE TUNG, 23andMe
JAMESON VOSS, Air Force Medical Support Agency
MICHAEL WATSON, American College of Medical Genetics and Genomics
KAREN WECK, College of American Pathologists
CATHERINE WICKLUND, National Society of Genetic Counselors
HUNTINGTON F. WILLARD, Geisinger National Precision Health
JANET WILLIAMS, American Academy of Nursing
SARAH WORDSWORTH, University of Oxford
Roundtable Staff
SARAH BEACHY, Roundtable Director
SIOBHAN ADDIE, Program Officer
MEREDITH HACKMANN, Associate Program Officer
MICHAEL BERRIOS, Senior Program Assistant
ANDREW M. POPE, Senior Director, Board on Health Sciences Policy
This page intentionally left blank.
Reviewers
This Proceedings of a Workshop was reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise. The purpose of this independent review is to provide candid and critical comments that will assist the National Academies of Sciences, Engineering, and Medicine in making each published proceedings as sound as possible and to ensure that it meets the institutional standards for quality, objectivity, evidence, and responsiveness to the charge. The review comments and draft manuscript remain confidential to protect the integrity of the process.
We thank the following individuals for their review of this proceedings:
YARIMAR CARRASQUILLO, National Center for Complementary and Integrative Health, National Institutes of Health
ASHLEY FARLEY, Bill & Melinda Gates Foundation
RICHARD SEVER, Cold Spring Harbor Laboratory Press
GEETA K. SWAMY, Duke University
Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the content of the proceedings nor did they see the final draft before its release. The review of this proceedings was overseen by CAROLE LEE, University of Washington. She was responsible for making certain that an independent examination of this proceedings was carried out in accordance with standards of the National Academies and that all review comments were carefully considered. Responsibility for the final content rests entirely with the rapporteurs and the National Academies.
This page intentionally left blank.
Acknowledgments
Sponsorship from the Cell Press, The Lancet, the National Institutes of Health’s Office of Extramural Research and Office of Science Policy, and Nature Research made this workshop—and the fruitful discussions therein—possible. Support from the many annual sponsors of the National Academies of Sciences, Engineering, and Medicine’s Forum on Drug Discovery, Development, and Translation; Forum on Neuroscience and Nervous System Disorders; National Cancer Policy Forum; and Roundtable on Genomics and Precision Health is crucial to support this and other work of the National Academies.
The National Academies staff wish to express their gratitude to the speakers whose presentations helped inform workshop discussions on the state of transparent reporting in preclinical biomedical research; the members of the planning committee for their work in developing the workshop agenda and shaping the discussions; and the many other National Academies staff without whom this workshop and the accounting thereof would not have been possible: Jeanay Butler, Daniel Cesnalis, Robert Day, Sadaf Faraz, Greta Gorman, Anna Camilo Javier, Bardia Massoudkhan, Bettina Seliber, Lauren Shern, and Taryn Young.
This page intentionally left blank.
Contents
Organization of the Proceedings
Overview of Reproducibility and Replicability in Science
Sustaining Public Trust in Science
3 APPROACHES TO CULTIVATE TRANSPARENT REPORTING IN BIOMEDICAL RESEARCH
Early Career Investigator Perspective
Culture Change Organization Perspective
Scientific Society Journal Editor Perspective
National Institutes of Health Perspective
4 LESSONS LEARNED AND BEST PRACTICES
Lessons from the SPIRIT Initiative
The Institution’s Role in Improving Reproducibility
Funder/Foundation Role in Influencing and Enabling Reproducibility
Checklist Implementation by Life Science Journals: Toward Minimum Reporting Standards for Research
Approaches to Improve Adherence to Checklists and Guidelines
6 TOWARD MINIMAL REPORTING STANDARDS FOR PRECLINICAL BIOMEDICAL RESEARCH
What Transparent Reporting Means for Reviewers
Improving Assessment of Reproducibility
Culture Change for Journal Publishers
Engaging Research Support Staff
Librarians Driving Institutional Change
Applying a Systematic Framework to Developing Mimimal Reporting Standards
The Impact of Minimal Standards on Improving Methodology
7 STAKEHOLDER OPPORTUNITIES FOR PROMOTING TRANSPARENT REPORTING
Small Group Discussion Reports
B BACKGROUND DISCUSSION DOCUMENT: SELECTED GUIDELINES FOR TRANSPARENT REPORTING
Acronyms and Abbreviations
AlzPED | Alzheimer’s Disease Preclinical Efficacy Database |
ARRIVE | Animal Research: Reporting of In Vivo Experiments |
CEO | chief executive officer |
CONSORT | Consolidated Standards of Reporting Trials |
COPE | Committee on Publication Ethics |
CRO | clinical research organization |
CTSA | Clinical and Translational Science Awards |
ECNP | European College of Neuropsychopharmacology |
EQIPD | European Quality in Preclinical Data |
EQUATOR | Enhancing the QUAlity and Transparency Of health Research |
FSU | Florida State University |
ICMJE | International Committee of Medical Journal Editors |
IRB | Institutional Review Board |
JoVE | Journal of Visualized Experiments |
MD | Standards for Preventing and Handling Missing Data |
MDAR | materials, design, analysis, and reporting |
METRICS | Meta-Research Innovation Center at Stanford University |
NAS | National Academy of Sciences |
NC3Rs | National Centre for the Replacement, Refinement and Reduction of Animals in Research |
NCCIH | National Center for Complementary and Integrative Health (NIH) |
NIH | National Institutes of Health (Department of Health and Human Services) |
NINDS | National Institute of Neurological Disorders and Stroke (NIH) |
NPQIP | Nature Publishing Group Quality in Publication |
ORI | Office of Research Integrity |
PCORI | Patient-Centered Outcomes Research Institute |
PLOS | Public Library of Science |
PRISMA | Preferred Reporting Items for Systematic Reviews and Meta-Analyses |
PTSD | posttraumatic stress disorder |
RCR | responsible conduct of research |
RQR | rigor, quality, and reproducibility |
SEPTRE | SPIRIT Electronic Protocol Tool and Resource |
SPIRIT | Standard Protocol Items: Recommendations for Interventional Trials |
STEM | science, technology, engineering, and mathematics |
TOP | Transparency and Openness Promotion |