National Academies Press: OpenBook
« Previous: Appendix B: Background Discussion Document: Selected Guidelines for Transparent Reporting
Suggested Citation:"Appendix C: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2020. Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25627.
×

Appendix C

Workshop Agenda

Enhancing Scientific Reproducibility in Biomedical Research Through
Transparent Reporting—A Workshop

September 25–26, 2019

National Academy of Sciences Building, Lecture Room
2101 Constitution Avenue, NW, Washington, DC 20418

Suggested Citation:"Appendix C: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2020. Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25627.
×

DAY 1: September 25, 2019

8:00 a.m. Breakfast available outside the Lecture Room
8:30 a.m. Welcome and opening remarks
HARVEY FINEBERG, Workshop Chair

President

Gordon and Betty Moore Foundation

Highlights and related recommendations from the National Academies report on Reproducibility and Replicability in Science
9:15 a.m. Q&A with audience
SESSION I CULTIVATING TRANSPARENT REPORTING IN BIOMEDICAL RESEARCH

Session Objectives:

  • Highlight current efforts by researchers, institutions, funders, and journals to increase transparency in proposing and reporting preclinical biomedical research.
Suggested Citation:"Appendix C: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2020. Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25627.
×
  • Discuss the incentives, disincentives, challenges, and opportunities for researchers when it comes to transparent reporting of preclinical biomedical research (e.g., pressure to publish, institutional resources, training, funding).
  • Discuss experience with implementation of policies to encourage transparent reporting across the biomedical research life cycle.
  • Consider the role of stakeholders in supporting a cultural shift toward transparent reporting in preclinical biomedical research. For more information on cultural barriers as sources of non-reproducibility, see p. 58, p. 97, and p. 104 of the National Academies’ Reproducibility and Replicability in Science consensus study report.
9:30 a.m. Opening remarks by session moderator
ALEXA MCCRAY

Professor of Medicine

Harvard Medical School

9:40 a.m. A researcher (early career) perspective
YARIMAR CARRASQUILLO

Investigator

National Center for Complementary and Integrative Health, National Institutes of Health (NIH)

9:55 a.m. A researcher/researcher support perspective
BRIAN NOSEK

Co-founder

Center for Open Science

10:10 a.m. A researcher (later career)/society publisher perspective
ARTURO CASADEVALL

Professor, Molecular Microbiology and Immunology, Johns Hopkins University

Editor-in-Chief, mBio

10:25 a.m. An NIH perspective
CARRIE WOLINETZ

Acting Chief of Staff and Associate Director for Science Policy

Office of the Director, NIH

10:40 a.m. Audience Q&A with the panel
Suggested Citation:"Appendix C: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2020. Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25627.
×
Discussion questions:
  • What forces are influencing the culture of biomedical research, and how is it changing?
  • What actions could influence practice and support a cultural shift toward more transparent reporting?
  • What influence might transparent reporting or required reporting of rigor elements have on grant applications? Is there a role for preregistration of preclinical studies?
11:10 a.m. BREAK
SESSION II ANSWERING THE CALL FOR CHANGE: LESSONS LEARNED AND BEST PRACTICES

Session Objectives:

  • Consider lessons learned from institutional and/or field-specific best practices for increased transparency in reporting rigor elements (i.e., research design, methodology, analysis, interpretation, and reporting of results) that are generalizable across biomedical research domains.
  • Consider available tools and best practices for increased transparent reporting that support researchers and are generalizable across biomedical research domains.
  • Discuss the roles of educational institutions, professional societies, researchers, and funders in improving computational reproducibility (Reproducibility and Replicability in Science Recommendation 6-6).
  • Discuss how funding agencies and organizations could invest in research and development of open source, usable tools, and infrastructure that support reproducibility for a broad range of studies across different domains in a seamless fashion, as well as in outreach to inform and train researchers on best practices (Reproducibility and Replicability in Science Recommendation 6-1).
11:30 a.m. Opening remarks by session moderator
VERONIQUE KIERMER

Executive Editor

Public Library of Science (PLOS)

Suggested Citation:"Appendix C: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2020. Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25627.
×
11:40 a.m. A clinical researcher perspective: Lessons from the SPIRIT initiative
AN-WEN CHAN

Phelan Scientist, Women’s College Research Institute

Associate Professor, University of Toronto

11:50 a.m. An institution perspective
GEETA SWAMY

Vice Dean for Scientific Integrity

Associate Vice President for Research

Duke University

12:00 p.m. A funder perspective
MAGALI HAAS

Chief Executive Officer and President

Cohen Veterans Bioscience

12:10 p.m. Moderated panel discussion among speakers
12:30 p.m. Audience Q&A with the panel
Discussion questions:
  • How can challenges with preregistration, image analysis, cell line authentication, statistical analysis, or other rigor elements be addressed?
  • What actions can institutions or professional societies take to educate and support their constituents on best practices? How could this information be best provided?
  • How might funding agencies and organizations invest in development of open source reusable tools and infrastructure to support transparent reporting seamlessly across different domains?
  • What actions could funding agencies and organizations take to inform, train, and support researchers on best practices in transparent reporting?
  • What has been learned from open access mandates and from implementing policies around sharing data in preclinical research? How could those lessons inform transparent reporting guidance and adoption?
1:00 p.m. BREAK (Lunch available outside the Lecture Room)
Suggested Citation:"Appendix C: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2020. Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25627.
×
SESSION III STAKEHOLDER PERSPECTIVES ON CHECKLISTS AND GUIDELINES

Session Objectives:

  • Discuss journal and funder assessments of researchers’ adherence to transparent reporting guidelines, including discussion of the effectiveness of checklists.
    • Highlight empirical assessments of checklist application from funders, journals, and researchers; and
    • Consider practical application and effectiveness of checklists and guidelines to encourage or require transparent reporting of preclinical biomedical research.
  • Discuss how funders could require thoughtful discussion in grant applications of how uncertainties will be evaluated, along with any relevant issues regarding replicability and computational reproducibility (Reproducibility and Replicability in Science Recommendation 6-9).
  • Discuss how journals and scientific societies could disclose their policies relevant to achieving reproducibility and replicability, and how journals could be encouraged to set and implement desired standards of reproducibility and replicability and adopt policies to reduce the likelihood of non-replicability (Reproducibility and Replicability in Science Recommendation 6-7).
2:00 p.m. Opening remarks by session moderator
BARRY COLLER

Physician-in-Chief, Vice President for Medical Affairs, and David Rockefeller Professor

The Rockefeller University

2:10 p.m. The checklist approach at life science journals—challenges and opportunities
SOWMYA SWAMINATHAN

Head of Editorial Policy

Nature Research

MALCOLM MACLEOD

Professor

University of Edinburgh

Suggested Citation:"Appendix C: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2020. Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25627.
×
2:30 p.m. An NIH funder perspective
SHAI SILBERBERG

Director, Research Quality

National Institute of Neurological Disorders and Stroke, NIH

2:40 p.m. Moderated panel discussion among speakers
3:10 p.m. Audience Q&A with the panel
Discussion questions:
  • How valuable are checklists? How valuable is guidance such as the CONSORT statement? What are observed challenges to adherence, and how could they be addressed?
  • How could checklists be improved and/or complemented to further encourage transparent reporting?
  • What resources do researchers need to be able to submit proposals, publish, or otherwise report on specific rigor elements?
  • How might funders require thoughtful discussion in grant applications of how uncertainties (e.g., in measurement, computation, knowledge, modeling, or methods of analysis) will be evaluated by researchers?
  • Should scientific societies be encouraged to develop policies relevant to transparent reporting?
SESSION IV PART 1 TOWARD MINIMAL REPORTING STANDARDS FOR PRECLINICAL BIOMEDICAL RESEARCH

Session Objective:

  • Discuss opportunities for improving the consistency of reporting guidelines and requirements for rigor and transparency by journals, funders, and institutions across the biomedical research life cycle.
Suggested Citation:"Appendix C: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2020. Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25627.
×
4:00 p.m. Discussion with audience on potential steps stakeholders could take to support harmonizing reporting guidelines
HARVEY FINEBERG, Workshop Chair and session moderator

President

Gordon and Betty Moore Foundation

BENEDICT KOLBER

Associate Professor

Duquesne University

RICHARD NAKAMURA

Former Director (Retired)

Center for Scientific Review, NIH

FRANKLIN SAYRE

STEM Librarian

Thompson Rivers University

VALDA VINSON

Editor, Research

Science

5:00 p.m. ADJOURN WORKSHOP DAY 1

DAY 2: September 26, 2019

8:00 a.m. Breakfast available outside the Lecture Room
8:30 a.m. Welcome and overview of day 1
HARVEY FINEBERG, Workshop Chair

President

Gordon and Betty Moore Foundation

9:00 a.m. Keynote address
MARCIA MCNUTT

President

National Academy of Sciences

9:20 a.m. Q&A session
Suggested Citation:"Appendix C: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2020. Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25627.
×
9:30 a.m. BREAK
SESSION IV PART 2 TOWARD MINIMAL REPORTING STANDARDS FOR PRECLINICAL BIOMEDICAL RESEARCH

Session Objectives:

  • Consider approaches to compare reporting of rigor elements proposed in grant applications to those included in publications.
  • Suggest stakeholder actions to encourage transparent reporting and practical next steps toward establishing minimal reporting standards for preclinical biomedical research.
10:00 a.m. Opening remarks by session moderator
HARVEY FINEBERG, Workshop Chair

President

Gordon and Betty Moore Foundation

10:10 a.m. An early career researcher perspective
MICHAEL KEISER

Assistant Professor

University of California, San Francisco

10:20 a.m. An institution perspective
MELISSA RETHLEFSEN

Associate Dean, George A. Smathers Libraries

Fackler Director, Health Science Center Libraries

University of Florida

10:30 a.m. A research educator perspective
STEVEN GOODMAN

Professor of Medicine and Health Research and Policy

Co-director, Meta-Research Innovation Center at Stanford

Stanford University

10:40 a.m. Moderated panel discussion among speakers
11:10 a.m. Small group table discussion and reporting
Discussion questions:
  • What actions should funders, researchers, institutions, and journals take to drive widespread adoption of minimal reporting standards?
Suggested Citation:"Appendix C: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2020. Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25627.
×
  • Are reporting categories in guidelines for publishing (e.g., materials, design, analysis, and reporting) relevant for funders? For institutions? For small publishers/professional societies?
  • What other information or reporting categories would be relevant?
  • How should funders instruct reviewers of grant applications to reinforce transparent reporting? How much information should funders request, that is, to what level of detail, in grant applications? Is it possible to obtain sufficient information about transparent reporting in grant applications without dramatic expansion of the application?
12:25 p.m. Workshop wrap-up and concluding discussion with audience
12:30 p.m. ADJOURN WORKSHOP DAY 2
Suggested Citation:"Appendix C: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2020. Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25627.
×
Page 111
Suggested Citation:"Appendix C: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2020. Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25627.
×
Page 112
Suggested Citation:"Appendix C: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2020. Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25627.
×
Page 113
Suggested Citation:"Appendix C: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2020. Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25627.
×
Page 114
Suggested Citation:"Appendix C: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2020. Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25627.
×
Page 115
Suggested Citation:"Appendix C: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2020. Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25627.
×
Page 116
Suggested Citation:"Appendix C: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2020. Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25627.
×
Page 117
Suggested Citation:"Appendix C: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2020. Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25627.
×
Page 118
Suggested Citation:"Appendix C: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2020. Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25627.
×
Page 119
Suggested Citation:"Appendix C: Workshop Agenda." National Academies of Sciences, Engineering, and Medicine. 2020. Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/25627.
×
Page 120
Enhancing Scientific Reproducibility in Biomedical Research Through Transparent Reporting: Proceedings of a Workshop Get This Book
×
Buy Paperback | $58.00 Buy Ebook | $46.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Sharing knowledge is what drives scientific progress - each new advance or innovation in biomedical research builds on previous observations. However, for experimental findings to be broadly accepted as credible by the scientific community, they must be verified by other researchers. An essential step is for researchers to report their findings in a manner that is understandable to others in the scientific community and provide sufficient information for others to validate the original results and build on them. In recent years, concern has been growing over a number of studies that have failed to replicate previous results and evidence from larger meta-analyses, which have pointed to the lack of reproducibility in biomedical research.

On September 25 and 26, 2019, the National Academies of Science, Engineering, and Medicine hosted a public workshop in Washington, DC, to discuss the current state of transparency in the reporting of preclinical biomedical research and to explore opportunities for harmonizing reporting guidelines across journals and funding agencies. Convened jointly by the Forum on Drug Discovery, Development, and Translation; the Forum on Neuroscience and Nervous System Disorders; the National Cancer Policy Forum; and the Roundtable on Genomics and Precision Health, the workshop primarily focused on transparent reporting in preclinical research, but also considered lessons learned and best practices from clinical research reporting. This publication summarizes the presentation and discussion of the workshop.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!