Developing a Toolkit for Fostering Open Science Practices Proceedings of a Workshop (2021) / Chapter Skim
Currently Skimming:
Appendix C: Toolkit Elements
Appendix C: Toolkit Elements
Pages 31-104
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 31... ...
This database compiles research articles, perspectives, case studies, news stories, and other materials that demonstrate the myriad ways in which open science benefits researchers and society alike.
|
From page 32... ...
This table enumerates the range of research products stakeholders may choose to consider as they develop open science policies. The toolkit is primarily intended to assist university leadership, academic department chairs, research funders, learned societies, and government agencies about how such a toolkit might be used, what additional materials are needed, and how such a toolkit should be disseminated for broad adoption.
|
From page 33... ...
This increased accessibility facilitates better collaboration and outcomes as a function of collective intelligence. By prioritizing shared discovery over individual and institutional agendas, open science practices are spurring the knowledge economy, generating broad social and public benefits, strengthening cultural values for scientific literacy and education, and improving public policy and democracy (Tennant et al., 2016; Zuccala, 2010)
|
From page 34... ...
This momentum is reflected partly by the choices that researchers make regarding how their data are shared. In one survey, the number of researchers who reported making their data openly available increased from just over 55 percent to 64 percent between 2016 and 2018.
|
From page 35... ...
By facilitating freer flows of information among scientists, research institutions, and firms, open science practices can accelerate the discovery pro cess and commercialization of scientific research. The inherently transparent nature of open science also makes testing the repro ducibility and replicability of scientific research substantially more efficient.
|
From page 36... ...
The myriad of public and pri vate economic benefits created by the Human Genome Project (estimated at $965 billion and nearly four million jobs between 1988 and 2012; Tripp and Grueber, 2011) have established it as a model for the effective use of open data, providing a picture of what the future of science and innovation could look like with greater adoption of open science practices (SPARC, n.d.)
|
From page 37... ...
portal was designed according to best practices in open science to facilitate open, coordinated, and sustained data sharing to advance the United Nations 2030 Agenda for Sustainable Development, the Paris Agreement, and the Sendai Framework for Disaster Risk Reduction. In addition to enabling communication between researchers and governments, "data products and information derived from GEO data can be useful for individuals to better understand the environment in which they live and work, and pro tecting the health of their family, and better educating themselves, and through the positive results of many other generative and even serendipitous applications" (Benkler, 2006; Mayo and Steinberg, 2007; NRC, 2009; and Zittrain, 2006; cited in Uhlir, 2015, 13)
|
From page 38... ...
. The California Policy Lab utilizes the Open Science Framework and has established data-sharing agreements with more than a dozen county agencies in Los Angeles, Sonoma, and San Francisco covering "medical, mental health, criminal justice, social service, and homeless management information systems" (Cali fornia Policy Lab, 2018)
|
From page 39... ...
. Researchers become understandably hesitant to make their data and findings openly available out of fear of being "scooped" by other researchers (Berg and Niemeyer, 2018)
|
From page 40... ...
Open science practices, in contrast to traditional models of knowledge production, emphasize that open, transparent, and collaborative research dissemination practices more properly balance collective, institutional, and individual benefits. Open science represents a positive evolution of the research endeavor along three dimensions: • Collaboration drives innovation with the potential for broad social impact.
|
From page 41... ...
Fortunately, the values that underpin open science -- such as inclusiveness, collaboration, social impact, and scientific literacy -- are mutually reinforcing to the missions of the research institutions, agencies, and funding organizations that support scientific research. Forward-thinking organizations have already begun to implement incentives for open science practices that provide a model for others to follow, which have taken several forms, including the following: 1.
|
From page 42... ...
The Bill & Melinda Gates Foundation and the Wellcome Trust, which funded $1.3 billion and $1.2 billion in global health research, respectively, joined a consortium of 11 European funding agencies that require all funded research to be free immediately upon publication. This incentive effectively requires scientists to publish papers in open access journals rather than those that charge subscriptions (Stokstad, 2018)
|
From page 43... ...
California Policy Lab.
|
From page 44... ...
2015. The Value of Open Data Sharing – A White Paper for the Group on Earth Observations.
|
From page 45... ...
Accessed January 19, 2021. Zittrain, J
|
From page 46... ...
instances where you have engaged in "open" activities (such as making articles open access and sharing data/code according to FAIR principles [Findability, Accessibility, Interoperability, and Reuse of digital assets]
|
From page 47... ...
instances where you have engaged in "open" activities (such as making articles open access and sharing data/code according to FAIR principles)
|
From page 48... ...
If applicable, describe, in the context of this funded project, (1) instances where you have engaged in "open" activities (such as making articles open access and sharing data/code according to FAIR principles)
|
From page 49... ...
3. Describe the impact that your openly available research outputs from this evaluation period have had from the research, public policy, pedagogic, and/or societal perspectives.
|
From page 50... ...
. Describe the impact that your openly available research outputs have had from the research, public policy, pedagogic, and/or societal perspectives.
|
From page 51... ...
to the extent that the researcher has generated the extent that the researcher has that the researcher has generated research outputs beyond articles generated research outputs beyond research outputs beyond articles and data, making at least one of articles and data, making some (more and data, making the majority of these materials openly available than one, but less than most) of these these materials openly available for access and reuse.
|
From page 52... ...
(include DOIs, if others. outputs have been used deeply open research outputs have been possible)
|
From page 53... ...
to the extent the extent that the researcher has researcher has generated research that the researcher has generated generated research outputs beyond outputs beyond articles and data, research outputs beyond articles articles and data, making at least making most of these materials and data, making these materials some of these materials openly openly available for access and reuse. openly available for access available for access and reuse.
|
From page 54... ...
to the extent outputs generated research outputs beyond researcher has generated research outputs that the researcher has generated openly articles and data, making at least one beyond articles and data, making some research outputs beyond articles and available for of these materials openly available (more than one, but less than most) data, making the majority of these access and for access and reuse.
|
From page 55... ...
have been used other disciplines, and/ open research deeply within a specific community. or outside of academia outputs (for (include DOIs, if this project/ possible)
|
From page 56... ...
to the extent researcher has generated research outputs the extent that the researcher has access and that the researcher has generated beyond articles and data, making most of generated research outputs beyond reuse. research outputs beyond articles these materials openly available for access articles and data, making these and data, making at least some of and reuse.
|
From page 57... ...
(b) consistently demonstrating open science hygiene (e.g., use frequently demonstrating good good open science hygiene of DOIs, ORCID iDs, Creative open science hygiene (e.g., use (e.g., use of DOIs, ORCID Commons licenses)
|
From page 58... ...
articulating a clear and demonstrates an awareness of demonstrates an intent to consistent plan to engage in at least some aspects of good engage in good open science good open science hygiene open science hygiene (e.g., use hygiene in most instances (e.g., (e.g., use of DOIs, ORCID of DOIs, ORCID iDs, Creative use of DOIs, ORCID iDs, iDs, Creative Commons Commons licenses)
|
From page 59... ...
consistently demonstrating open science hygiene (e.g., demonstrating good open good open science hygiene use of DOIs, ORCID iDs, science hygiene (e.g., use of (e.g., use of DOIs, ORCID Creative Commons licenses)
|
From page 60... ...
a in your discipline, in narrower range of their open narrower range of their open other disciplines, and/ access articles (for this project/ access articles (for this project/ or outside of academia period) have been used deeply period)
|
From page 61... ...
articulating a clear and demonstrates an awareness of demonstrates an intent to consistent plan to engage in at least some aspects of good engage in good open science good open science hygiene open science hygiene (e.g., hygiene in most instances (e.g., (e.g., use of DOIs, ORCID use of DOIs, ORCID iDs, use of DOIs, ORCID iDs, iDs, Creative Commons Creative Commons licenses)
|
From page 62... ...
. open science hygiene (e.g., use (e.g., use of DOIs, ORCID of DOIs, ORCID iDs, Creative iDs, Creative Commons Commons licenses)
|
From page 63... ...
articulating a plan that (b) articulating a clear and demonstrates an awareness of at demonstrates an intent to consistent plan to engage in least some aspects of good open engage in good open science good open science hygiene science hygiene (e.g., use of hygiene in most instances (e.g., (e.g., use of DOIs, ORCID DOIs, ORCID iDs, Creative use of DOIs, ORCID iDs, iDs, Creative Commons Commons licenses)
|
From page 64... ...
. science hygiene (e.g., use of DOIs, ORCID iDs, Creative DOIs, ORCID iDs, Creative Commons licenses)
|
From page 65... ...
articulating a plan that a plan that demonstrates (b) articulating a clear and demonstrates an awareness of an intent to engage in good consistent plan to engage in at least some aspects of good open science hygiene in good open science hygiene (e.g., open science hygiene (e.g., use most instances (e.g., use of use of DOIs, ORCID iDs, of DOIs, ORCID iDs, Creative DOIs, ORCID iDs, Creative Creative Commons licenses)
|
From page 66... ...
research outputs beyond outputs beyond articles and of recent (<5 years) research beyond sharing articles in making research outputs articles and data available data available openly for access outputs beyond articles and and data, including beyond articles and data openly openly for access and reuse; and and reuse.
|
From page 67... ...
make all activities in the future, make at least some research some research outputs beyond research outputs beyond articles appropriate research outputs beyond the open outputs beyond articles and articles and data available and data available openly for beyond articles and data sharing of articles and data available openly for access openly for access and reuse; access and reuse; and (b) engage available openly for access data and reuse.
|
From page 68... ...
. science hygiene (e.g., use of DOIs, ORCID iDs, Creative Commons licenses)
|
From page 69... ...
, enumerate make at least some research some research outputs beyond research outputs beyond articles appropriate research outputs your plans to engage in outputs beyond articles and articles and data available and data available openly for beyond articles and data open activities beyond data available openly for access openly for access and reuse; access and reuse; and (b) engage available openly for access sharing articles and data and reuse.
|
From page 70... ...
data principles support open research, data can be FAIR without being open. The FAIR principles can accommodate legitimate exceptions to open sharing practices such as data with PII, as mentioned above.
|
From page 71... ...
C Burgelman, Vrije Universiteit Brussel Anne-Marie Coriat, Wellcome Anne Koralova, Helmsley Trust Heather Pierce, Association of American Medical Colleges Dawid Potgieter, Templeton World Charity Foundation Greg Tananbaum, Open Research Funders Group Many organizations, particularly those that perform or fund research, are in the information-gathering stage with respect to their open science policies and practices.
|
From page 72... ...
Does the policy require that the articles be made openly available immediately, or is some embargo (e.g., 6 months) permissible?
|
From page 73... ...
in Harvard's institutional repository.b • The Academic Senate of the University of California adopted a systemwide open access policy in 2013 designed to make research articles authored by faculty available to the public at no charge.c a See https://obamawhitehouse.archives.gov/sites/default/files/micro sites/ostp/ostp_public_access_memo_2013.pdf. b See https://osc.hul.harvard.edu/policies.
|
From page 74... ...
can provide support and insight into best practices and available resources.8 The ORFG Incentivization Blueprint provides model language that can be adapted and adopted by funders and other organizations.9 It offers a stepwise approach to deploying a policy that can grow to encompass not only open access articles but also data, code, and other research outputs. 5 See http://researchsharing.sparcopen.org/articles.
|
From page 75... ...
They can also shed light on research that is not published, which can occur when tested hypotheses are not confirmed or research is considered unproductive, thereby mitigating publication bias and improving the efficiency of the research process, and can lead to novel lines of inquiry. In particular, shared data can be reused for new analyses, whether independently or in combination with other data.
|
From page 76... ...
• Data Management Plans. What support and guidance will the organization provide to help the researcher clearly articulate at the outset of a project what, how, and where data will be shared?
|
From page 77... ...
While data can be FAIR without necessarily being publicly open, the FAIR principles broadly support open science. Specific definitions and operationalizations of each of these principles, together with practical guidance on how to satisfy each requirement, have been prepared by the GO FAIR Initiative.10 To render data FAIR, metadata and datasets should be prepared in a standardized, descriptive manner that makes it easier for both humans and machines to find and use.
|
From page 78... ...
issued a Data Man agement and Sharing Policy that applies to all data generated by funded researcha as well as specific policies that apply to genomic data, clinical trial data, and other specific research programs and data types.b NIH has also provided information for selecting a data repository.c • The American Heart Association requires grant applicants to include a data-sharing plan as part of the application process. Any research data that are needed for independent verification of research results must be made freely and publicly available within 12 months of the end of the funding period (and any no-cost extension)
|
From page 79... ...
.12 Resourcing For data specifically, it is important to ensure that appropriate metadata and documentation are provided so that datasets are properly contextualized. Organizations will also benefit from in-house or outsourced expertise to assess the appropriateness of data management plans and informed consents, to ensure these allow data sharing to the extent that the organization desires.
|
From page 80... ...
• The Open Research Funders Group Incentivization Blueprint offers sample open data policy language that can be adapted for a range of use cases.15 • The American Heart Association's website contains a detailed FAQ page that articulates questions commonly asked by researchers subject to an open data policy.16 • The DMPTool site is an excellent resource for both browsing the data policies of hundreds of organizations and generating data management plans to fit a range of requirements and circumstances.17 • NIH is developing various resources to assist researchers in com plying with its Data Management and Sharing Policy, including clarifications about the contents of a data management and sharing plan, selection of data repositories, and allowable costs.18 14 See https://osf.io/bcj53. 15 See http://www.orfg.org/incentivization-blueprint.
|
From page 81... ...
Preregistration is particularly important for studies that make an inferential claim from a sampled group or population, as well as studies that are reporting and testing hypotheses. After a project is completed, protocols and preregistration analysis plans can be used in conjunction with the final study and analysis by researchers seeking to replicate, reproduce, and build upon findings.
|
From page 82... ...
For clinical trials of health-related interventions, NIH's ClinicalTrials.gov is the default system.19 Within the social, behavioral, and preclinical sciences, the Open Science Framework is becoming a default registry.20 Some public repositories tend to be disciplinarily focused. • Timing.
|
From page 83... ...
to provide unbiased evaluation and expert advice on developing protocols and statistical analysis plans, and evaluation of prepared study protocols.c • Arnold Ventures requires all funded empirical studies that involve statistical inference to be preregistered before the start of inter vention or data collection on the Open Science Framework.d See https://cziscience.medium.com/power-to-the-protocols a 388fe92001be and https://www.protocols.io/workspaces/neurodegeneration method-development-community1. b See https://www.socialscienceregistry.org.
|
From page 84... ...
Organizations and publishers will also need to ascertain how to indicate where preregistration records and protocol information exist for a published article. Multiple publishers and other organizations offer modalities for publishing study protocols, laboratory protocols, and registered reports.
|
From page 85... ...
APPENDIX C 85 The Center for Open Science provides multiple resources on how to preregister studies and analytic plans, including templates.25 NIH provides a number of resources to facilitate the development of protocols, including the National Institutes of Health e-Protocol Writing Tool and protocol templates for clinical trials and behavioral/social science research.26 25 See https://www.cos.io/initiatives/prereg and https://osf.io/zab38/wiki/home/? view.
|
From page 86... ...
• Research Scope. Registered reports are best for studies that test hypotheses and in disciplines that could suffer from publication bias (typically against null results)
|
From page 87... ...
Principal investigators can be encouraged to notify their communities -- via social media, their websites, CVs, and other appropriate channels -- when their precollection hypotheses and data analysis plans have been reviewed and registered. Organizations may also wish to educate researchers on the benefits of registered reports, particularly researchers in domains where the practice is not currently widespread.
|
From page 88... ...
The Center for Open Science also provides a simple Q&A tutorial to assist authors in the drafting of registered reports.30 The Open Science Framework provides a searchable database of registered reports across a range of disciplines.31 These may offer useful guidance to better understand the core elements of a well-constructed registered report. 28 See https://katiedrax.shinyapps.io/cos_registered_reports.
|
From page 89... ...
Does the policy require that the code or software be made openly available immediately upon the posting of research findings (e.g., publication of an article, deposit of a dataset) , or is some embargo (e.g., 6 months)
|
From page 90... ...
34 openly available. Both of these organizations deposit these research outputs (software as a product, not a byproduct, of the grant)
|
From page 91... ...
• The U.S. government's Federal Source Code Policy includes a pilot program that "requires agencies, when commissioning new custom software, to release at least 20 percent of new custom-developed code as Open Source Software for three years."b • Several learned societies that publish flagship disciplinary journals, including the American Geophysical Union and the American Astronomical Society, require or strongly encourage authors to make openly available any code used to generate results or analyses reported in their papers.c a See https://earthdata.nasa.gov/collaborate/open-data-services and-software/esds-open-source-policy.
|
From page 92... ...
Other organizations take a more engaged approach, requiring proof of compliance from researchers and checking this against internal expectations and guidelines. Next Steps The TOP Guidelines provide sample language for three levels of opencode policies.36 This wording can be adapted and adopted to suit the specific circumstances of a range of organizations.
|
From page 94... ...
The database is being developed by Arizona State University in collaboration with the Open Research Funders Group. An initial version is being made available as part of the background material for the November 5, 2020, National Academies workshop on Developing a Toolkit for Fostering Open Science Practices.
|
From page 95... ...
The authors believe it will be crucial to ensure that the science community takes an expansive view of the types of research products that should be "open" -- available for access and reuse without gatekeeping or payment. 40 The views expressed are those of the authors and do not necessarily reflect the official policies or positions of their employing organizations.
|
From page 96... ...
widely and freely available to any interested party. Succinctly, research findings are not fully open unless the tools necessary to understand and test them are also made available.
|
From page 97... ...
These materials extend the utility of primary research and widen the prospective audience to include policy makers and the general public. Data Subject to personal privacy, regulatory, The independent confirmation of results • Stewardship/ownership of and legal restrictions, data underlying and conclusions is critical for understanding repositories -- ensuring these are open specific claims in a research project scientific soundness and informing future and sustainable.
|
From page 98... ...
Monographs, Books, All monographs, books, book chapters, Unrestricted access to, and reuse of, • Open access for books and longer form Book Chapters, and/or and/or edited volumes should be made monographs, books, book chapters, and/ content is less developed than journals. Edited Volumes immediately available (open access with or edited volumes benefits the research Few options/models.
|
From page 99... ...
has undergone, and it provides them with a window into the editorial process. Because • Providing credit for peer reviews peer reviews are an essential component of without compromising anonymity (see the research endeavor, publishing referee ORCID PLOSa collaboration)
|
From page 100... ...
support preprints. For example, ASAPbio.org is a comprehensive resource for information on preprints, peer reviews, transparency, and so forth.c Transpose is a directory of journal policies, co-reviewing, and preprints.d
|
From page 101... ...
to conducting the work (i.e., preregistering When results are reported, make a with an analysis plan) the scientist makes • Also see the Registered Reports section, clear distinction between the planned a clearer distinction between planned below, as a way to practice prospective research and any unplanned research or hypothesis tests (i.e., confirmatory tests)
|
From page 102... ...
In publication bias against null results. It also to support registered reports activities is project time-line documentation, provides the benefits of preregistration limited.
|
From page 103... ...
and sustainable, including detailed or extend research findings should Biological materials, such as cell lines, are descriptions of samples. be made freely available in an open fundamentally different from data and repository no later than the publication even software as they may embody a type of the first paper based on the of "machine" that, through cell expression materials.
|
From page 104... ...
NOTE: CC BY – Creative Commons Attribution License.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.