Proceedings of a Workshop

OPEN SCHOLARSHIP PRIORITIES AND NEXT STEPS

Proceedings of a Workshop—in Brief

The Roundtable on Aligning Incentives for Open Science of the National Academies of Sciences, Engineering, and Medicine (the National Academies) brings together stakeholders to discuss the effectiveness of current incentives for adopting open science practices, barriers to adoption, and ways to move forward. According to a 2018 National Academies report, open science “aims to ensure the free availability and usability of scholarly publications, the data that result from scholarly research, and the methodologies, including code or algorithms that were used to generate those data.”¹ With the Roundtable coming to the end of its initial phase, a virtual workshop, held December 7, 2021, provided an opportunity to review lessons learned over the past 3 years and discuss next steps for Roundtable members, the National Academies, and others interested in advancing open science and open scholarship.

OPENING REMARKS AND GOALS OF THE WORKSHOP

Roundtable co-chair Thomas Kalil (Chief Innovation Officer, Schmidt Futures) thanked sponsors and participants and noted that the group’s two public workshops to date² have highlighted the diversity of activities under way, surfaced best practices and promising experiments, identified challenges, and built a “durable coalition of the willing and able as we turn from information-gathering to action.” Open science, he stressed, is not a goal in itself but a means to maximize research utility and democratize information. However, he continued, the current incentive structure and established metrics and markers push scientists and institutions to maintain the status quo, and the current system discourages bold thinking, sharing, and innovation. It also limits knowledge of adjacent fields and disciplines and reduces the ability to take full advantage of machine learning and other technological advances.

Three years into co-chairing the Roundtable with Keith Yamamoto (Vice Chancellor for Science Policy and Strategy, University of California, San Francisco), Kalil said he is more convinced than ever that open science means more rewarding, transparent, inclusive, and reproducible science. He noted the toolkit published as part of Developing a Toolkit for Fostering Open Science Practices: Proceedings of a Workshop³ is helping organizations find on-ramps to open science, and the investment is starting to pay dividends to change practice.

Kalil noted that hundreds of consultations across several fields helped crystallize the Roundtable’s overall theory of change, by which mutually reinforcing vectors are developed so researchers hear consistently from different directions that open practices are encouraged and rewarded. It is important to apply and scale the theory of change, and he recognized the efforts of many other people and groups who are advancing new ideas to create an open science and open scholarship ecosystem.

Yamamoto emphasized open science’s opportunities, challenges, and urgency. Policies in academia, industry, and funding agencies are holding back the progress that could be made in science and technology, he said. Recognizing the issues is a critical first step, and he agreed with Kalil about the value of a coalition of the willing across sectors.

__________________

REPORT-OUT ON THE PRESIDENTS’ COHORT

Randolph Hall (Director, Center for Risk and Economic Analysis of Threats and Emergencies [CREATE], and Professor, University of Southern California) reported on a Roundtable-inspired initiative focused on engaging college and university presidents to help lead change toward open scholarship through the alignment of incentives within their institutions. This alignment involves multiple entities, including academic units, faculty affairs offices, libraries, information technology, and research, all of which converge in reporting to the president. The goal with these leaders is to raise awareness about open scholarship, motivate action, participate in a cohort, and engage to ensure success. The approach is to lead by example, with strong support by the presidents who are already members of the Roundtable and with succinct, clear communication. In early 2021, an ad hoc committee developed a two-page guide on Open Scholarship.⁴ A convening activity that included presentations from four presidents and a variety of stakeholder perspectives occurred in October 2021 with follow-up to invite participation in an ongoing cohort.

In a pre-recorded video, Ronald J. Daniels (President, Johns Hopkins University [JHU]), one of the participating presidents, spoke about the importance of open science to institutions of higher education, and more broadly to science and to democracy. “In addition to its other benefits, a thoughtful and purposeful move toward open science could play an important role in bridging the widening gulf between experts and the public,” he said. To answer those who say that openness will come along on its own and there are higher priorities, Daniels countered, “If we don’t embrace openness now, we may find ourselves on the wrong side of history,” given that the research enterprise is on the edge of technological disruption. Evidence includes a flood of open access journals, many of which are legitimate but some of which are predatory.⁵ With the rise in the number of preprints in biomedical papers, for example, COVID-19 has shown how more openness in sharing data and results can lead to breakthrough discoveries, although he acknowledged the propagation of some falsehoods and inaccuracies. “We should not blindly follow openness wherever it might lead,” Daniels said. “We should not be utopian, but we should not turn away from the promise of openness.” He called for creation of policies and infrastructure to capture the opportunities while also building guardrails to protect peer review, integrity, rigor, and other features of the research enterprise. “Above all, that requires us to build towards this moment of disruption rather than wait passively for it to overtake us,” he stated.

Hall explained the follow-up with presidents asks them to elevate open incentives and practices as a strategic priority, designate a high-level representative who meets regularly with the president, and develop a timeline to create an open research strategy. Other aims are for campus engagement and collaboration, both within and across institutions. At the time of the workshop, the Presidents’ cohort was comprised of 57 institutions in 29 states, including 8 of the top 12 U.S. News-ranked universities and three Historically Black Colleges and Universities. Outreach about this initiative may be extended to other stakeholders, including journals, professional societies, funders, and others.

Discussion

Greg Tananbaum (Roundtable Secretariat) noted that the community of practice is flexible by design. The main requirement is a will to engage, he stressed, and invited others to join. Participants noted that the messages about open science extend beyond academia, including public agencies, funders, and publishers. Yamamoto said he wanted to amplify that the Roundtable serves as connective tissue across these different groups. “One of the challenges in germinating change is that the problems are big and they involve different groups,” he recognized. It is easy for one group to say that its individual action will not solve the problem, and then no one acts and nothing happens, he noted, adding, “the connective tissue strategy that the Roundtable can undertake is an essential part of moving past that kind of blockade.”

EXTENDING THE WORK OF THE ROUNDTABLE “IN THE WILD”

The publication Developing a Toolkit for Fostering Open Science Practices: Proceedings of a Workshop was formally released several weeks prior to the workshop, explained Tananbaum, and the toolkit it contains is designed to help stakeholders move from thinking and talking about open science to doing and implementing it. It is a modular set of resources—including best practices, success stories, sample language, and rubrics—that can be used by university leadership, department chairs, librarians, philanthropic program officers, and other interested parties. Users are invited to adapt and

__________________

customize the toolkit to specific use cases, he stressed, which is why it is licensed under CC BY terms. Three panelists described how they have done so.

Erin McKiernan (Community Manager, Open Research Funders Group [ORFG]) reported on how ORFG has used the toolkit to advise funders on open policy, whether they are learning what open research is, developing their own policies and plans to map with their goals, or implementing action and communicating with stakeholders (Figure 1). In 2021, ORFG developed a 6-month Funder Cohort Program. The goal is by the end of the program, funders can move policies forward within their organizations. ORFG uses the toolkit to help funders consider what kinds of projects or outputs that they support might be included in an open policy, such as reimagining outputs and good practices. ORFG has also adapted the toolkit to provide guidance on preprints.

Belinda Orland (Senior Manager, American Heart Association [AHA]) continued by describing how AHA used the work ORFG did on preprints to develop its own preprint policies. AHA invests in research to accelerate the pace of scientific discovery, encourage innovation, enrich education, and improve the public good, all of which, she noted, can be maximized through open science. AHA developed open science policies in 2015 with the intent to build on them. It benchmarked against other funders and observed a growing trend toward preprint acceptance.

To better understand the usage of preprints, AHA is now using the primer adapted by ORFG for internal discussion with staff and volunteers, promoting preprints in its newsletter and website, and communicating about the policy with applicants, awardees, and others. AHA also shared the primer with other members of the Health Research Alliance. Orland explained that preprints are explicitly encouraged for AHA awardees before or alongside the formal peer-review process.⁶ AHA’s grant management system now allows preprints to be entered any place in an application that publication information is requested, which Orland noted is of particular benefit to early-career investigators.

Sayeed Choudhury (Associate Dean, Research Data Management, JHU) discussed expanding and augmenting the sections of the toolkit relevant to code and software. Choudhury noted that an important community of interest in this case are companies in the private sector, which have been developing and collaborating on open source software for nearly 20 years. Opportunities for openness related to code and software differ to publications, data, or other research outputs, he pointed out. For example, from a reproducibility standpoint, the sharing of code and software are critical.

A research product may consist of articles, data, and software, which can be translated for use outside the university with proper curation. Choudhury and colleagues adapted the sections of the primer focused on translation so

__________________

the outputs are more actionable and accessible to those not involved with the original code. He noted the importance of working with the private sector to develop relevant guidance to understand the barriers encountered in the past when the private sector tried to work with universities. One important recommendation, he related, is to create Open Software Program Offices, or OSPOs,⁷ which the private sector has used for many years. The model is being augmented for the government and university sector, and a group of OSPOs have come together to create a network (OSPO++)⁸ to amplify the primer. The Research Software Alliance has also shown interest in the toolkit, Choudhury said.

From a mechanical perspective, having software in a repository makes the world more open, he concluded. Beyond that, he urged thinking about open source software, like open science, as a means and not an end in itself. Talent and creative ideas are everywhere, and open source software is a way for universities and communities to work hand in hand with mutual respect. He said the primer serves as an excellent foundation for this goal.

HIGH-LEVEL LESSONS LEARNED FROM THREE YEARS OF THE ROUNDTABLE

As Secretariat of the Roundtable, Tananbaum presented a review of the Roundtable and next steps. Launched to bring together senior leaders across the research ecosystem to discuss incentives and barriers related to open research, he noted 2019 was spent in an information-gathering mode, 2020 was used to develop and test the toolkit, and 2021 focused on “socializing” the toolkit by working with others.

He identified several challenges. The first he referred to as the hero’s quest, in which people express a willingness to embrace open science practices or incentives, but only if or after another person or office takes action in a complicated web of university leaders, departments, professional societies, funders, and others. This relates to the second challenge, one of inertia. Transition to a system that rewards openness is pain-staking work, he acknowledged, and the easiest thing is to do nothing or simply give the idea lip service. The kinetic energy needed to overcome inertia is larger than what is at play now, he said. While there have been some visible changes in organizations, they have not been leveraged across disciplines and stakeholder classes. However, he continued, each challenge has honed Roundtable activity. A theory of change was developed resting on mutually reinforcing vectors and recognizing the need for system-level changes carried out in parallel and not serially.

To date, about 300 organizations have provided input and nearly 1,000 people are on the Roundtable’s distribution list. The Presidents’ cohort is an example of a collation to convert allies to more active members of a coalition of the willing. A cohort of professional societies, with assistance from the American Geophysical Union and the Federation of American Societies for Experimental Biology, is also being organized to provide guidance and suggest incentives within and across disciplines. “This is the flip side of inertia,” he commented. “We are starting to activate the energy that can create a body in motion.” The toolkit has been downloaded approximately 1,500 times in 57 countries to date. He noted the heterogeneous mix of universities and other stakeholders that have come together and the power of doing the work through the National Academies. He concluded that the interest and initiatives related to open scholarship reflect a deep appetite to make research more accessible, equitable, trustworthy, and rigorous.

UPDATE ON GLOBAL AND DOMESTIC OPEN SCIENCE DEVELOPMENT

Arthur Lupia (Assistant Director, National Science Foundation [NSF]; Co-Chair, National Science and Technology Council Subcommittee on Open Science) underscored that the conversations and work on open science are critical to the nation and the world. Millions of people will benefit from increased access to a reliable scientific corpus, he said, and thus there is an urgency to getting open science right. He noted the shared endeavor to increase the public value of rigorous, precise research that empowers people and improves the quality of life.

Lupia then explained that the goal of the Subcommittee on Open Science is to advance efforts to increase access to, and use of, the results of federally funded research and development. About 30 agencies are represented. At NSF, for example, funded researchers must send their articles to a public access repository that is being expanded and made easier to use. Deposit rates are up 230 percent per year over the last four years. Similarly, the National Institutes of Health has undertaken such efforts as PubMed Central and is continually innovating to support open science.

He noted that a key part of the Evidence Act, passed in 2018, created new data-sharing requirements by

__________________

government agencies and put the government on a path in which people can access government data in a way that is privacy protected but also provides increased access. Also under discussion are topics such as how to increase equity in access to federally funded research outputs; protect privacy and security; improve collaboration to support a more open, equitable, and secure scientific enterprise; improve the benefit-to-burden ratio of data-related open science practices; and provide the infrastructure necessary to achieve all of this.

Luke Drury (Chair, Open Science Task Force, European Federation of Academies of Sciences and Humanities [ALLEA]) explained that ALLEA represents more than 50 academies in more than 40 countries, both within and outside the European Union. ALLEA originated the European Code of Conduct for Research Integrity and has a strong focus on policy for open science, integrity, and related issues.⁹ Drury said he came to chair ALLEA’s Open Science Task Force when he drafted ALLEA’s response to Plan S, an initiative for open access publishing,¹⁰ in 2018. ALLEA welcomed the goal behind the initiative, he explained, but also noted that the move to full open access must be accompanied by concurrent reforms across the research system, especially for early-stage researchers, disadvantaged institutions and communities, and those working in specialist disciplines. In October 2021, ALLEA issued a statement on equity in open access during International Open Access Week.¹¹ In particular, “gold” open access publishing routes and large read-and-write deals contribute to inequitable structures within academia, ALLEA warned. “Open means not only being able to read the science, but also participate in the science,” he stressed, including the importance of open science for trust in science and scientists. Big commercial publishers may dominate, which affects researchers outside of large institutions, in smaller disciplines, and/or the Global South, Drury said. He called for incentives to be aligned properly and for open science to be inclusive.

Although academies do not have direct power, Drury said, they do have moral authority to promote open research as the correct way of doing science. It is important to invest human and financial capital in institutional and community repositories, including preprints.¹² Infrastructures need to be supported, especially with human resources to operate and advise how to use them, which he likened to the great libraries of the past. DORA (Declaration on Research Assessment) principles must be explicit in hiring, funding, and promotion, with evidence of open science outputs provided.¹³ It is important to be wary of commercial capture of the research process, he warned, as seen in some publishing. He cautioned to beware of unintended consequences and to think through the impact of incentives on all aspects of open research, including the need for community governance and community-owned science communication infrastructure.¹⁴

Drury concluded, drawing on a graphic from the European Open Science Cloud, that the goal of open science is to take science as it is and, through open access, FAIR (findable, accessible, interoperable, and reusable) data, and embedding citizen science, to transform science to where it should be.

Steven Crawford (Senior Program Executive, Scientific Data and Computing, National Aeronautics and Space Administration [NASA]) reported on NASA’s steps to transform to open science. In 2019, the Science Mission Directorate (SMD) released a strategy for data management and computing for groundbreaking science. The Chief Science Data Officer set up the Open Source Science Initiative to enable open source science in NASA SMD. Crawford clarified that Open Source Science builds on concepts from open source software, applied to the scientific process to accelerate discovery from project initiation through implementation. Open source science makes the science transparent, accessible, inclusive, and reproducible (Figure 2), he said.

__________________

In all, $130 million in divisional investments in data management and open source science will be made through FY2027. An initial release was SPD-41, a Scientific Information Policy that has become part of many funding solicitations.¹⁵ A Request for Information was released in late 2021 on proposed additions to the policy based on new federal guidance, NASA policy, National Academies studies, and community best practices.¹⁶

An initiative called Transform to Open Science (TOPS) will begin in 2022, with heightened visibility beginning in 2023, which NASA has designated the Year of Open Science. TOPS will increase understanding and adoption of open science principles and techniques in SMD mission and research communities; accelerate major scientific discoveries through supporting the adoption of open science; and broaden participation in the science by historically excluded communities.¹⁷ Opportunities include building on training materials, supporting open science in policy and solicitations, and creating clear guidance on licenses and how to share publications, data, and software. He said NASA would like to share its resources with others and engage with professional societies, universities, and other stakeholders.

Discussion

After the three presentations, Tananbaum asked Lupia what the open science community can provide the federal government as it develops policies and process. Lupia replied that the need to understand what works and evidence that shows how open science has made a demonstrable difference in people’s lives is a top priority. Tananbaum noted an essay, infographic, and database in the toolkit provide examples but agreed the evidence needs to be aggregated.

When asked to share insights from a European perspective, Drury said he agreed with earlier comments that moving to open science requires collective action, although pointed out that one lesson from Plan S is funders can have a major impact on the system. Plan S helped jump-start action on open access, he related.

CLOSING DISCUSSION

Tananbaum posed two questions to participants: What can the Roundtable do collectively to harmonize norms, practices, and incentives? What are we missing in the approaches discussed during the workshop?

Drury commented that the Astrophysics Data System provides a good model of a platform-agnostic discovery tool with data and publications accessible via a single search engine or interface. It also helps put research on an even footing, he added, in which the quality of an article is the focus rather than the specific journal where the article was published.

A participant suggested making sure vendors know about the toolkit, the expectations it contains and how their platforms need to adapt to meet these expectations. If features and functions are preset, research administrators do not have to set them up, which she said would be a great next step. Tananbaum observed that a potential benefit of

__________________

a community of practice with an agreed-upon framework is to have the market power to approach vendors and other providers with these expectations.

Another participant commented that movements of all types succeed when they have convergence: that is, people working in different spheres converge on a common set of issues. They observed that the Roundtable has the capacity to convene and converge as a powerful starting point.

The Roundtable co-chairs shared closing thoughts. Kalil reflected on the progress on open science reported by presenters but also echoed the concern to foresee and avoid unintended consequences. Yamamoto commented on the inspiration that came from the discussions. “This session has proven that in the realm of open science and taking on the challenges of changing the culture in fundamental ways, there’s a lot of activity,” he said. He said that level of activity can lead to coalitions of the willing “in which people act in areas where they can have some impact, knowing that in other sectors, other groups are pushing from their directions. This is what will make things move.”

DISCLAIMER: This Proceedings of a Workshop—in Brief was prepared by Paula Whitacre as a factual summary of what occurred at the workshop. The planning committee’s role was limited to planning the workshop. The statements made are those of the rapporteur or individual workshop participants and do not necessarily represent the views of all workshop participants; the planning committee; or the National Academies of Sciences, Engineering, and Medicine.

REVIEWERS: To ensure that it meets institutional standards for quality and objectivity, this Proceedings of a Workshop—in Brief was reviewed in draft form by Boyana Konforti, Howard Hughes Medical Institute (ret.), and Mark Musen, Stanford University. The review comments and draft manuscript remain confidential to protect the integrity of the process.

PLANNING COMMITTEE: Keith Yamamoto (NAS/NAM), University of California, San Francisco (Chair); Heather Joseph, SPARC; and Thomas Kalil, Schmidt Futures. Staff: Thomas Arrison, Director, Board on Research Data and Information (BRDI); Greg Tananbaum, Roundtable Secretariat; George Strawn, Scholar, BRDI; Ester Sztein, Deputy Director, BRDI; Emi Kameyama, Program Officer, BRDI; and Olivia Torbert, Senior Program Assistant, BRDI.

ROUNDTABLE MEMBERS: Thomas Kalil (Co-Chair), Schmidt Futures; Keith Yamamoto (NAS/NAM) (Co-Chair), University of California, San Francisco; Elizabeth Albro, U.S. Department of Education*; Danny Anderson, Trinity University; Roslyn Clark Artis, Benedict College; Chris Bourg, Massachusetts Institute of Technology; Courtney Brown, Lumina Foundation*; Anne-Marie Coriat, Wellcome Trust*; Michael Crow, Arizona State University; Mark Cullen (NAM), Stanford University; Ronald Daniels, Johns Hopkins University; Tashni-Ann Dubroy, Howard University; Susan Fitzpatrick, James S. McDonnell Foundation*; Maryrose Franko, Health Research Alliance*; Nicholas Gibson, John Templeton Foundation*; Daniel Goroff, Alfred P. Sloan Foundation*; Heide Hackmann, International Science Council*; Randolph Hall, University of Southern California; Robert Hanisch, National Institute of Standards and Technology*; Patricia Hswe, Andrew W. Mellon Foundation*; Adam Jones, Gordon and Betty Moore Foundation*; Renu Khator, University of Houston; Boyana Konforti, Howard Hughes Medical Institute*; Richard McCullough, Florida State University; Peter McPherson, Association of Public and Land-grant Universities*; Ross Mounce, Arcadia*; Sarah Nusser, Iowa State University; Loretta Parham, Atlanta University Center Woodruff Library; Heather Pierce, Association of American Medical Colleges*; Dawid Potgieter, Templeton World Charity Foundation*; Brian Quinn, Robert Wood Johnson Foundation*; Robert Robbins, University of Arizona; Jerry Sheehan, National Institutes of Health*; Barbara Snyder, Association of American Universities*; Bodo Stern, Howard Hughes Medical Institute*; Geeta Swamy, Duke University School of Medicine; Shirley Tilghman (NAS/NAM), Princeton University; Alan Tomkins, National Science Foundation*; Roger Wakimoto, University of California, Los Angeles; Thomas Wang, American Heart Association and Vanderbilt University Medical Center*; Jennifer Weisman, Bill & Melinda Gates Foundation*; Richard Wilder, Coalition for Epidemic Preparedness Innovations*; and Duncan Wingham, UK Research and Innovation*

* Denotes ex-officio member.

SPONSORS: This workshop was supported by the National Library of Medicine, the Open Research Funders Group, and the Robert Wood Johnson Foundation. During its first 3 years, the Roundtable also received support from the Arcadia Fund, Arnold Ventures, the Leona M. and Harry B. Helmsley Charitable Trust, the Open Society Foundations, Schmidt Futures, and the Wellcome Trust.

For additional information regarding the workshop, visit: www.nas.edu/brdi.

Suggested citation: National Academies of Sciences, Engineering, and Medicine. 2022. Open Scholarship Priorities and Next Steps: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. https://doi.org/10.17226/26557.

Policy and Global Affairs

Copyright 2022 by the National Academy of Sciences. All rights reserved.