Skip to main content

Currently Skimming:

5 Checklists and Guidelines
Pages 53-72

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.


From page 53...
... (Coller, Swaminathan) • Few institutions provide formal training in the design and conduct of research.
From page 54...
... . Sowmya Swaminathan, head of editorial policy and research integrity at Nature Research, and Malcolm Macleod discussed the impacts of several current checklists and provided an overview of the Minimum Standards Working Group's development and pilot testing of the materials, design, analysis, and reporting (MDAR)
From page 55...
... Although pilots needed to make many decisions to fly the B-17 safely, the number of decisions was finite; however, he said the number of decisions involved in performing and reporting science today is "virtually infinite." A B-17 pilot's dependence on others involved a limited team, while performing and reporting science depends on a greatly expanded universe of others. Finally, Coller said, the dependence on "black boxes" by pilots in 1935 was finite and he noted they could actually "kick the tires." In science, what happens in the black boxes can be vital, and it is increasingly difficult to know the quality (e.g., an error in one line of code in one algorithm can have far-reaching effects if that algorithm is used widely)
From page 56...
... More than 1,000 publications involving animal research were assessed for their reporting of the four key items recommended by Landis and colleagues as the minimum necessary for transparent reporting: blinding, inclusion and exclusion criteria, randomization, and sample size calculation (Landis et al., 2012)
From page 57...
... In this regard, 78 percent of respondents said they continue to use the checklist to some extent in their own 1work, regardless of planned journal submission (see Figure 5-1, panel B) .5 A B Areas where quality has improved Degree to which the checklist has seen following implementation of the checklist continued implementation irrespective of planned journal submission in the future Better reporting of statistics 83% Increased data deposition in public 22% 23% 58% repositories Better reporting of reagents 55% To a large extent To a moderate extent Better reporting on animal models 30% To a small extent Not at all Adoption of anti-biasing practices 27% 27% 28% Other 4% n = 172 FIGURE 5-1 Impact of checklist on published papers and research practice.
From page 58...
... Minimum Standards Working Group The Minimum Standards Working Group includes editors and experts in reproducibility from Nature Research, the Public Library of Science, Science/American Association for the Advancement of Science, Cell Press, eLIFE, Wiley, the Center for Open Science, and the University of Edinburgh. The aim of the working group was to "improve transparency and reproducibility by defining minimum reporting standards in life sciences," which Swaminathan said includes biological, biomedical, and preclinical research.
From page 59...
... MDAR Framework Elements Swaminathan elaborated on the four reporting categories of the MDAR framework, listing the key elements that the working group identified for each: • "Materials: biological reagents, lab animals, model organisms, ani mals in the field, unique specimens • Design: study/experimental design, protocols, statistics, method ologies, dual-use research consent • Analysis: data, code, statistics as relevant to analysis • Reporting: discipline-specific guidelines and standards." The framework also discusses two levels of reporting, the "minimum" required level and a recommended "best practice" level, both of 6 The three key outputs of the Minimum Standards Working Group are available at https://osf.io/xfpn4 (accessed November 20, 2019) , https://osf.io/bj3mu (accessed November 20, 2019)
From page 60...
... MDAR Checklist Pilot Testing Author and Editor Perceptions Survey The first objectives of the MDAR pilot test were to collect authors' and editors' perceptions of the checklist (e.g., usefulness, accessibility, missing elements, impact on manuscript processing times)
From page 61...
... Not all checklist items are relevant at all times, and a "dynamic checklist" that offers fields relevant for a particular journal, for example, might be useful. Agreement between assessors was limited, and confidence intervals were wide, but the areas of disagreement should be highlighted as areas to focus on with regard to clarity of the checklist item and the information provided in the elaboration and explanation document.
From page 62...
... A checklist, by definition, is "a list of items required, things to be done, or points to be considered, used as a reminder." Checklists are for the specific task at hand, he said. Less Is More A systematic review published in 2012 compared the completeness of reporting of randomized controlled trials in journals that had endorsed CONSORT versus those that had not (Turner et al., 2012)
From page 63...
... He observed that the discussions at this National Academies workshop have also emphasized the roles of all stakeholders, and he encouraged participants to ask themselves what they can contribute to creating change and promoting transparent reporting. The need for change in the research culture has been raised throughout this National Academies workshop, Silberberg said, and changing the culture requires education and a change to the incentive structure.
From page 64...
... They called for "the establishment of communities of champions within and across institutions to share resources, change culture, and support better training at all academic levels." Each stakeholder organization has champions for culture change, Silberberg said. To bring them together, NINDS has created a mechanism on its website for champions of rigorous research practices to self-identify and connect with others in their institution.11 NINDS is currently considering how best to support interactions of these communities of champions with others regionally, nationally, and even globally.
From page 65...
... DISCUSSION Motivating Action: Champions for Culture Change During the discussion, panelists expanded on the topic of the need for champions of culture change, including the need for grassroots efforts and what motivates stakeholders to take action. Institutional Leadership Coller observed that, although the focus of the workshop is transparent reporting, it has been noted throughout the discussions that reporting is the end of the process, and there should be more attention to improving the rigor of research from the start.
From page 66...
... Grassroots Stakeholder Efforts As an example of a grassroots approach to championing rigorous science, Macleod mentioned the UK Reproducibility Network. The network includes self-organized local groups of early career researchers who connect for mentoring and journal clubs that promote openness and reproducibility; stakeholders (e.g., journals, funders)
From page 67...
... Arturo Casadevall called on the scientific elite to step up and require rigorous research. "Most scientists today want to do rigorous good sci 12 See https://sites.google.com/view/evir-funders-forum/home (accessed November 20, 2019)
From page 68...
... He supported the concept of prioritizing key items and proposed pilot testing checklists to determine what might be useful for end users. Another problem, as illustrated by dual evaluation of the MDAR checklist discussed by Macleod, is the lack of agreement by checklist
From page 69...
... than the language used in the checklist instrument. Swaminathan agreed and said in implementing the Nature Research checklist, for example, they found that authors conflated the experimental unit with the number of times an experiment had been replicated, presented aggregate data from multiple experiments as if from a single experiment, and confused technical and biological replicates.
From page 70...
... Students become "sensitized to the weaknesses" in the literature in their field and are empowered and motivated to contribute to change. Macleod said the doctoral program in neuroscience at the University of Edinburgh is doing this by having small groups of students conduct a systematic review of the models they will use in their laboratory research.
From page 71...
... The Press Macleod mentioned that the UK government inquiries into the practices of the British Press included inquiries into press coverage of scientific issues. Reports about what causes or cures a disease one day often contradict what was reported the previous day.
From page 72...
... Depending on the type of intervention, a manufacturing process control chart could be used to monitor change, or a randomized controlled trial might be needed to determine benefit. A challenge, he said, is that vocabulary and methodology do not yet exist for this type of "research on research." He noted the need to proceed cautiously and in a scientific way when "demanding our colleagues and peers change their practice" -- he suggested developing the science and methodology and collecting evidence of the impact of interventions to effect lasting change to research practice.


This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More information on Chapter Skim is available.