Reproducibility and Replicability in Science (2019) / Chapter Skim
Currently Skimming:
7 Confidence in Science
7 Confidence in Science
Pages 143-162
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 143... ...
The chapter concludes with a consideration of public understanding and confidence in science. When results are computationally reproduced or replicated, confidence in robustness of the knowledge derived from that particular study is increased.
|
From page 144... ...
Research synthesis addresses the central question of how the results of studies relate to each other, what factors may be contributing to variability across studies, and how study results coalesce or not in developing the knowledge network for a particular science domain. In current use, the term research synthesis describes the ensemble of research activities involved in identifying, retrieving, evaluating, synthesizing, interpreting, and contextualizing the available evidence from studies on a particular topic and comprises both systematic reviews and meta-analyses.
|
From page 145... ...
. The ensemble of studies identified by the search is evaluated for relevance to the central scientific question, and the resulting subset of studies undergoes review for methodological quality, typically using explicit criteria and the assignment of quality scores.
|
From page 146... ...
Some research teams are beginning to monitor the scientific literature on a particular topic and conduct periodic updates of systematic reviews on the topic.2 Prospective research synthesis may offer a partial solution to the challenge of biased datasets. Meta-research is a new field that involves evaluating and improving the practice of research.
|
From page 147... ...
Events can be categorized according to their statistical properties, however, such as the parameters of their space, time, and size distributions. The satisfactory explanation of an emergent phenomena requires building a geosystem model (usually a numerical code)
|
From page 148... ...
The cyberinfrastructure constructed to support operational forecasting also enhances capabilities for exploratory science in geosystems. Natural hazards -- from windstorms, droughts, floods, and wildfires to earthquakes, landslides, tsunamis, and volcanic eruptions -- are notoriously difficult to predict because of the scale and complexity of the geosystems that produce them.
|
From page 149... ...
This is a familiar illustration of gaining confidence in scientific knowledge without doing repeat experiments. GENETICS One of the principal tools to gain knowledge about genetic risk factors for disease is a genome-wide association study (GWAS)
|
From page 150... ...
However, there is no consensus within the field on this point. Some researchers believe that the field is rife with lax methods that threaten validity, including low statistical power, failure to clarify between a priori and a posteriori hypothesis testing, and the potential for p-hacking (e.g., Pashler and Wagenmakers, 2012; Simmons et al., 2011)
|
From page 151... ...
As discussed throughout this report, no field of science produces perfectly replicable results, but it may be useful to estimate the current level of replicability of published psychology results and ask whether that level is as high as the field believes it needs to be. Indeed, psychology has been at the forefront of empirical attempts to answer this question with large-scale replication projects, in which researchers from different labs attempt to reproduce a set of studies (refer to Table 5-1 in Chapter 5)
|
From page 152... ...
Second, some high-profile replication projects (e.g., Open Science Collaboration, 2015) may have underestimated the replication rate by failing to correct for errors and by introducing changes in the replications that were not in the original studies (e.g., Bench et al., 2017; Etz and Vandekerckhove, 2016; Gilbert et al., 2015; Van Bavel et al., 2016)
|
From page 153... ...
, social scientists in communication research, psychology, sociology, and political science routinely analyze a variety of information disseminated on commercial social media platforms, such as Twitter and Facebook, how that information flows through social networks, and how it influences attitudes and behaviors. Analyses of data from these commercial platforms may rely on publicly available data that can be scraped and collected by any researcher without input from or collaboration with industry partners (model 1)
|
From page 154... ...
In both models, changes implemented by social media platforms in algorithms, APIs, and other internal characteristics over time make it impossible to computationally reproduce analytic models and to have confidence that equivalent data for reproducibility can be collected over time. In summary, the considerations for social science using big data of the type discussed above illustrate a spectrum of challenges and approaches toward gaining confidence in scientific studies.
|
From page 155... ...
The Science & Engineering Indicators surveys ask respondents about their understanding of three aspects related to the scientific process. In 2016, 64 percent could correctly answer two questions related to the concept of probability, 51 percent provided a correct description of a scientific experiment, and 23
|
From page 156... ...
Some data are available on uncertainties surrounding public opinion poll results. In a 2007 Harris interactive poll,7 for instance, only about 1 in 10 Americans (12%)
|
From page 157... ...
cientific researchers are dedicated people who work for the good of humanity." Even for potentially controversial issues, such as climate change, levels of trust in scientists as information sources remain relatively high, with 71 percent in a 2015 Yale University Project on Climate Change survey saying that they trust climate scientists "as a source of information about global warming," compared with 60 percent trusting television weather reporters as information sources, and 41 percent trusting mainstream news media. Controversies around scientific conduct, such as "climategate," have not led to significant shifts in public trust.
|
From page 158... ...
As a result, scientific news coverage often tends to favor articles about singlestudy, breakthrough results over stories that might summarize cumulative evidence, describe the process of scientific discovery, or delineate between systemic, application-focused, or intrinsic uncertainties surrounding science, as discussed throughout this report. In addition to being event driven, news is also subject to audience demand.
|
From page 159... ...
. Thus, there is currently limited evidence that media coverage of a replication crisis has significantly influenced public opinion.
|
From page 160... ...
Examples include concerns about hyperbolic claims in university press releases (for a summary, see Weingart, 2017) and a false balance in reporting, especially when scientific topics are covered by nonscience journalists: in these cases, the established scientific consensus around issues such as climate change are put on equal footing with nonfactual claims by nonscientific organizations or interest groups for the sake of "showing both sides" (Boykoff and Boykoff, 2004)
|
From page 161... ...
Similarly, no one should take a new, single contrary study as refutation of scientific conclusions supported by multiple lines of previ ous evidence.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.