A Decadal Survey of the Social and Behavioral Sciences A Research Agenda for Advancing Intelligence Analysis (2019) / Chapter Skim
Currently Skimming:
Appendix C: Reproducibility and Validity
Appendix C: Reproducibility and Validity
Pages 357-364
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 357... ...
Within scientific fields, particularly including branches of psychology and the life sciences and computer science, there has been a growing recognition that many studies cannot be replicated -- an issue that has attracted significant attention and is referred to as the replicability crisis. Advances in computational power, the collection of vast datasets, and the plurality of statistical methods have raised additional challenges associated with issues of the reproducibility, validity, and generalizability of results.
|
From page 358... ...
fields, in allied fields that use big data and computation, and in biometrics and behavior metrics is likely to continue, but in this appendix, we review the main components of reproducibility and describe three ideas that have been suggested for ensuring that research is reproducible. COMPONENTS OF REPRODUCIBILITY When researchers report the results of an experimental or computational study, another researcher or laboratory will ideally be able to carry out the same or a similar experiment and analysis, and derive similar findings and conclusions.
|
From page 359... ...
procedures to be repeated by others Reproducibility of results An independent study with procedures or methods ("replication") matched as closely as possible yields the same results, subject to statistical variation in samples Inferential reproducibility Equivalent inferences from the same data with independently conceived analyses Computation Empirical reproducibility Replication enabled by providing details of data methods and data collection and the data themselves being available Statistical reproducibility Specifying the models and statistical tests used and their parameters to enable independent replication Computational reproducibility Making the codes, software, hardware, and implementation details used to conduct the original research freely available Validation Surface validity Experimental procedures, results, and inferences are appropriate for the domain of application Model validation Methods for evaluating the match between the results of a model and data from the real-world system Robustness or generalizability Similar results and inferences obtain even with some variation in procedures or samples SOURCE: Generated by the committee drawing on Goodman et al.
|
From page 360... ...
The recommendations of the National Academies committee and discussions of standards and practices in specific field domains should support future research practices and implementations. PRACTICES Certain fields have embarked on systematic efforts to improve replication of experiments, while others have argued that replication may slow science by diverting resources (Bissell, 2013)
|
From page 361... ...
2 Bayesian analysis addresses research questions about unknown parameters using probability statements. Bayesian approaches allow researchers to incorporate background knowledge into their analyses, taking into account the issues of reproducibility and replication.
|
From page 362... ...
. Repeatability and reproducibility of fetal cardiac ventricular volume calculations using spatiotemporal image correlation and virtual organ computer-aided analysis.
|
From page 363... ...
Washington, DC: The National Academies Press. Open Science Collaboration.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.