Skip to main content

Reproducibility and Replicability in Science (2019) / Chapter Skim
Currently Skimming:

Appendix E: Conducting Replicable Surveys of Scientific Communities
Pages 231-234

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.


From page 231...
... Unfortunately, even some deviations from scientific protocols can produce significantly skewed results that provide little information about what one wants to measure. Attempts to measure or even accurately record the attitudes of scientists about potential concerns related to replicability and reproducibility face a particularly difficult task: for scientists in general, or even researchers in a particular field, there is no easily accessible comprehensive list of scientists or researchers, even within any given country.
Read the entire page →
From page 232...
... RESPONSE BIASES Minimizing potential biases related to sampling, however, is not just a function of defining a systematic, transparent sampling frame, but also a function of using probability sampling techniques to select respondents.
Read the entire page →
From page 233...
... Again, many of the published surveys of scientists on replicability and reproducibility issues either rely on single-contact data collections with limited systematic follow-up or do not contain enough published information for other researchers to ascertain the degree or potential effect of systematic nonresponse. QUESTION WORDING AND ORDER Survey results depend heavily on how questions are asked, how they are ordered, and what kinds of response options are offered (for an overview, see Schaeffer and Presser, 2003)
Read the entire page →
From page 234...
... In some surveys, 70 percent of Americans supported "affirmative action programs to help blacks, women, and other minorities get better jobs and education." In other surveys that rephrased the question and asked if "we should make every effort to improve the position of blacks and minorities, even if it means giving them preferential treatment," almost the same proportion, 65 percent, disagreed.1 This problem can be exacerbated by social desirability effects and other demand characteristics that have the potential to significantly influence answers. It is unclear, for example, to which degree author surveys sponsored by scientific publishers about a potential crisis incentivize or disincentivize agreement with the premise that there is a crisis in the first place.
Read the entire page →

Key Terms

  • question wording
  • potential biases
  • systematic nonresponse
  • response rates
  • public opinion

  • response biases
  • survey results
  • reproducibility issues
  • potential crisis
  • particular field

  • search strings
  • response options
  • biases related
  • potentially controversial
  • skewed results

  • nonresponse based
  • potential concerns
  • articles published
  • crisis incentivize
  • published surveys

  • disincentivize agreement
  • little information
  • mine academic
  • minimizing systematic
  • significantly skewed

  • provide little
  • biases minimizing
  • citation thresholds
  • disincentivize participation
  • data collections


    • This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
    More information on Chapter Skim is available.