Cognitive Aspects of Survey Methodology Building a Bridge Between Disciplines (1984) / Chapter Skim
Currently Skimming:
Appendix A: Background Papers
Appendix A: Background Papers
Pages 71-148
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 71... ...
The authors first present a conceptual model of response effects, i.e., sources of variation in the quality of data obtained in surveys, and a general model for human information processing. They then contrast the research methods used in the cognitive sciences and in survey research methodological studies.
|
From page 72... ...
Marquises paper reviews alternative designs for validity checks and discusses their strengths and weaknesses. ~~ ~~ ~ ~ ~ ~ ~ He argues that certain designs lead to wrong conclusions about the direction and size of survey reporting bias and gives some examples from record checks associated with health surveys.
|
From page 73... ...
In this review, I shall risk far more undercoverage than any survey researcher. I shall be purposive in my -sampling, fitting the selection to my twin aims: my review will focus on the areas of cognitive science that have the most direct bearing on how surveys are conducted, arid it will focus on one or two arbitrarily selected areas where the cognitive sciences could benefit most directly from the use of surveys.
|
From page 74... ...
In the next four sections I examine each of these processes-comprehension, recall, judgment, and response -- in more detail; I consider the main theoretical approaches used in studying them and attempt to draw out the implications of the theories for the practical problems of survey research. Comprehension Research on comprehension has, for reasons of methodological convenience, concentrated on the comprehension of written material.
|
From page 75... ...
75 notionn -- the profound impact of context, the use of prior general knowledge by the reader or listener, and the influence of inferential processes during comprehension. The difference in the approaches lies mainly in their views on the nature of the information we use in interpreting a text.
|
From page 76... ...
and the details of the passage fit into that picture. All this is not to deny the importance of prior knowledge or of higher-level cognitive structures in the interpretation process: we may not know much about building log cabins but we understand that Thoreau needs the axe for that purpose.
|
From page 77... ...
We go far beyond the information given in interpreting a text; we fill in gaps and add details, making inferences that our background knowledge and the text at hand seem to call for. The inferences we make depend on the prior knowledge that is activated by the passage and its context; context guides the selection of the relevant prior knowledge.
|
From page 78... ...
Survey researchers are hardly unaware of many of these points. Bradburn (1982)
|
From page 79... ...
The best I can hope to do is to highlight a few key ideas that relate most directly to the problems faced by survey researchers. Episodic Versus Semantic Memory Hemory researchers distinguish between different stores for information, ranging from the very brief persistence of visual information in iconic storage to the more or less permanent retention of information in long-term memory.
|
From page 80... ...
His encoding specificity principle states that sometimes we [ail even to recognize a previously learned item because at recognition the item [ails to reinstate the exact encoding it was given during learning. For example, when one puts eggs on a shopping list, he or she may fail to purchase the right item if candy Easter eggs were intended and hen's eggs were recalled.
|
From page 81... ...
The subjects drew such inferences as "smashed into" seemed to warrant; later they could not disentangle what they had seen from what they had inferred. Sources of Forgetting As with research on comprehension, so with research on memory: it is more than a little hazardous to extrapolate from the sort of research conducted by cognitive psychologists to the practical concerns of the survey researcher.
|
From page 82... ...
In longitudinal studies and studies using diaries of course there are more opportunities to train respondents in the use of these mnemonic tactics. Still, tricks that can be applied at the retrieval stage long after the relevant events have taken place are of more general use to survey researchers.
|
From page 83... ...
Survey researchers have not ignored the possibilities of using special diaries, logs, and so on for improving recall. Also, some surveys have incorporated existing records as a Jog to respondents' memories; a respondent's checkbook may contain the most useful cues to help him or her recall a visit to a doctor.
|
From page 84... ...
different judgment strategies. I then examine in more detail the process by which attitude questions are answered.
|
From page 85... ...
We use the availability heuristic when we judge the frequency of a class of events based on the number of examples we can generate or the ease with which we generate them. For example, people incorrectly .judge that words beginning with the letter R are more frequent than words whose third letter is R -- pre~umably because they can more easily call to mind words that begin with R
|
From page 86... ...
take pains to argue that, despite the thousand natural errors that the mind is heir to, we often do make valid judgments and sound decisions. Implications for Survey Research It would be nice for everyone if' we could reduce this long list of biases to one or two fundamental errors from which the rest could be derived, but no one has thus far succeeded in imposing order on this error-tilled crew.
|
From page 87... ...
Attitude Questions Attitude questions are an interesting case because very little is known about how we answer them. The process in likely to be quite complex, involving both the use of judgmental heuristics and the application of integration rules.
|
From page 88... ...
~ It is an if people saw these issues as manifestations of some larger pattern; in the case of busing, opponents seem to see it as an example of a general schema in which liberal "reforms pose a threat to traditional American values . In the case of symbolic attitudes, then, what gets retrieved in answering an attitude question is a kind of schema that includes deeply felt emotions.
|
From page 89... ...
Having rendered the Judgment that the question demands, he or she must now select (or formulate) a responses This section deals exclusively with the selection of a response from a pre-established Ret of response categories.
|
From page 90... ...
The widespread use of ~satisficing~ (Simon and Stedry, 1969) rules creates some headaches for the survey researcher.
|
From page 91... ...
, which are as well-known to survey researchers as to cognitive scientists, there are probably a number of subtler reporting biases . For example, people tend to exhibit consistency among their attitudes (Abelson et al., 1968)
|
From page 92... ...
Two phenomena -- forgetting and optimism-illustrate the advantages of adding a little more variety to the techniques of cognitive science. Forgetting The issue with forgetting is how far the principles that apply to forgetting in the laboratory generalize to other settings and other types of material.
|
From page 93... ...
impassivity to the notion that happy outcomes are better attended to and consequently are more likely to be recalled. Since most of the research on optimism has been conducted with Cal lege students, the question naturally arises whether these optimistic beliefs are not, in some sense, realintic -- after all, college students in the United States have a fairly cushy existence, sheltered from much of life 's storm and stress.
|
From page 94... ...
1976 ~~y~,_~,_~ ~~. Hilledale, N.J.: Lawrence Erlbaum Associates.
|
From page 95... ...
1972 Experiments on semantic memory and language comprehension.
|
From page 96... ...
~ . Hilladale, N J: · ~ Lawrence Erlbaum Associates.
|
From page 97... ...
97 Rintach, W., and van Dick, T 1978 Toward a mode]
|
From page 98... ...
Orne, M 1962 On the social psychology of the psychological experiment: with particular reference to demand characteristics and their implications.
|
From page 99... ...
M 1973 Encoding specificity and retrieval processes in episodi c memory.
|
From page 100... ...
loo Tversky, A 1972 Elimination by aspects: a theory of choice. PsYchn Review 79:281-299.
|
From page 101... ...
We then proceed to review some of the major response effects discussed in the survey research literature in the light of what appears to be the most relevant cognitive lit erasure. Conceptual Model of Response Effects.
|
From page 102... ...
The potential for response effects due to differences in interviewer behavior is real, even in the most tightly structured survey. The task should be defined carefully by the investigator.
|
From page 103... ...
Conceptual Model for Human lutormation Processing There are a number of models for human information processing; they differ in their details. We shall adopt a very general model that seems to be common to almost all approaches, without attempting to suggest that any particular model is the more nearly correct one.
|
From page 104... ...
Since a survey interview consists of a sequence of questions and answers, we look to cognitive studies for understanding of comprehension of oral or written text on the input side and of information retrieval from episodic memory on the output side. There are several additional concepts that are useful for thinking about information processing related to survey questions.
|
From page 105... ...
Temporal aspects of memory are important to survey researchers because many surveys involve questions about frequency of events or events that occurred at particular times. The placement of events in time may depend on events being coded with specific markers (e.g., dates)
|
From page 106... ...
the past learning of the respondents. Applying Cognitive Research to the Study of Response Effects Because we are reviewing sources of error in survey questioning in the light of cognitive research, we organize our discussion more in line with the questions that are most studied by survey researchers rather than the other way around.
|
From page 107... ...
College students had a great deal of difficulty understanding this passage unless they were told that it was about washing clothes before they heard it and thus knew the context within which to process the information. Pichert and Anderson (1977)
|
From page 108... ...
asked questions about support for the Korean War. The Gallup question was: ado you think the United States made a mistake in deciding to defend Korea, or note The NORC question was: Ado you think the United States was right or wrong in sending American troops to stop the Communist invasion of South Korean The NORC question consistently drew more support for the war than the Gallup question.
|
From page 109... ...
The problem of synonyms is a difficult one for question wording. Enough is known from methodological studies of question wording to know that the path is fraught with booby traps (Payne, 1951 )
|
From page 110... ...
A systematic investigation of different elements such an these in typically not done in surveys. Even with the same question wording, respondents may vary in the degree of generality with which they interpret questions.
|
From page 111... ...
Even this type of prompt may rail to elicit good recall since the respondents still must have the particular organizations that they are members of coded into the categories presented, and the cue of the category must trigger a search that will retrieve the fact that they are members. This type of prompting is called aided recall by survey researchers, and it produces a higher level Or response to recall questions than forms of the question without such aids.
|
From page 112... ...
The encoding specificity principle is useful in reminding us to pay more attention to the ways in which events are encoded and to develop the wording of questions TO as to match those codes. For example, significant subgroups in the population (e.g., those identified by ethnic group, region, education ~ may have different ways of encoding some types of events so that we might have to alter question wording for different groups.
|
From page 113... ...
The incorporation of post-event information can be attenuated to the extent that respondents are more certain about particular memories it recall has been made prior to the introduction of the additional information and if the source of the information is clearly suspect (Loftus, 1982~. Response Categories One of the oldest and most puzzling phenomena in survey research concerns the differences between open- and closed-ended questions.
|
From page 114... ...
An experiment by Bradburn et al . ~ ~ 979 ~ compared the effects of preceded response categories and verbatim recording on behavior reports about topics for which there is a presumption of considerable underreporting, such an alcohol consumption.
|
From page 115... ...
Respondents rarely use the Bothers category in closed forms. Thus it would appear that one of the important effects of providing response categories is to define much more narrowly the range of categories that in to be included in the response to the question.
|
From page 116... ...
Thus the DK propensity dimension influences the giving of DK responses on both forms, but there is no special trait that distinguishes floaters as such, nor is there a special group of people to be set apart and described by thin term. Additional questions of interest to survey researchers concern the effects of different response categories, such an the use of middle alternatives as opposed to dichotomous agree/disagree or approve/ disapprove type questions, the effects of using differing numbers of points on rating scales, and the use of familiar analogies such as ladders or thermometers for giving ratings of intensity of [avorableness or unfavorablene~ toward specified attitude ob jects .
|
From page 117... ...
Somewhat similar results have been reported by Schuman and Presser ~ ~ 98 ~ ~ for general and specific attitude items related to abortion . Here higher levels of support were found for a general question about abortion than when the general question came before a series of specific questions about approval of abortion in particular situations, such as in cases of rape or damage to the mother's health.
|
From page 118... ...
In their review of methodological studies on response effects, Sudden and Bradburn (1974) found a slight overall negative response effect (that is, a net underreporting)
|
From page 119... ...
" The bounded-recall procedure requires a considerable amount of control over the data in order to provide interviewers with the correct information from the previous interview and thus has not been used as often as it deserves to be. Respondents in surveys are not only asked to report on their behavior or attitudes, but they are also asked to make judgments about how often something has happened or how frequently they fee]
|
From page 120... ...
They found that subjects were able to judge categorical frequencies in a surprise posttest : Judgments of category frequency increased as actual frequency increased, and mean judgments for items occurring with different frequencies were significantly different from each other. It has further been argued that the ability to make frequency judgments does not involve a learned component.
|
From page 121... ...
In the first place it is generally known that people are not very accurate in their absolute frequency Judgments. The typical result is that for E-LF events people tend to overestimate low frequency es and underestimate high frequencies ~ Hasher and Chromiak, ~ 977; Hintzman, 1969~.
|
From page 122... ...
The context in which an event is presented also affects the absolute size of frequency judgments but not the discrim~nability of events occurring with different frequencies. The general finding is that for items of equal frequency, if repeated items are placed in a different context on each occurence, then the judged frequency will be lower than if the context is the came on each occasion.
|
From page 123... ...
~ A number of the factors affecting frequency Judgments may lead, under some circumstances, to an incorrect assessment of the occurrence of behavior by the survey researcher. Age differences in the absolute size of response (e.g., Hasher and Chromiak, 1977; Warren and Mitchell, 1980)
|
From page 124... ...
The distinction between semantic and episodic memory (e.g., Tulving and Thomson, 1973) is in part a reflection of this fact.
|
From page 125... ...
Conclusion In this paper we have reviewed the kinds of problems that are of concern to those working in the field of response effects in surveys and related them to some of the pertinent theories and findings of cognitive research. A number of the [actors shown to have effects in the experimental literature may not have analogous results outside of the laboratory.
|
From page 126... ...
folds 3: Bradburn, N.M. 1983 Response effects.
|
From page 127... ...
L g72 Context effects on frequency Judgments or word sentences.
|
From page 128... ...
Neter, J., and Waksberg, J 1964 A study of response errors in expenditures data from household surveys.
|
From page 129... ...
Tulving, E., and Thomson, D.M. 1973 Encoding specificity and retrieval processed in episodic memory.
|
From page 130... ...
These studies also suggest that the net response biases for survey estimates of health service use are close to zero. Some implications of the zero net survey bias are mentioned in the last section.
|
From page 131... ...
Net survey bias is the discrepancy between the survey estimate of the population characteristic and the true value of the characteristic. In the framework used here, the survey estimate is (A + B)
|
From page 132... ...
Illustration 1: Bins Estimates From Partial Designs are Too Large Because the Estimators Use the Wrong Denominator One of two problems in the partial design bias estimators is that they use a denominator that excludes come relevant information. This results in an estimate of the response bias that is too large.
|
From page 133... ...
= ~ e .10 ; C = - 10 - -.17 A ~ C 60 . The correct denominator for an AC design estimate is A + B ~ C + D and, if used in this example, provides the desired estimate of the survey bias, _ 10 = _.10.
|
From page 134... ...
The illustration to follow shows that, under the "no nonmatch" a~s~,mption, the full design estimate of net response bias is unaffected by match errors (see Neter et al., 1965, for the proof) , but the partial design estimates are biased when match errors are present.
|
From page 135... ...
Both the adjusted AB and AC approaches overestimate the ~ absolute value of the) survey bias by including half of the unsystematic (off setting ~ errors in their estimate of the systematic survey bias .
|
From page 136... ...
Since random mistakes probably are inevitable, the continued use of incomplete record-check designs will continue to mislead us about the direction and size of net survey response biased. Illustration 4: The Combined Effects An a final example, I have generated a matrix reflecting the various kinds of systematic and unsystematic errors discussed above to chow how full and partial record-check design estimators handle them.
|
From page 137... ...
Record-Check Estimates of Reporting Errors in Health Surveys How well do the logic-based principles of record-check problems hold up in practice? In this section we look at actual record-check estimates of survey reporting bias for hospital stays and physician visits.
|
From page 138... ...
bias and the AB designs imply a net positive (i.e., telescoping ~ response bias. The two ABC design studies come close to full design procedures; both find approximately equal rates of underreporting and overreporting; one estimates a slightly higher relative underreporting rate while the other estimates a slightly higher overreporting rate.
|
From page 139... ...
139 Cot a.
|
From page 140... ...
(1979) used the information in these studies to pr de full design estimates of the net response bias.
|
From page 141... ...
141 o ~ A: o ~ ~ ~ C' a: o _ ho · o ho A: o o Cat o of lo; · - be of I: of C)
|
From page 142... ...
The empirical findings from Holly design" record checks of hospital slay and doctor visit reports suggest that the net response bias is approximately zero, at leant for surveys designed an those involved in the cited record checks. ~ see three implications of these findings.
|
From page 143... ...
Thus, cognitive research that focuses only on forgetting hospital stays and doctor visits may not have much applied value for contemporary health surveys. A second implication for cogn' Live research on health surveys in that it should contain criterion validity features such as fully designed record checks or carefully thought-out strategies of construct validity.
|
From page 144... ...
Thus, a different error model may be needed for reporting of hospital stays and physician visits. We do observe, in the Feather data, an increase in both kinds of response errors with greater elapsed times between the survey and the event occurrence.
|
From page 146... ...
Balamuth, E., Shapiro, S., and Densen, P.~. 1961 Health interview responses compared with medical records.
|
From page 147... ...
H 1 97Ba Inferring health interview response bias from imperfect record checks.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.