National Academies Press: OpenBook
« Previous: CHAPTER 1 THE SEMINAR
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 25
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 26
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 27
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 28
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 29
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 30
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 31
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 32
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 33
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 34
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 35
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 36
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 37
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 38
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 39
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 40
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 41
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 42
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 43
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 44
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 45
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 46
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 47
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 48
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 49
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 50
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 51
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 52
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 53
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 54
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 55
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 56
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 57
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 58
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 59
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 60
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 61
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 62
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 63
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 64
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 65
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 66
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 67
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 68
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 69
Suggested Citation:"CHAPTER 2 AFTER THE SEMINAR." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 70

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

CHAPTER 2 AFTER THE SEMINAR Chapter ~ summarized the views expressed by the CASM project participants at the St. Michaels and Baltimore meetings, with emphasis on their suggestions for cro~s-disciplinary research by cognitive scientists and survey researchers. This chapter describes some of the outcomes of the project, including research plans and activities developed by partici- pants after the St. Michaels meeting and dissemination of project results through the publication or presentation of papers and other means. Each of the first six sections of this chapter describes a research program or activity initiated by CASM participants working as individuals or in small groups. The first four sections describe plans for rather substantial research efforts. The first section describes a multiyear collaborative research program involving cognitive scientists and survey researchers . The plan for this program, which is already under way, was developed by CASM participants Sirken and Fuch~berg for the National Center for Health Statistics (NCHS ~ . The program, described in the second section was developed by CASM participants Tourangeau, Salter, D'Andrade, and Bradburn, along with other cognitive scientists. The program, whose objective is to study the cognitive underpinnings of the survey interview process, is also funded and under way. The third and fourth sections contain prospectuses for survey collections of data that would be of considerable interest to cognitive scientists. Converse and Schuman propose to investigate personal interpretation of recent historical events for a sample of the U.S. population; funding for this project is expected soon. Tulving and Press present a proposal for a national memory inventory in which memory capabilities and other cognitive abilities would be tested for a large probability sample of the U.S. population; although the authors are not now in a position to pursue their proposal, they welcome and would cooperate with efforts by others to undertake the proposed research. The fifth and sixth sections describe research done by students under the direction of Loftus and Ross, two of the cognitive scientists who participated in the CASM project. The last section of. this chapter describes outreach activities: steps taken by the CASM participants to share the ideas developed during and after the seminar with others and to recruit new members of the interdisciplinary network that has been established. 25

LABORATORY-BASED RESEARCH ON THE COGNITIVE ASPECTS OF SURVEY METHODOLOGY National Center for Health Statistics (Monroe Sirken and Robert Fuchaberg) The research project outlined in this plan uses the National Health Interview Survey (NHIS) as a test bed for research and experimentation of the sort discussed at the CASH seminar. Purpose and Objectives Questionnaire design and data collection procedures are among the weakest links in the survey measurement process, and past efforts to improve the quality of survey instruments and procedures have posed serious and difficult methodological problems that are unlikely to be recolored by traditional survey research methods. Therefore, it is essential to test nontraditional modes for conducting research on survey methods. The objective of this project is to investigate the cognitive laboratory as the setting for conducting research on the cognitive aspects of. survey methodology. It will tackle three of the most important questions to emerge from CASH. Namely, under what conditions are laboratory methods 1 ikely to: (~) produce results similar to or different from traditional field methods? succeed where traditional methods have failed? enhance the results obtained by traditional methods? Although survey researchers and cognitive scientists are both concerned with the manner in which individuals handle information, their approaches to the problem and the methods used to study the problem are quite different, and there has been very little communication between them. Surrey researchers are concerned about the survey measurement process and use field experiments to test response effects in terms of the wording, response categories, and orderings of. questions. They make very little, it any, use of controlled laboratory experiments to investigate the ma=er in which the respondents and interviewers process the information presented by the survey instrument. The traditions method of developing, testing, and evaluating survey instruments involves sizeable field pretests and pilot studies of questionnaires that are developed by survey statisticians and tested under Formals survey conditions by trained interviewers. Cognitive scientists, on the other hand, are concerned about the system individuals use in processing information. Cognitive psychologists conduct controlled experiments in a laboratory setting involving direct and intensive interaction with relatively small samples of subjects to investigate the mental procedures by which information is processed. A mayor objective of this project is 26

27 to contribute to the advancement of both disciplines, and to effect communication between them, including collaboration in research studies. The demonstration will conduct laboratory-based research on the cognitive aspects of survey design using the combined methods of the cognitive and statistical sciences. Cognitive knowledge and techniques will be used to gain a better understanding of the effects of cognitive factors in the survey measurement process. From these laboratory findings statistical models will be developed for controlling survey measurement errors. Benefits The laboratory is the ideal setting for conducting interdisciplinary research in which the combined technologies of the cogniti Ire, social, biological, and computer sciences can be applied in researching the cognitive aspects of survey methodology. Participation of NCHS staff in the interdisciplinary laboratory, as described later in this plan when discussing collaborative arrangements, will help to bridge the gap that currently exists between government agency survey methodologists and university survey researchers and social scientists. There will be potential benefits for all disciplines. The project will provide additional methodologies for researching cognitive issues in surveys, new phenomena to examine in basic research in the cognitive and related sciences, and tested strategies for producing improvements in the methods and statistics of federal statistical surveys in general, and in NHIS, in particular. A note of caution is in order about the potential benefits of this project. It is not expected that the project will produce definitive substantive findings with respect to any cognitive issues, although it may provide important leads for subsequent research. The major emphasis will be methodological rather than substantive. Even so, it is recognized that the methodological findings obtained in thin or any single study will not be conclusive until verified by other researchers in subsequent trials. Collaborative Arrangements This demonstration project will be conducted in a collaborative mode. Since NCHS has neither a cognitive research laboratory nor a staff of cognitive scientists, it will make contractual arrangments with universities to have the experiments conducted in their laboratories and with their scientists. Not only will this arrangement be cost-effective for this project, but it will, as noted earlier, have the major long-term benefit of establishing closer research ties between the federal statistical establishment and universities. The NCHS and university laboratory staffs will collaborate in all research aspects of this project, and in the preparation of research reports, many of which will be suitable for publication in scholarly

28 journals. The NCHS will be primarily responsible for the -surrey and statistical methods and the contractor for the cognitive methods. Work Plan Experiments will be conducted to tent the application of laboratory-based methods for two broad types of questionnaire design problems: development of survey instruments (2) investigation of specific cognitive issues The NHIS questionnaire will be used by the laboratory as the survey instrument for both types of experiments. The supplement to the NHIS questionnaire will be used to test the development of survey instruments. Thin part of the questionnaire collects information on specific health topics (child care, health promotion, prescribed medicine, etc.) and changes annually. The specific cognitive issues will be generic to surveys and could arise in either the NHIS supplement or the core of the NHIS questionnaire. The latter collects basic information about the nation's health (health status, utilization of health services, eSc.) and undergoes virtually no change from year to year. The workplans for developing and pretesting a supplement to the NHIS questionnaire and for conducting laboratory research on specific cognitive issues relating to the NHIS questionnaire, respectively, are discussed in the next two sections. These plans were developed within the context of the collaborative mode in which the project will be conducted. On the one hand, these plans are intended not to overly restrict the Center 'a or the subcontractor' creativity as the research progresses. This is a concession to the nature of this research project, and also a major advantage of having the laboratory research conducted outride NCHS. On the other hand, the plans were developed with the view to pursuing certain objectives and producing particular products within a specified time frame. This is a requirement necessary to ensure project accountability, and also a major advantage of having the project administered by NCHS. Survey Instrument Development The Center's schedule of activities to develop and tent the 1987 NHIS supplement and a proposed schedule of the laboratory's activities are presented below.

29 Date ~.~e~ 1785 Develop analysis plan 6/85 Complete first draft of supplement 10/85 Prepare pretest version of supplement 12/85 Start 0~3 pretest clearance 3/86 4/86 Conduct field pretest Participate in field pretest 6/86 Prepare pilot study version of supplement 7/86 Start of OMB pilot study clearance 10/86 Conduct pilot study 10/86 Design NHIS supplement 1/87 Start 1987 NHIS Complete first draft of supplement Complete testing of first draft of supplement Complete tenting of pretest version of the supplement ~ contingent on a three- month extension of the laboratory subcontract In accordance with the Center's established timetable for constructing its annual supplements to the NHIS questionnaire, the topic for the 1987 NHIS supplement will be selected during 1984 and the literature search will be completed by January ~ 985, exact' y two years before the NHIS commences. During the two-year period, January 1985-1 987, the Center staff will be engaged in the tightly scheduled set of activities as noted above. These activities exemplify the traditional method of constructing survey instruments. The sine qua non of this method is that the instruments are field tented under conditions that simulate the actual survey conditions as closely as possible. This approach is in sharp contrast to the proposed laboratory activities which would be conducted under controlled laboratory conditions. The schedule of laboratory activities is linked to the NHIS schedule of established activities so as to maximize the laboratory 's potential contributions in developing and testing the ~ 987 NHIS supplement, subject to the condition that these activities should not interfere with nor Jeopardize the basic integrity of the NHIS established testing practices. This phase of the prospect will delineate the potential role of. laboratory-based research in developing and testing survey instruments. The project could result in the development of improved NHIS pretesting protocols, including improved field pretesting methods for training and debriefing interviewers, and ninnovative" laboratory-based methods for conducting unstructured interviews and group interviews. Developing the First Draft of the NHIS Supplement The laboratory will devote the five-month period, January-May 1985, to developing the first draft of the NHIS supplement. During this same period the NCHS staff will be independently developing its own first draft of the NHIS supplement.

So In early January 1985, NCHS will provide the laboratory with the items of information that will be collected in the 1987 NHIS supplement, and the contractor will transform these items into survey questions and procedures. Using cognitive techniques, such an protocol analysis, the laboratory will investigate the manner in which respondents process the required information, and on the basin of these findings, draft questions and design its own first draft of the NHIS supplement. Comparing the first drafts of the NHIS supplements that are developed separately by the NCHS and the laboratory will indicate the extent to which the laboratory method is a surrogate for the traditional NHIS method and to what extent it produces different results. Merging what appear to be the best features of both versions of the questionnaire and comparing the combined result with the questionnaire that was developed entirely by traditional methods will provide a basis for assessing the enhancement value, if any, of developing questionnaires in the laboratory as an adjunct to, or in place of, traditional developmental methods. Testing the First Draft of the NHIS Supplement During the six-month period from June 1985 to November 1985, the first drafts of the NHIS supplement will be pretested in the laboratory. Possibly three different versions will be laboratory tested: one that was developed by the NHIS staff, another developed by the laboratory staff, and possibly a third which incorporated what are Judged to be the best features of the other two versions. The laboratory will assess whether the drafts of the NHIS supplement are eliciting the kinds of information they are supposed to elicit. Laboratory testing will be performed on a variety of subjects who will be selected because they are expected to experience different types of cognitive problems with the questionnaire. The criteria for selecting sub jects will depend somewhat on the topic covered by the THIS supplement, but certainly they will reflect demographic, ether and economic differences in the population. The laboratory pretest findings will be discussed with NCH~ staff during November 1985, so that they can be incorporated into an improved draft of the questionnaire that the NHIS staff would be preparing to accompany its request [or OMB clearance in December 1985 to conduct a field pretest in 1986. Testing the Field Pretest Draft of the NHIS Supplement During the four-month period, December 1985 until the subcontract ends in March 1986, the field pretest versions of the NHIS supplements will be pretested in the laboratory and then independently field pretested during April 1986. Pretesting identical drafts of the NHIS supplement by both laboratory and field methods will make it possible to compare and evaluate how well each method assessed whether the NHIS supplement was doing what it is supposed to be doing and, if not, what revisions were needed. Relative costs and turnaround times of conducting pretests by each method would also be compared.

31 Specific Cognitive Issues As noted earlier, the manner in which people handle information is of common interest to cognitive and survey scientists, but objectives and methods of the two sciences are quite different. Survey scientists conduct field experiments to evaluate the quality of responses elicited by survey instruments. Cognitive scientists, on the other hand, establish generalizations about the mental systems people use for processing information by conducting laboratory experiments. The mission of this project is to demonstrate the enhancement value to both scientific fields of conducting laboratory research on particular cognitive issues that have been implicated by survey scientists as adversely affecting the quality of survey responses. Cross-fertilization of the two scientific fields is the keynote of this pro ject. Cognitive issues that arise in surveys are representative of. wider classed of cognitive phenomena that are being studied in cognitive science, but under restricted and unnatural laboratory conditions. Therefore, it is believed that bringing the survey cognitive issues and the survey experience with there issues into the cognitive laboratory will generate ideas for basic and applied research in cognitive science. And feeding the laboratory research findings on these cognitive issues back to survey scientists will, in turn, --simulate the development of improved statistical models of survey measurement errors and improved methods of constructing survey instruments. For cognitive science, the ultimate benefit will be a better understanding of the way people process information, and for survey science, it will be improved control over the cognitive component of survey measurement. Three well-known, but largely unresolved, survey problems are presented as possible candidates for laboratory research. They are: telescoping, conditioning, and the respondents ' perceptions of the confidentiality of their responses. Each problem involves cognitive issues that are poorly understood and, as will be explained later, seem to present interesting material for laboratory research. For example, for unknown reasons the effects of conditioning and telescoping are asymmetric. The conditioning effects of ordering questions or response categories are often more pronounced when ordered one, rather than another, way. Similarly, the telescoping effects of allocating events either to earlier or later periods than those in which they actually occurred usually results in inaccurately allocating fewer events to lens recent than to more recent periods. Telescoping Failure of respondents to recall events and to recall correctly when the events occurred are mayor sources of error in the collection of survey data. The errors associated with these two cognitive sources are often confounded in surveys, which may help to explain why the classical negative accelerated forgetting curve predicted by cognitive science does not necessarily hold in surveys.

32 The tendency of survey respondents to allocate events either to earlier or later periods than those in which the events actually occurred is called telescoping. Typically, survey respondents report retrospectively about events that occurred during a reference period, which is a calendar period of specified length that precedes the interview date. The telescoping phenomenon has been observed both for reference periods that are bounded by prior interviews and for unbounded reference periods. The findings of survey research indicate that unbounded recall has a net forward telescoping effect, that is, more events are shifted forward in time and erroneously reported in the reference period than are shifted backward and erroneously not reported in the reference period. In addition, events that are correctly placed within the reference period tend to be reported as having occurred more recently than they actually did. The importance of the event, the length of the reference period, and the characteristics of respondents all appear to have an effect on the telescoping phenomenon. With bounded recall there appears to be telescoping within the reference period itself, with the net forward effect being greatest for the most recent part of the reference period. The telescoping phenomenon is representative of a wider class of cognitive phenomena involving temporal judgments. There is no doubt that cognitive scientists appreciate that the process of making temporal judgments is an important component of event memory; however, their understanding of the event-dating process is based primarily on laboratory experiments which involve neither naturally occurring events nor long-term memory. Consequently, it is unclear how well existing cognitive theory on temporal judgment applies to the real world of personal events such as those respondents are asked to recall in surveys. Apparently, the telescoping phenomenon, per se, has not been investigated in the cognitive laboratory, and it is proposed that the survey experience with this phenomenon may offer interest)" leads for designing innovative laboratory experiments. Conditioning All scientific investigations are subject to the risk that the measuring instruments will disturb the phenomenon under observation and thereby affect the accuracy of its measurement. In this broad sense, the conditioning concept in survey science is analogous to Heidelberg 's uncertainty principle in physics, but without the latter ' ~ specificity. Conditioning in survey research usually refers to the distorting effect of the total survey measurement process on survey responses, but in the narrower sense used here it refers more specifically to the response effects of collecting an item or ret of items of information on another item or set of information items. For example, it refers to the response effects of adding one set of questions to another set of questions, such as the effects of NHIS core questions on the questions in the NHIS supplement or vice versa. It also refers to the effects of. ordering a particular set of questions or response categories, or reinterviewing the same respondents, an in panel and quality check

33 surveys. The narrower definition is adopted here because it makes the survey conditioning phenomenon more amenable to laboratory experimentation. Although there are many examples of conditioning effects in THIS and other scientific surveys, problems often arise unexpectedly since the phenomenon is not well understood by survey scientists. For example, in his CASH paper, Bradburn refers to a mysterious asymmetric effect of question and response ordering. He notes that a different effect is observed when questions are ordered in one way than when they are ordered in another way. It is proposed that this curious effect of conditioning in surveys may offer leads for designing critical laboratory experiments on the conditioning effects of rotating the order in which the material is presented. Perceptions of Confidentiality The response effects of asking for information about sensitive topics is a major survey concern because (1) policy makers and other users of health survey data often require this type of information, and (2) respondents are usually reluctant to provide this information and the quality of the information reported is often suspect. Examples of sensitive topics are: illicit behavior such as drug use, drunk driving, low-esteem behavior such as excessive drinking, overeating, and diseases with social stigma such as cancer, venereal diseases, tuberculosi a, mental illness, etc . Although scientific surveys subscribe to a strict policy of protecting the confidentiality of the reported information, assurances of this policy are often insufficient to overcome the suspicions of respondents that their responses may be disclosed in an identifiable form to third parties or their reluctance to report sock ally undesirable behavior to an interviewer. Survey scientists try to reassure respondents by using data collection techniques that seek to preserve the anonymity of the persons for whom sensitive information is reported in household surveys. Although it seems obvious that the success of these techniques would be greatly affected by the respondents' perceptions of the confidentiality protection afforded by these techniques, their perceptions have not been subjected to in-depth research and hence they are largely unknown. There are three survey techniques often used for preserving respondent anonymity: (1) Randomized response--a respondent is simultaneously presented the sensitive question and another non-sensitive question, each of which can be answered yes or no. He/she answers only one question and he/she alone knows which one, because he/she selected the question to be answered by a random process such as flipping a coin. (2) Network sampling--the respondent serves as an informant for other persons to whom he/she is linked by virtue of kinship, friendship, or some other designated relationship, but who are otherwise unidentified.

34 (3) Self-enumeration--the respondent writes the answers to the sensitive questions on a blank sheet of paper that he/she seals in a elf-addressed envelope and mails to the survey organization. The problem of respondent compliance in surveys on sensitive topics is representative of a wider class of cognitive phenomena in which people are faced with the task and the rink of making decisions on the basis of information that they may neither fully comprehend nor believe. Numerous applications of the anonymity techniques in surveys on sensitive topics have produced mixed results that traditional survey research methods have been unable to explain satisfactorily. It is expected that laboratory research on respondents' perceptions of these techniques under varying conditions may improve the design of surveys on sensitive topics and may lead to an improved understanding of the cognitive processes by which people assess risks on the basis of incomplete information.

COGNITIVE PROCESSES IN SURVEY RESPONDING: PR W ECT SUMMARIES Roger Tourangeau, Villiam Salter, Roy D'Andrade, Norman Bradburn, and associates Researchers at the National Opinion Research Center (NORC), Yale University, and the University of Chicago have proposed three interrelated research programs to carry out a series of studies on the cognitive underpinnings of the survey interview process. All three projects share a common framework, which is described here briefly. The framework assumes that respondents in surveys proceed through three mayor stages in answering survey questions: they interpret the question, retrieve the relevant information, and formulate a response. It is our shared belief that response effects in surveys can bent be understood by examining these processes in detail. Since different types of questions make different demands at each stage, we have also adopted a simple scheme for classifying survey quest ~ ons. We distinguish three broad classes: questions that elicit attitudes or opinions; questions that ask about behaviors; and questions that concern the causes or reasons for behavior. The framework thus suggests nine areas of. investigation def ined by the three stages of survey responding and the three types of questions. The NORC research program, developed by Roger Tourangeau, Roy D'Andrade, and Norman Bradburn, is entitled Recognitive Processed in Survey Responding: Attitudes and Explanations. ~ It concerns two types of survey questions--those concerning attitudes and reasons--and includes studies on all three stages of survey responding--interpretation, retrieval, and Judgment. The Yale program, developed by Robert Abelson, is entitled "Cognitive Processed in Survey Responding: Multiple Schemas and the Role of Affect. It deals with the same two classes of survey questions as the NORC research program and offers a complementary perspective on some of the same issues explored there. It dovetails with the NORC work in other ways--it extends the analysis of attitudes in terms of cognitive schemata and incorporates studies on the role of affect in survey responses. The University of Chicago program, developed by William Salter, Steven Shevell, Lance Rips, and Norman Bradburn, in entitled Recognitive Processes in Survey Responding: Time and Frequency Estimation.8 It focuses on the remaining class of survey questions, those that concern behavior. It includes studies on the retrieval and Judgment processes and on how these processes interact when respondents must Judge the timing or frequency of events. All three research programs share the interdisciplinary perspective of the Advanced Research Seminar on Cognitive Aspects of Surrey Methodology. The research teams for each project include researchers who have done cognitive research or survey research or work in both fields. In addit ion, all three include a commitment to the replication and extension of laboratory findings to the field setting. Each project incorporates plans for split-ballot studies to be conducted within the context of a national surrey. 35

36 Although the research has been divided into three separate projects, we expect to carry out the work as collaborators and to hold periodic meetings of the entire group. We believe that collaboration will enrich the quality of all our work and that the resulting whole will be greater than the Rum of the parts. NORC Research Program Roger Tourangeau, Roy D'Andrade, and Norman Bradburn We propose a series of laboratory studies that explore cognitive processes in survey responding. The studies focus on questions that elicit attitudes or opinions and on those that ask for the reasons or causes of behavior. In our view, knowledge and beliefs about attitude issues often form organized cognitive structures, or schemata. For the enduring issues that appear regularly in opinion surveys, several competing schemata are often available within a culture. Many respondents subscribe weakly to several of the available schemata for such issues; which schema guides their response to an attitude question is affected by variations in question wording and context. In a prestudy, we will identify the prevailing schemata for several typical attitude issues. The main studies will explore how question wording and context inf luence which schema is triggered by the question and, thus, affect how respondents interpret the question, think about the issue, and formulate their answer. Questions that elicit reasons initiate a process in which potential explanations are generated and then tested for plausibility. Because the generation step is not exhaustive and the tenting step not rigorous, respondents often accept the most readily available explanation. Three studies will explore the interpretation of questions that ask for explanations and the process of generating and evaluating potential explanations. The findings of the laboratory studies will be replicated and extended in a split-ballot field experiment. The research program will help resolve a number of methodological puzzles in the survey research literature, including the effects of question order, wording, and context. Yale Research Program Robert Abelson Research on cognitive processes is planned that will have direct and indirect bearing on the improvement of questionnaires used in surveys of attitudes and beliers. Early phases involve laboratory and small-sample questionnaire research. In the later phase, national samples of survey respondents will be used. Present plans call for three lines of research. The first is concerned with Affective carryover effects," in which the feelings associated with the response to one attitude question may influence

37 responses to later questions. The proposed studies experimentally create differing emotional contexts prior to critical target items. Results could have important implications for context artifacts in surveys. The second research area will address the circumstances under which respondents entertain multiple possible explanations [or their own and others' behavior, rather than single "sufficiently explanations. The third research line proposes methods, drawn from cognitive science, for distinguishing between symbolic and instrumental attitudes. In the former, beliefs are more rigidly attached to attitudes, and emotions may play a large role. The two types of attitudes may have different relations to demographic and other variables and may be important to distinguish in surveys. University of Chicago Research Program Lance Rips, Norman Bradburn, Steven She~rell, and William Salter This research will examine the cognitive processes and knowledge representations that are implicit in responses to many survey items. Survey questions that appear to require simple recall often involve complex estimation (e.g., "In that last year, how often have you seen or spoken to a medical doctor or assistant about your illne~?n). This research will investigate the mental processes used to make estimates and will explore differences in response accuracy achieved by using various estimation strategies. One set of studies will examine processes and knowledge structures used to estimate dates and durations. Other research will investigate estimates of quantities. ~~~~~~ ~ Additional work will explore processes used by respondents when revising initial estimates. Laboratory results will be tested in field settings. This research will guide improvements in survey design that can substantially increase the precision of results from surveys.

THE INTERSECTION OF PERSONAL AND NATIONAL HISTORY Howard Schuman and Philip Converse Each of us carries in our memory accounts of those events that make up the recent history of our society. Depending on our age, we have memories and reconstructed images of the Depression, the sixties, the Vietnam War, the Carter years, and so forth. Some of these memories are based on proximate contact, as in the case of someone who served as a combat soldier in Vietnam or was an active demonstrator against that war. Other memories are more distal, as in the case of those who watched both the war and the demonstrations on television. History at the individual level ts necessarily fragmented and idiosyncratic, but it is also vivid and personally important in a way that the historian ' ~ larger view of events can seldom be for readers. Therefore, it is likely to be significant as a person faces new events and problems that require interpretation and solution. This is widely recognized at important national decision points, for example, in the form of debates about whether, say, Lebanon in or is not similar to tier, and what that means. Vietnam becomes the model not only because of whatever similarities it has to the new problematic event, but also because Vietnam is resonant both to commentators and to much of their audience. We are proposing to investigate systematically the versions of recent history that are in the heads of the American population. Such an investigation is of interest in its own right and will also be valuable in helping us to understand how the American population responds to new events, those present and those soon to appear. From a practical point of. view, thin latter concern has to do with how personal experience of the past influences views or, and therefore actions toward, the present and the future. We do not assume that this process of drawing on the past is either automatic or simple, since many other factors--novel events, charismatic leaders, unanticipated external forces--play a part and may in some circumstances be more crucial. What we do assume is that models of the past, as personally experienced and learned, are one important factor in evaluating new experiences and are therefore worth studying systematically. We expect perceptions, interpretations, and inf erences from the past to be patterned to an important degree along social and cultural lines. Most fundamental of all should be age itself. Our investigation will attempt to capture the range from being in the midst of current news to learning history that is embodied in the dimension of age. Given the exploratory nature of the study, there will be an uncommon interest even in the straight distributions of. many of the responses, such ~s the relative salience of , diverse familiar historical events, the levels of personal and national importance ascribed to each, and the nature of the impacts felt to have resulted. Nonetheless, given the t~me-dependent focus of the study, it will be important to do much analytic work having partitioned the sample by age cohorts. Although age is expected to be the most basic explanatory variable in this study, there is good reason to expect other societal divisions to enter 38

39 significantly. For example, the civil rights changes of the 1950~ and early 1960s should be more salient to black than to white Americans, and to white southerners than to white non-southerners. Other societal divisions may play a part in the perception of other events: high unemployment for those who have themselves been unemployed, the worsens movement for committed feminists, etc. In sum, our study will need to obtain and use a number of standard and probably Rome not-so-standard ways of separating the total population along lines relevant to differential personal experience and involvement in the past. Some other individual difference measures, such as global versus more detailed schema construction, may also be attempted. What Population Wou' d be Studied and How We are proposing to translate theme general aims into a concrete research project by interviewing a probability sample of the United States adult population. We would like a sample size of 1,600, which would provide an adequate number of respondents at each decade of life (20s, 30s, 40s, 50s, 60s, 70 and above). If the initial investigation proves fruitful enough, it could well be extended to other age groups and cultures. The study would be carried out using experienced professional Survey Research Center telephone interviewers. This will considerably reduce the cost over face-to-face interviewing, without appreciably reducing coverage or response rates. Extra training in nondirective probing will be provided because of the open-ended nature of several of our central questions. Our design will allow us to obtain interviews in blocks of 400 at different points over a six- to twelve-month period, and will therefore provide data to separate the momentarily salient from the event or change that has more importance . ~ This same feature of the design permits a study of recency effects, as described below. ~ The Questions The questions developed for this investigation would be of several types. Since we are interested in obtaining the frame of reference that respondents use in organizing their memories of the past, it is important to allow maximum scope initially for spontaneous expression. At the same time, some more standardized questions are necessary in order to be able to make systematic comparisons in terms of certain mayor events and changes. Initial Frame of Reference Our first goal would be to obtain from each individual his or her own identification of the major events of the past 50 years. The following question, already pretested and piloted, would be asked at or very near the beginning of the interview.

40 Thirst, I'd like you to think back and tell me what you personally feel have been the one or two most important national or world events or changes in the past 50 years. ~ ~ If only one given: "Is there any other national or world event or change over the past 50 years that you personally feel was important? The question is stated in a general enough way so that it applies both to older respondents who have lived through the full period and to younger respondents who know parts of it only second- or third-hand. In addition, both event and change are stressed, since we are equally interested in discrete events and more diffuse changes. The answers to this question will provide initial evidence of the organizing scheme in the minds of respondents. We expect some events to be fairly frequent across most types of respondents, especially when age in held constant, while mention of other events should be characteristic of particular groups or categories of people. Significance of Events Central to our investigation is how people remember/reconstruct the meaning of events. Do they see them as successful or unsuccessful in outcome, symbolic of something good or bad, providing a positive or a negative lesson for the future? Questions about perceived importance will be asked for each of the events mentioned. Personal Connections One of our main interests is in whether and how the events or changes mentioned by respondents are tied to their own lives. Our pilot work indicates that some people note this spontaneously, but for those who do not, we will pose questions about how these events changed their lives or their way of thinking about things. Opinions of the Pant Although the concern of this research is primarily with memories, it is clear that we shall seldom be in a position to make direct comparisons between these memories and what they are supposed to be memories of. Indeed, this is not really the purpose . If. age produces relationships that are meaningful in terms of direct experience vat. secondary or tertiary learning, the reported memories can be attributed to pant experience, but of course this is inferential. However, this limitation, which would be a major one for psychologists concerned with memory per se, is not especially serious for the proposed research. In addition to the now generally accepted fact that all complex memories involve a considerable degree of

41 reconstruction, rather than being essentially photographs of. the past, our primary concern is with the memories as such, not with their accuracy. Nonetheless, for some events--especially con~croversial ones like the Vietnam War--it would be of interest to learn something about the drift in collective memory an the event has receded in time. Toward this end, we would like to resurrect one or more opinion poll items that were frequently used at the time to measure the division of sentiment, and pose it again to our "moderns respondents, first by way of asking what their current reactions to the event are; and second, by asking them to respond to it as they feel they would have at the time. Such a procedure would pick up any self-recognized drift of opinion, a phenomenon that would be of interest in its own right. And while we would not expect that respondents would be capable of recalling their actual early positions with any accuracy, we would be able to compare Unremembered opinions with distributions actually generated at the time of the event, which would provide some assessment of a more unconscious drift of opinion retrospectively. Not only could gross distributions be compared, but various background correlations of stable demographic attributes with original and remembered opinion as well. Such a tactic would only be available with respect to a very few events that were controversial at the time and well monitored by opinion polls with standard items, but it would seem important to seize upon such opportunities in this fashion. Secular Societal Trends Up to this point, we have dealt with major sevenths that have some staccato quality, even though certain of them cannot be very clearly demarcated in time. We are also interested in the felt impacts of some of the major social and technological trends that have characterized the past century. We would proceed in much the same way, asking first in unstructured form for the kinds of things that the respondents [eel have been most important in these regards, but then subsequently covering the most central possibilities in more structured or standardized form. The questions that we have in mind require more time perspective than our younger respondents could be expected to have. Thus we would ask persons 50 or older what they would see as the most important differences in daily life when they were growing up and life today, and why they make the selections they do. For trends such as these, we would like to know how the respondent [eels each has affected his or her life, and whether the change is generally beneficial or undesirable. This line of questioning would naturally conclude with a global question as to whether, all things considered, the quality of everyday life has improved or deteriorated in the respondent 's lifetime.

42 Background Questions In addition to standard background questions on age, sex, education, occupation, income, place of origin, marital status, and ethnicity, it would be useful to obtain some factual measures of the respondents and their close relatives' involvement in past events. Contributions of the Research To our knowledge no similar information on what might be called the collective memory has been previously gathered, and we believe the descriptive data yielded by the study will have some intrinsic interest for many social scientists. However, our interest is also analytic, and we will attempt to account for why historical memories vary over the population and how they influence judgments of the future . To give one partial example, our pilot study suggested that there in a gender difference in persona' connection to events and changes, with women more likely than men to explain the importance of an event or change in terms of some personal impact on their own lives (e.g., loss of a parent in a war). It is possible that this connection plays a role in the frequently reported tendency for women to be more reluctant than men to support new military actions. lathe research can also be viewed an a study of social memory that parallels laboratory studies of memory. Those studies document both primacy and recency effects in single-~ession tests of memory (Martindale, 1 981) . We expect also to find at least traces of primacy and recency effects in these long-term natural memories. Primacy effects should come about because of sheer rehearsal, much an in laboratory studies. Recency effects, on the other hand, cannot be due to short-term memory, as is assumed in laboratory studies, but to the temporary rehearsal of events that have recently occurred. In our pilot study, for example, mention was made of the Korean Airlines plane that had recently been shot down, but we doubt that that would occur even a month or so later. Since our design calls for gathering our interviews over at least six months, we will be able to go some distance in separating recency effects as such from the intrinsic importance of recent events. The hypothesis of primacy effects connects this research in a broader sense with the hypothesis that major events occurring early in life have the largest impact (Mannheim, ~ 927 )--other things equal. Without necessarily subscribing to any Mingle critical age, we will be able to examine the connection between recalled events and the ages at which they occurred. Do people in their 70s disproportionately cite the Depression, people in their 60s World War II, people in their 50s the Cold War or associated events, etc? The survey data will allow examination of this issue as well as of the effects of such experience on broader political and social Judgments about the interpretation of the events and their implications for future events (Lang and Lang, 1978~. Finally, it is true that a good deal of professional history attempts to address the longer-run impacts of both specific events and larger technological and social trends on the populations experiencing them. Some of there impacts are important whether salient, merely cognized, or

43 neither. At the same time, attention is frequently addressed to the perceived long-run significance of such events and trends on those who have lived through them, typically based on documents and testimony from individuals who are obviously unrepresentative survivors. We think it would be il ruminating to assemble the most salient recollections of a group more representative of the nation as a whole. References Lang, E., and Lang, G.E. 1978 Experience and ideology: the influence of the sixties on the intellectual elite. Pp. 197-230 in Serial ~mve-~. ~ and Change, Vol. 1. Greenwich, Conn.: JAI Press. FIannheim, K. 1927 The problems of generations. Reprinted in of edge, 1952. New York: Oxford University Press. Martindale, C . 1 981 5_. [?omewood, Ill.: the Dorsey Press.

A PROPOSAL FOR THE DEVELOPMENT OF A NATIONAL M13HORY INVENTORY Endel Tuiving and S. Ages Press This is a proposal for the development of a national memory inventory. Memory capabilities of the population, together with other cognitive abilities, represent an important part of the nation's intellectual resources. No general and systematic information regarding the quality of these resources is available at the present time. For this reason alone, it would be desirable to have an estimate of memory, an well as other cognitive abilities, on a nationwide basis. The inventory would not only provide an objective picture of the current state of these abilities, but also make it possible to monitor changes in memory and cognitive abilities over time, and to relate such changes to the changing age composition of the nation. Age-related memory and cognitive functions would be of particular interest in this context. The inventory would provide a set of national norms against which memory performance of individuals or groups of individuals and any impairments in such performance can be evaluated. This kind of evaluation is becoming increasingly critical in a society in which a sizeable proportion of. older people suffer from various forms of senUc dementia, such as Alzheimer's disease. Impairment of memory and cognitive functions is among the earliest symptoms of these dimensions; the detection of such impairment is, therefore, of Considerable importance. The existence of national norm of memory, as well as other cognitive abilities, with which the test scores of individuals can be compared, may significantI, facilitate clinical assesee~ent of impairment of such ability. A national inventory of memory could be used for classif ication of subjects with memory impairments. It could also be used to scale the quality of memory functions of individuals in situations in which such [unctions play an important role. For instance, it might be possible to Recalibrates eye-witnesses in court trials on the basis of a battery of suitable tents and to develop "weighted for responses given by individual respondent to rec~l-type questions on sample surveys of different kinds. This proposal is one of the products of the CASH seminar at St. Michaels and its follow-up meeting in Baltimore. The two authors of the proposal do not wish to claim any proprietary rights to, or special interest in, the further development of the ideas contained herein, or its eventual implementation, although they are willing to collaborate with other interested individuals and agencies in any further development. The mayor purpose of the proposal is to stimulate and encourage further thought and possible action along the general lines discussed herein. The proposer consists of two main parts. The first part contains a short background statement about memory and nte~tinge of memory; a short general description of the proposed battery, together with a listing of criterieria used in selecting individual components of the battey; and a S',mmary of the procedures it be used in the collection and analysis of 44

45 the data. The second main part of the proposal consists of a description of memory tanks consituting the battery, together with instructions and examples of the kinds of materials that might be used. Memory and Memory ~Tests" Psychological study of memory can be approached from two different vantage points. One is that of cognitive psychology. In this view, memory is a set of interrelated cognitive processes that allow a person to acquire, store, and subseqently retrieve information about the world. These processes are described with reference to a typical individual, the Standard rememberer. n The other vantage point in that of psychometrics. According to this view, memory is an ability or skill that individuals possess and with respect to which different individuals vary. Memory is conceptualized as a unitary entity in neither the cognitive psychology nor the psychometric approach. Rather, both assume that the concept of memory covers a number of different forms or kinds of acquiring and using knowledge and information. These different kinds of memory operate according to somewhat different principles and, at the level of psychometric analyses, show different correlational patterns of individual differences. Thus, students of memory have talked about visual versus auditory memory, verbal memory versus pictorial memory, rote memory versus meaningful memory, voluntary versus involuntary memory, as well as about facial memory, spatial memory, recall memory, recognition memory, and many other sorts of specialized memories. Two mayor distinctions concerning different kinds of memory that are useful to make in the present context are those (~) between episodic and semantic memory (Tulving, 1972, 1983) and (2) between primary and secondary memory (Waugh and Norman, 1965; Craik and Levy, 1976~. E~l~A4r me rev refers to memory for concrete, personally experienced events; 2~ ~~Y~ ACE refers to a person's abstract knowledge of the world. For example, it a person sees and later recalls a familiar word or a drawing of a common object in a memory test or memory experiment, that person 'a episodic memory is being tented. If, on the other hand, the person is shown the picture of a public figure and asked to name the figure, it is semantic memory that is being assessed. primary w~morv (sometimes also labelled short-term memory) refers to memory for perceived stimuli within a few seconds of their presentation, before the representation of the stimuli hen completely left the individual 'a consciousness; ·eo~oda~v ·~ror~ (sometimes also labelled long-term memory) refers to memory for information that has left the person's consciousness and has to be brought back into it through particular retrieval queries or cues. Given the complexity of processes and abilities that the term memory covers, it is generally accepted that there is no simple way of measuring people's memory. Certainly there does not exist a single convenient memory test that could be used to assess the memory abilities of a group of individuals. Instead, a battery of tents is necessary to capture different forms and kinds of memory.

46 The term memory test is somewhat ambiguous. Its meaning can be clarified by drawing the distinction between a memory task and a memory test. Although the term test is frequently used in the psychometric tradition, referring to the whole operation that permits the attachment of a numerical value to a person's performance on a memory task, it actually, or more precisely, refers to just one component of such an operation. A memory test for the material that the person has learned in a particular situation (an episodic memory test ~ constitutes only the final stage of a memory task that consists of the following sequence of events: (1) a person examines (observes, studies) some material; (2) there is an interpolated interval of variable duration, usually filled with mental activity involving material other than that studied in the first part of the task; and (3) the person is given a test of what he or she remembers from the initial, study phase of the task. (Note that in the realm of semantic memory, there is usually no need to distinguish between the memory task and the memory test: a semantic memory tack can consist of nothing else but a test.) No generally accepted standard battery of memory tasks exists. When a person's memory has to be assessed for clinical purposes--as in cases of known or suspected brain damage--various instruments have been used. The most popular of these is the Wechaler Memory Scale. It consists of seven sublets, some of which are concerned with questions as to the respondent's awareness of and orientation in space and time (~How old are youth nWhat day of the month is thieve), and only come of which tap the respondent's memory for newly presented information. But all of these true memory tests measure the respondent's short-term memory only. This characteristic seriously limits the usefulness of the scale. The Wechaler scale was standardized in the 1930s and 1940s (Wecheler, 1945) on approximately 200 haphazardly selected adults between the age n of 25 and 50. A personas overall score on the battery can be evaluated against the distribution of scores from the standardization group, and on the basis of this evaluation it can be expressed as his or her MQ (memory quotient), whose meaning or interpretation is roughly comparable to that of IQ. The clinical condition known as amnesia can be operationally defined in terms of an abnormally large difference between a per~on's IQ and MQ. Testing of larger groups of subjects on various memory tasks has been undertaken only in factor-analytic studies of memory (e.g., Relley, 1964; Underwood, Boruch, and Malmi, 1978~. In these studies, several hundred subjects are typically given a large number of memory tasks, and the scores from the tests are used to derive the [actor structure of the tests employed. These studies, too , suffer from the limitation of employing almost exclusively short-term memory tasks in which the subjects are tented immediately after the presentation of the to-be-remembered material. A Short Description of the Battery It is probably impossible to construct a completely adequate battery of memory tasks at the present time. Because of lack of appropriate

47 empirical evidence, there is no general agreement as to how many different kinds of tasks would be necessary to assess most of the important aspects of people's memory performance. Different materials and different conditions under which the materials are studied, retained, and tested are known to influence the performance of a given individual relative to that of others. Thus, there exists a potentially very large set of memory tasks, that is, combinations of materials and conditions of their study and test. The exact consititution of the battery that eventually would be used represents one of the many sub-problems that would have to be solved in the course of the project. The scope and organization of such a battery would necessarily have to reflect a compromise between what is scientifically desirable and what is practically feasible. To make the battery suitable for use in a large, heterogeneous population, the materials for each tack would need to be carefully screened and tested to minimize biases favoring the performance of one cultural, ethnic, or socioeconomic group over another. The battery described in what follows consistitutes only one of many possibilities. The sample battery consists of two mayor parts, A and B. Part A consists of the study stages, and in some cases immediate (short-term) testing, of six tasks, together with a test for the memory of the order of the six tasks. Part B consists of the delayed (long-term) teats of five of the six tasks. The sequence of events constituting the battery is summarized next. A more complete description of the tasks--materials, instructions, and test forms--will be round in the second mayor section of the proposal. Part A ~ ~ ~ ~--32 words (such as HYDRANT and BLUEBIRD) presented for study. Delayed recognition and word-fragment completion tents are given in Part B. ~ 2 ~ _--24 line drawings of coupon ob Sects (such as a basket, a glove, and a lion) are presented for naming by the respondent. Delayed recognition test is given in Part B. (3) Ad ~ ·octatas--12 pairs of words (such as CLAMP-VALET and RURAL-HEAVE) are presented for study and immediate paired-associate test. This ~tudy-test procedure is repeated on the second trial, with the came 12 pairs. Delayed paired-associate test is also given in Part B. ( 4 ) Faces-- 16 faces of unknown people presented for study. Delayed recognition test is given in Part B. ~ 5 ~ ~--3 familiar words (such as CARP, MINNON, BARRACUDA, or SPRUCE, POPULAR, WILLOW) from each of 6 different conceptual categories ~ a total of ~ ~ words) are presented to subjects initially for identification of category membership (fish or treed) and subsequently for an immediate free-recall test of the 18 words. Delayed cued-recall test is also given in Part B.

48 (6) 5b~t _~ ~ ~~Y~C~ ~~Ct~--15 three-letter words (such as GUN, ART, ILL, BAY) presented for study and immediate free recall. This study-teQL procedure in repeated on the second trial with the same 15 words. (7) Order of ta~ks--given a descriptive listing of the six tasks of Part A, respondents are asked to reproduce the order in which they encountered the six tasks. Part B (1) r~..~--d .-ris--cued recall test. Respondents are given the names of the six categories of words they saw in Part A(5), and they try to recall the three instances presented in each category. The maximum score is 18. (2) Faces--two-alternative forced-choice recognition test. Respondents are shown ~ 6 pa' rs of faces, one pair at a time . Each pair contains one of the faces seen in Part A ~ 4 ~ and a new face. The respondent has to choose one of the faces in each pair as the one he or she saw earlier. The maximum score is 16. ~ 3 ~ ~ :_~--cued recall test . Respondents are given the left-hand members of each of, the 12 pairs of words seen in Part A(3), one word at a time, and their task is deco produce the name of the corresponding right-hand member of the pair. The maximum score is 12. (4) ~ --tour-alternative forced-choice recognition test. Respondents are shown 24 nets of four different line drawings of common objects, one set at a time. Each set depicts an ob ject ~ such as a basket, or a glove, or a lion) in four different ways. One of these was seen by the respondent in Part A(2), the other three are new. The respondent 's tack in to select the one he or she saw before. The maximum score is 24. ~ 5 ~ ~ v ~--yes/no recognition test . Respondents are shown 32 words (such as HYDRANT and COPYCAT), one word at a time. Half of these test words appeared in Part At ~ ), half are new. The respondent 'a task is to identity each word as old or new. The maximum score for the old tent words is ~ 6, for the new tent words, 1 6 . (6) I ~v --word-fragment completion test. Respondents are given 32 word fragments (such as __ U B _ ED and _ O _ O _ UT). Half of these fragments correspond to words they saw in Part A (such as BLUEBIRD), while the other half belong to words not previously seen in the session. The respondent's task is to complete the fragment by replacing dashes with letters and thus converting the fragment into a word. Note that respondents are not asked to produce words that they saw in Part A, their task is to produce the word that fits the fragment . Fragments are so constructed that they fit only one word in English. The maximum score for each of the two subsets of test words is ~ 6. Criteria for the Selection of Tasks The criteria governing the selection of tasks for the instrument (the battery) proposed here include the following:

49 (1) Emphasis on long-term episodic memory. The tasks in the battery are primarily concerned with Memory proper," that is, long-term episodic memory. (2) Multiple materials. The tasks in the battery tap memory for both verbal and nonverbal material. ~ 3 ~ Multiple test types . The tasks in the battery assess both recall memory and recognition memory. (4) Sensitivity to age differences. The tanks in the battery include those that are expected to be sensitive to age differences in the population as well an those that are not, or are lens sensitive (Craik, 1977) (5) Length of the testing period. It should be possible to administer the whole battery in a single session of approximately one hour's duration. (6) No special equipment. The battery could be administered under less-than~perfect laboratory conditions without any special equipment. (7) Group testing. The administration of the battery could be modi flied to make it possible to test small groups of respondents simultaneously, if such a procedure has certain practical advantages. (~) Alternative response modes. The battery consists of tasks in which either oral or written responses could be given by the respondents without greatly biasing the results. (9) Range of scores. The tanks in the battery can be fine-tuned in pretesting to minimize Ceiling effects" in performance while permitting a very large majority of respondents to perform in a way that would justify the examiner to provide occasional positive encouragement to the respondent. ( 1 O ) Alternative forms. It is possible to construct alternative forms of the battery, entailing different versions of the same tasks, that would yield comparable normative data from the population. (~) Clinical use. In addition to assessment of memory abilities of samples of the general population, the battery can be used for clinical evaluation of individuals with milder forms of memory impairment. General Characteristics of the Battery The battery is designed to measure both short-term and long-term memory. In one of the tasks (A6), only short-term tents are given, albeit on two separate learning trials. In four of the tasks (B2, B4, B5, and 86), only long-term tests are given. In four other tanks (A3, A5, 81, and B3), both ~hort-term and long-term tests are given. Delayed ~ long-term) tents are given for tasks in which the respondents are unlikely to be confused as to exactly what it is that they have to try to remember in any given tent. For thin reason, no delayed free-recall tests are included in the battery. The test for the order of the six tasks of Part A (A7) is included as an attempt to assess ~pure" episodic memory: memory for the temporal sequence of otherwise easily remembered personal events. Other tests tap only the respondent 's knowledge of the semantic contents of these events . It in the only test that is likely to produce ceiling effects,

50 but the data from it may be revealing in cases where subjects do make errors on the test. The test takes only a little time to administer. Three tasks in the instrument probe people's recognition memory for verbal materials (B5), appearances of objects (B4), and {aces (Bag. For both of the two latter kinds of materials (B4 and B2), verbal mediation in remembering is precluded: remembering Just the name of the originally perceived object will not permit the subject to choose the correct alternative in the tent, since all four tent items in a set have the same name, and faces are difficult to code verbally to begin with. Yet two different recognition tasks tapping purely visual memory (line drawings and faces ~ are included because it is quite possible that the ability to remember faces is not highly correJ ated with the ability to remember the appearances of other visually perceived objects. All recognition tests are given after longer retention intervals: immediate tests of relatively small sets of to-be-remembered materials would be subject to ceiling effects. Recall tests in the battery are of two kinds, free recall (A5 and A6) and cued recall (A3, B1, and Bay. In free recall, the respondents' task in to produce as many studied items an possible, in any order, in response to general instructions to do so. In cued recall, subjects are provided with specific cues for recall of individual items. Two kinds of relations between cues and items to be recalled (or, between cues and targeted are studied in cued recall teats. In the paired-associate task, the cue consists of a word semantically unrelated to the target that was paired with the to-be-recalled word at -study; in the categorized words task, each cue, associated with three target words, is represented by the semantically meaningful category name. The fragment completion test (B6 ~ is known to show the effects of memory in a fashion uncorrelated with other measures of memory (Tulving, Schacter, and Stark, ~ 982 ~ and is therefore of especial interest . Respondents' performance with the ~new" fragments of this test provides a measure of one aspect of their semantic memory. Procedures of Data Collection and Analysis The idea in to administer such a battery of memory tasks to a large probability sample of the U.S. population. Before a large national sampling of the country is carried out, it would of course be necessary to do pilot testing with small samples consisting of several hundred subjects. In addition, some preliminary studies would be necessary to establish and tent the feasibility of the statistical procedures for analysis of the data collected. A number of procedural problems must be solved in the course of the pursuit of. the ultimate objective of this proposal. Some Or these involve improving the currently available fundamental understanding of the determinants of people's performance on memory tasks, and, therefore, involve cognitive psychology as well an other aspects, of psychology. Others involve computer science, psychometrics, sociology, and statistics. The procedures to be developed would involve at least the steps indicated in the following paragraphs.

51 Identification of Indicators A Ret of p-variables that can be measured by individual testing are derived from the tasks that consitute the battery. These variables are indicators of the dimensions that characterize the diverse aspects of human memory. The measured performance of a subject on various tasks is referred to as the raw scores. Some psychometric unfolding" techniques could be used here to determine an appropriate space of q flower than p) dimensions. (For some earlier work in this area, see Underwood et al., 1978.) Techniques such as factor analysis, principal components analysis, multidimensional scaling, etc., could be used based upon p- te~t~ that probably are not orthogonal in terms of memory characterization. The results of such unfolding analysis is a q-vector of "reduced scores" (factor scorers for any tented individual, x:(qxl). That is, a low dimensional space of q dimensions will be determined that in sufficient to characterize the determinants of memory performance of interest. Each subject will then be scored in the q-space. Data Collection A pilot survey should be carried out to collect p-vectors of. raw scores for N people. (The raw scores can later be converted into reduced scores.) The N people should be selected by stratified random sampling, stratifiction to take place on variables known to be correlates of memory variations ~ for example, age ~ . Such stratification variables need to be identified; a sampling frame needs to be constructed; an appropriate sampling procedure must be established; and a final survey instrument needs to be constructed. Thi ~ should all be done on a small scale before anything major in attempted. The survey instrument would include a questionnaire designed to provide background information on each subject appropriate to relating reduced memory scores (x) to background variables (memory correlates). Population Determination A important part of the procedure entails establishing what might be called a typical or normal memory. There will be a multidimensional distribution or such typical memories. Subsequently, a clustering analysis should be undertaken of the N reduced-score vectors, each defined in q-space, in order to try to separate the N points into two populations, typical and atypical, or perhaps several well-defined groups, instead of lust two. Scores in the typical group (it there were only two) could be used to establish the distribution of a Standard population, as the forerunner of national norms. The atypical group (or groups) would have its own distribution and norms .

52 Distributional Analysis This aspect of the procedure involved an anal ysis to determine the empirical form of the distributions of the various clusters. An attempt should be made to establish theoretical population functional forms and fit parameters. For example, suppose, just for simplicity of explication, it turned out that for each population cluster (i ~ of interesS, the vectors of reduced scores for subjects, x:qxl, were all normally distributed N(thetai,~igma~ ). That in, the vectors of reduced scores for people with typical memories each followed a multivariate normal distribution with mean vector thetas and covariance matrix sigma). People with memories that are in some atypical cluster have reduced-score vectors that also follow a multivariate normal distribution, but with some other mean vector and covariance matrix. If there were two such atypical clusters the population parameters might be (theta2·sigma2) and (theta3,sigma3), respectively, for each cluster. Once these distributions were established and the population parameters estimated, any individual person's total memory performance could readily be classified by conventional stati stical classification techniques into one of the available memory clusters. It will be necessary to define a sample of individuals who will be used to establish the basic populations that will then be used for future classification purposes. The Memory Battery: Instructions, Materials, and Tests A somewhat more complete description of a possible battery of memory tasks is given next . For each task, instructions to participants, examples of materials to be used, and the nature of the test ~ s) are provided. Part A (~) Low-Freguenev Words Instructions: "For your first task of the day, I am going to show you a number of words that you will be asked to remember later on. Since you are going to see many other words later on today, and since we want to be able to talk about the lot ~ am going to chow you now, we will call there words 7- and 8-letter words, because they all contain either 7 or ~ letters. Please pay close attention to each word as it appears as you will have only a few seconds to study it. Are you ready? Here we gOIn Presentation: 32 words presented at the rate of 3 seconds/word Sample materials: HYDRANT BLUEBIRD NICKNAME: DAFFODIL BATHROBE OCTOPUS ALMANAC MACKEREL SMALLPOX MOONBEAM CLARINET PACIFIST BAGPIPE PENDULUM COCONUT MOLECULE

53 Transition: ethic t~ the end of those 7- and 8-letter words. As I said, we will come back to ached later. We now go on to the next task." (2) ~ Instructions: ethic time ~ am going to chow you a number of pictures of common objects, one picture at a time. When you see a picture, you should name the object Chat the picture depicts. Later on, I will ark you to remember these pictures. But when you first see each picture, ,just call its name out aloud, using a short, general description . (Demonstrate: tree ~ For instance, you would call this picture a tree. Here is another picture. (Demonstrate: Jug) What would you call this one? That's correct. Jug, or water Jug, in `Just right. You've got the idea. (Or correct the sub ject. ~ So, let us begin with the real pictures. Just call out their names now. Presentation: Present 24 pictures at the rate of ~ seconds/picture. Materials: 24 line drawings of common objects (sample objects shown below): BASKET GLOVE LION LEAF MOUSE BELT COATHANGER FROG TIE BED BATH CHAIR CANDLE SNOWMAN BOTTLE BUTTERFLY BELL ZEBRA HAMMER AIRPLANE RIFLE SAILBOAT FLAG KEY in, Transition: nThat's all for the pictures. Again, we will return to hem later. ~ ~ 3 ~ ~ Instructions: nYour next task involves emory for pairs of words. Each pair contains two {ive-letter words that ill be shown together for a few seconds. After you have seen all the sirs, I will test your memory for them by showing you the first word of Itch pair and asking you to recall the second word that was paired with ;. OR? Here we gold

54 Pre-~entation--Trial 1: Present 12 pairs of words at the rate of 3 seconds/pair. Materials: CLAMP-VALET RURAL-HEAVE CONIC-A80UT SPICE-DUMPY ULCER-CHIME STORE-HITCH RAPID-BLUNT STALR-PORCH ALIVE-GLORY QUASH-FIBER STAGE-SHADE HEADY-FINAL Immediate test 1: "Here comes the test. ~ will show you the first word of each pair and you will try to recall the second. Do not worry it you do not get too many of them right, it is a difficult tent. Here we gold Presentation and immediate test--Trial 2: Repeat procedure of Trial 1: Present 12 pairs, and tent them as on Trial 1. Transition: You are doing all right. We change the pace again, and for the next task ~ will be showing you some photographs of people's faces. ~ 4 ~ Fat es Instructions: "This task, as I said ~ nvo' ves memory for photographs of people's faces. I will show you a number of faces and later on ask you to recognize them. Pay close attention to each [ace [or you'll see it only once, very briefly. Are you ready? Here we gold Presentation: Show ~ 6 photographs at the rate of 2 seconds/photograph. Materials: A selection of black-and-white photographs of. faces of people. The photographs are similar to those that might be used in a yearbook for a large school: small portraits with little other than facial features to distinguish one person from another ~ no examples shown ~ . Transition: "I will tent you for these faces later on today. We go on now to the next task. ~ ~ 5 ~ ~ Instructions: For thin tank you will see groups of words, 3 words at a time . The three words in each group belong to a particular category. For instance, if the words were 'London; San Francisco; Tokyo,' the category would be 'cities.' Your task is to identify and tell me the category to which the words in each group of three belong. Just give me a brief. label of each category an I show the words to you. If you cannot think of a suitable common name for the three words, Just say so, and ~ will give it to you Yell. After you have seen a number of these categorized words, I will ask you to recall them. So, pay close attention to all three words when you study each group. So, look carefully at all words in each group, name the category, and later on recall the words. Is this clear? If so, let us proceed.

55 Presentation: Present 6 categories of 3 words at the rate of 12 neconds/category. Record for each category whether the subject named it or whether you had to provide it. Materials: WASP LAWYER CANARY MOTH ACCOUNTANT HAWK COCKROACH FARMER ORIOLE ~ Insects ~ (Jobs ~ (Birds ASPARAGUS ORCHID BADMINTON CELERY AZALEA WRESTLING TURNIP ZINNIA VOLLEYBALL ~ Vegetables ~ ~ Flowers ~ ~ Sports ~ Immediate test: Hall right. Let us see now how many of these words that I showed you, you can remember. Tell me as many words now as you remember. Tell me only the words that I showed you, and NOT the category names that you yourself provided. Go ahead. Give the subject 60 seconds for recall. Transition: "That ~ ~ fine. That 's all for the recall of these categorized words. Let 's go on to the next task. ~ (6) Tort Words Instructions: "For the next taslc, I will show you a number of very short words. Each word consists only of three letters. Look at each word carefully and try to remember it. After you have seen the lot, I will ask you to recall them. You do not have to remember the order in which the words appear; when you recall them, you can recall them in any order that they occur to you. ~ will ask you to begin as soon as you have seen the last word in the lot, so be ready . Any questions? If not, get ready for the first word. ~ Presentation: Present 1 5 words at the rate of 2 seconds/word. Materials: GUN ART ILL BAY LID BIT OWN CUT ROB DIM SET EAR TRY FEE WIN remember. " Immediate te-~t--Trial 1: "Go ahead, tell me all the words you Give the subject 60 seconds for recall. Instructions and test--Trial 2: "We will try this list once more. I will show you the same set of short words again, and again, when you 've seen the lest one, you try to recall as many of them as you can, in any order in which they occur to you. When you recall the words the second time, recall everything that you can from the whole list, including those words that you already got the first time around. OK? Here we go I ~

56 Present and test the list in the same way as in Trial Transition: ~That's fine. You are doing all right. Part B 1. (~) Order of Tasks Instructions: "Now we are done with studying and looking at different materials. In the second part of the cession, I will ask you to remember the materials that you saw In the first part. The first thing I would like you to do is to tell me the orate" in which you saw different kinds of materials earlier today. There were altogether SIX things you did. They are briefly described on there six cards. Look at these descriptions and then tell me which of these came first, which one second, and no on to the one that you did last. Take a moment or two to refresh your memory for the tasks, and then order them in the way in which they were presented to you earlier. Tent: Six cards presented to the participant with the following descriptions: a. Looking at photographs of faces b. Looking at line drawings of common objects c. Recalling short 3-letter words d. Studying 7- and 8-letter words e. Categorizing and recalling groups of 3 words [. Studying and recalling pairs of five-letter words Participant is given up to 2 minutes to order the cards; the order is recorded. (2) C~i -so Wrrd- Instructions: "I am now going to test your memory for the words belonging to categories that you studied and recalled earlier. To help you recall the words, I am going to name all the categories that you saw. These category names are printed on this sheet. You remember that there were three words in each category that you Raw. (Give subject the cardboard with the category names.) Go ahead now and tell me the words from there categories that ~ showed you earlier. Keep thinking and recalling words until I ask you to stop. Do not guess wildly. If you are ready, go ahead to Delayed cued recall test of categorized words: The subject is given a cardboard with the following names of categories typed on it: a. Insects b. Jobs or professions c. Birds d. Vegetables e. Flowers f. Sports

57 The participant is allowed 2 minutes to recall the previously presented words. ~ 3 ~ Faces Instructions: "The next thing I have [or you to do in a tent of recognition memory involving the faces you saw earlier today. I will show you two faces at a time. One of the faces in each pair is one that you saw earlier and the other one in new. Your task is to say which of the two faces you saw earlier. This time you should guess, if necessary. Simply say 'left' or 'right ' to indicate which of the two you think you saw earlier. In addition to choosing one of the two faces in each pair as the one you think you saw earlier, you should also tell me each time whether you actually remember seeing the [ace or whether you are only guessing. You saw ~ 6 [aces earlier, thus you will see 1 6 test pairs . Any questions? If not, let ' ~ begin. ~ Two-alternati~re forced-choice recognition tent for faces: Present to the participant ~ 6 test pairs of [aces ~ at the rate of ~ seconds/pair . Record the participants ' choice, and whether the participant reports remembering or guessing. In each test pair, only one of the faces included will have been seen earlier in the set of [aces presented for study in Task A ~ 4 ~ . (4) ~;~ Instructions: ~Next, I am going to best your memory once more for those pairs of five-letter words that you saw and recalled earlier today . This test i ~ exactly like the one that you already took earlier. As before, I will show you the first word of a pair and your task is to try to recall the second member. Don' t guess wildly. We will go through the ~ 2 pairs one at a time. Ready?" Delayed paired-associates test: The subject is given the 12 left-hand members of the pairs, as in the immediate tests, one at a time, and allowed up to 10 seconds to produce the right-hand member of the pair. (5) _ Instructions: amour next task is to recognize pictures of objects that you saw earlier today. Each object will be tented by showing you four pictures of the object. Try your best to pick out the exact came picture that you new earlier. You will note that the four tent pictures are labelled A, B. C, and D. Look carefully at all four of them and then tell me the letter of the one that you know or think that you saw earlier. Again, you should choose one picture out of each set of four, guessing it necessary, and you should tell me each time whether you remember the picture you chose or whether you are guessing. Ready? Here we gold Four-alternative forced-choice recognition test of pictures of objects: Present to the subject 24 sets of test drawings and allow the subject up to 10 seconds/set to choose one of the alternatives. Record

1 58 subject up to 10 seconds/~et to choose one of the alternatives. Record the choice and whether the subject reports remembering or guessing. eon ~ ~ ~ 1~'t"~ ,r! ~ . _ ,_ , ._ In, r of,, f41 i,''.; mu.

59 (6) ~ Instructions: Remember those 7- and 8-letter words that you saw way back at the start of today's activities? We are now ready to test your memory for them. It is a recognition memory test. I will show you one word at a time and you tell me whether you remember seeing the word earlier today or not. For each word that I show you, you make the decision and say 'yes' it you remember it and 'no' if you do not. Again, as before, you should also tell me whether you are guessing or whether you are reasonably sure of your decision. So say either 'yes' or 'no' to each tent word, and also whether you are reasonably certain of your decision or whether you are guessing. Any questions? Here we gold Yes/no recognition test of low-frequency words: Show the subject 16 test words, one at a time, allowing up to 5 seconds per word for the subject to make the yes/no decision and three seconds to report the confidence `Judgment. Record the decision and whether the subject says he or she is remembering or guessing. Sample test words: DUCKLING HYDRANT NICKNAME BLIZZARD MOSQUITO BLADDER LETTUCE MACKEREL SMALLPOX OMELETTE MEMBRANE DAFFODIL PACIFIST PENDULUM MOLECULE APRICOT Transition: "You are doing fine. We are almost finished. (7) Low-Frequency Worda Fragment Completion Instructions: "This is the last task ~ am asking you to do today. It involves completing of words from which some letters have been deleted. I will show you a number of such incomplete words and you will try to guess what the word is by mentally filling the missing letters. (Demonstrate) For instance, look at this card: (CH_FM_NR). What is the word? Good. (Or: It's CHIPMUNK; do you see it?) Try another one. (_7MOC_AT). What's this one? Right. DEMOCRAT. Got the idea? All right , let 's start then. Do the best you can, and do not worry if you do not get too many of them. n Word-fragment completion test of low-frequency words: Present to the subject 16 word fragments, one at a time, and allow a maximum of 15 seconds/word for completion of the fragment. Sample fragments are presented below: _ U B RD P _ FF N MO _ B _ M AN _ MY _ L _ R _ _ ET B _ GP _ E FL EL AS8 _ O - O T US RN P R _ _ TH _ OB _ C TL _ R _ _ _ _ _ _ _ PA _ _ B _ W BA _ E _ OR AL _ N _ C _ O _ O _ UT Final word: nThat'~ all. Thank you very much. n

60 References Cralk, P. I. M. 1977 Age differences in human memory. In J.E. Birren and R.W. Schaie, eds., i. New York: Van Nostrand Reinhold. Craik, F.I.M, and Levy, B.A. 1976 The concept of primary memory. In W.~. Estes, ea., Hand~ok ~ , Vol 4. Hillsdale, N.~.: Lawrence Erlbaum Associates. Kelley, H. P. 1964 Memory abilities: a factor analysis. PA o ~aDhs ~ 1. Tulving, E. 1972 Epinodic and semantic memory. In E. Tul~ring and W. Donaldson. eds.. A. New York: Academic 1 983 Press. New York: Oxford Oni~rersity Press. Tulving, E., Schacter, D. L., and Stark, H. A. 1982 Priming effects in word-fragment completion are independent of recognition memory. ~ ~ 8: 336-342. Underwood, B.~., Boruch, R.F., and Mali, R.A. 1978 Composition of episodic memory. ~;_~ ~ 107: 393-419. . Waugh, N. C. ~ and Norman, D. A. 1965 Primary memory. ~ Rim 72:89-104. Wechaler, D. 1945 A standardized memory -scale for clinical use. Psychos on ~ 9 : 87-95.

PROTOCOL JlNALYSIS OF RESPONSES TO SURVEY Rl3CILL QUESTIONS Elizabeth Loflus One idea that was received enthusiastically at the St. Michaels seminar wan the suggestion for using protocol analysis to study how survey respondents retrieve information from memory to respond to questions about pant events. The first section of this item, prepared a few weeks after the seminar, expands on this idea and presents some results from five pilot interviews conducted to explore the potential utility of this technique. Subsequently, David Fathi, a student of the author, has conducted research in this Dame area for an honors thesis. The second section of this item describes some results from 23 protocol analyses of responses to two questions; one on use of. health care facilities and one on deposits to a credit account. One of the questions arising in this research was how to relate different methods of retrieval to the validity of responses. Since it proved difficult to obtain verification data for the health and credit deposit questions, some subsequent work by Fathi has related to questions asking students in an undergraduate psychology course to recall the exact dates of examinations given in the course. Experimental ProJect: Protocol Analysis In many national surveys, respondents are asked to recall personal events Prom their lives. For example, in the National Health Interview Survey, respondents are asked, "During the past 12 months, about how many times did (you) see or talk to a medical doctored In the National Crime Surrey, respondents are asked, non the last six months, did anyone beat you up, attack you, or hit you with something, such as a rock or bottle?. Very little is known about the precise strategies for retrieving personal information of thin sort. One method for learning about cognitive strategies is through the use of protocols (Ericsson and Simon, 1980~. In the protocol technique, people are asked to think aloud as they answer specific questions. The verbalizations produced are called protocols, and they can subsequently be transcribed and analyzed. This method has an advantage over the similar technique of asking people after the fact to describe how they arrived at a particular answer or estimate. The ~atter-the-fact. technique has the disadvantage that people often provide reasons or rationalizations for their bobavior that are not the true reasons but rather are strategies that subjects believe should have been appropriate (Nisbett and Ross, 1980~. To explore the feasibility of a protocol analysis approach to the problem of how people retrieve personal experiences of the type required on, say, the National Health Interview Survey, we asked five pilot subjects to think aloud while answering specific questions. We first gave subjects some practice questions so they could gain experience in Verbalizing their thought processes. Those were questions such an, .~n 61

62 the last 12 months, have you eaten lobster?. Then we asked some health-related questions and Rome crime victimization questions. For example, we asked subjects, .In the last 12 months, how many times have you gone to a doctor, or a dentist, or a hospital, or utilized any health care specialist or facility?. and, sin the last 12 months, have you been the victim of a crime? Many specific questions can be answered by examining the protocols produced by subjects. For example, one specific question Ad this: Do people answer the health question by starting from the beginning of the 12-month period and moving toward the present (the past to present approach), or do they start from the most recent event and move backward (the present to past approach)? One might predict that respondents would begin with the most recent events, since these might be more "available in memory (Tversk~r and Rahneman, 1973~. While our results must be considered preliminary, they indicate that, to the contrary, the past to present approach is the favored one. For example, one female respondent answered the health question by saying, Let's see . . . six . . . six months ago ~ went to the dentist. Last month I went to the doctor. I think that' s it . ~ It this tendency to pref er the past to present retrieval sequence for the health question were to be documented in a full study, it would suggest that people might be most efficient at retrieving information if prompted to do so by cues that allowed them to start in the past and work toward the present. Of course, this hypothesis would need to be explicitly tested mince we know that simply because most people perform acts in a particular way does not necessarily mean this is the most efficient way to do so. Another specific issue that could be addressed by analysis of the protocols is the extent to which respondents produce new information when asked further questions that relate to ones that were asked earlier. For example, the response of the female quoted above indicated two health related contacts. However, later this respondent was asked, nIn the last 2 months have you been to a dentistry Her answer: Let's see . . . I had my teeth cleaned six months ago, and no . . . and then I had them checked three months ago, and I had a tooth . . . yeah, I had a toothache about Harch . . . yeah. So, yeah, I have. n (Interview conducted in July 1983.) This protocol again indicates a preference for the past-to-present retrieval sequence, but also indicates the production of two additional dentist visits that were not provided earlier to the more general question. Although it is well suspected that additional questions will produce additional instances, it is not known why. Protocols could shed light on this issue. Furthermore, it is not known whether beginning an interview with a general question (e.g., shave you been to a specialist? is the optimal technique. It is possible that after having said, anon to the general questions, subjects may be less likely to search memory in an eSfort to answer the specific question than they might have had they not been asked the general question to begin with. He simply do not know whether this is the case. However, an examination of protocols given to specie to quest ions that either are or are not preceded by general ones would be more informative.

63 One interesting observation from the five pilot protocols is the large number of instances in which people change their answer as they are in the midst of speaking. For example, one female subject who was asked the crone question answered: "No, not that I can think of, unless . . . Oh, I had two dollars stolen at work, but that's it. n Another said: "No, I haven't, that I can remember . . . Yes, ~ was--T was thinking about my car, and I had come tapes stolen from my car, in Montlake, about six months ago." We could speculate that if subjects had been responding using a more formal checklist technique in which they simply had to Ray eyed or "no" that these two instances might never have been reported. Under the more leisurely approach provided by the protocol technique, the instances emerged from memory. One question that naturally comes to mind is whether we can improve on current interviewing techniques to take advantage of this possible discovery. For example, if respondents were asked to think for a minute, and then answer the question, would we be able to accomplish the same benefits within the context of the more typical interviewing procedures? In short, many interesting issues can be explored through the use of protocols. Specific hypotheses can be tested concerning how personal information is retrieved by people. Moreover, methods for improving the interview process can be tested in this fashion. Order of Retrieval in Free Recall of Autobiographical Material Six subjects were asked, "In the last 12 months, how many times have you gone to a doctor, or a dentist, or a hospital, or utilized any health care specialist or facility? Three subjects were asked, "In the last 12 months, I'd like you to try and recall all the times that you deposited money in your A La Card account., and for each time, try and give me the date as accurately as possible, n or a slight variation on this question. Seventeen subjects were asked both there questions, with the A La Card question coming first. Subjects were instructed to Think out loud" an they responded, and their remarks were taped and transcribed verbatim, yielding 23 protocols in response to the health care question and 20 in response to the A La Card question. Sixteen of the 23 health care protocols and ~ of the 20 A La Card protocols contained fewer than two instances of the behavior in question, thus yielding no information about order of retrieval. For those protocols containing 2 or more instances of the behavior in question, a "+~ was assigned for each time a subject went from a temporally more distant instance to one more recent, and a In for each tome the subject moved from a more recent instance to one in the more distant past. This was regarded as a rough way to quantify the degree to which subjects tended spontaneously to retrieve these autobiographical memories in one direction or another. MA system under which costs of meals eaten by students are charged against an account to which periodic deposits are made.

64 When all the + and - signs were counted up across all the protocols, the results were an follows: Health Care + 7 3 A La Card ~ 23 6 - - When the protocols were clas~ified according to whether the direction of recall was consistently forward (i.e., all +'s), consistently backward (all -'s), not clearly in one direction or another (both +'s and -'s), or there was no information about direction of recall available (fewer than 2 instances of the behavior in question produced), the results were as follows: Health Care: A La Card: all + all - both ~ and - no information 4 2 16 23 all + all - both + and - 3 no information ~ 20 7 2 These results were taken as evidence that, at least in response to these questions, subjects tend to retrieve autobiographical memories in a predominantly past-to-present, or forward, direction. References Ericsson, E.A., and Simon, H.A. 1980 Verbal reports an data. Nisbett, R., and Ross, L. 1980 ~ud~-e~t. Englewood Cliffs, N.J.: Prentice-Hall. T~ersky, A., and Rahneman, D. 1973 Availability: a heuristic for Judging frequency and probability. 0~;~· ~10~ 5:207-232. P~xrholu~al B~x 87:215-251.

TROUGHS AND RESEARCH ON ESTIMATES "OUT PAST IND FUTURE BEHAVIOR Lee Ross The Strategy and Tactics of Survey Methodology as They Pertain to Determining Past Actions and Out comes basic issue raised at the St. Michaels coherence concerned the special status of surveys, such as the National Hearth Interview Survey, which focuses primarily on relatively ob~t~ past actions and outcomes rather than opinions, preferences, fears, intentions, or other largely ~ b ~ ive responses. A great deal of the traditional science and art of the survey methodologist has focused on the problem of potential instrument or interviewer bias. Many strategic decisions about instructions, wording of items, and the role of the interviewer are designed to minimize such potential for bias by minimizing the role of both the instrument and interviewer in defining terms, suggesting response strategies, and especially in providing feedback about the responses themselves. While such precautions may be entirely appropriate in the context of political surveying or other attempts to ascertain attitudes, beliefs, or other subjective states of the respondent, they may be less necessary when the responses in question deal with specific concrete past actions and outcomes by the respondent. Furthermore, in attempting to determine the respondent's past actions and outcomes through measurement of his or her recollections or estimates, a rather different set of potential sources of error or bias come into play--i.e., the types of factors with which cognitive psychologists have long been concerned in their study of human memory and Judgment. Cognitive psychologists have identified many factors that impair performance or introduce error in recall or Judgment and, perhaps even more pertinent to present concerns, they have identified factors or strategies that lead to improvement in performance. If such psychologists were confronted with the problem of designing instruments and interviewer protocols for facilitating accurate recur and estimation of past actions and outcomes, ~ suspect that they would worry relatively little about traditional instrument or interviewer effects and a great deal about how to help the respondent remember the events in question or estimate the relevant magnitudes or frequencies associated with such events. They would worry about the impact of the respondents' general theories, scbemas, or expectations on their recall or estimates. They would worry about the effect of the respondents' concerns with self-presentation (and perhaps selt-perception and evaluation as well). Most importantly, perhaps, they would design instruments and procedures to overcome such obstacles through trial and error testing which measured accuracy of recall or esti~^tion--i.~., compared recollections and estimations to direct measures of the actions and outcomes in question. ~ could offer many Radicals suggestions about techniques that could enhance accuracy of recollections or estimates. One might make heavy use of models--i.e., letting the respondent see someone doing a good Job of being systematic and complete, or using specific memory aids or mnemonics 65

66 in recalling their medical history, consumer behavior, crime victimization experiences, or whatever topic is the focus of the survey. One might encourage the interviewer to offer suggestions about how to remember or estimate the responses or outcomes in question (and train that interviewer in how to prompt such recalls One might explicitly warn the respondent about common biases or errors. One might encourage the respondent to describe events in his or her own words before attempting to answer specific questionnaire items. Perhaps the most extreme possibility would be furnishing the respondent with informative nanchors~--e. g., mean Judgments, typical responses, or ~base-rates. for people in general or people who are like them in terms of pertinent demographic charact ecliptics . Research on Apart ~ and futures Behavioral Estimates for Self and Other One preliminary piece of research has been undertaken in my laboratory that was prompted by the foregoing comments, even though its direct relevance may not be immediately apparent. It was our thesis that recollection of past events, at leant in cases where Episodic recalls is likely to be imperfect at best, or even nonexistent, is closely akin to other related Judgment tacks. Specifically, we sought to compare estimates of specific past performances to parallel predictions about future response and to compare estimates and predictions about one's .°XP responses with parallel estimates and predictions about the responses of a peer whom one knows rather well and has ample opportunity to observe on a day-to-day basis. Ultimately, our concern will be with relative accuracy, and with the relationship between accuracy and confidence, in these four different domains (i.e., set[/other x past/future). Pursuing this concern, however, will demand that we choose responses for which we can independently assess actual behavior to which the relevant estimates or predictions can be compared. This will pose significant methodological hurdles and tax our ingenuity. Undoubtedly, it will also restrict our domain of. inquiry to responses that are normally recorded ~ e. g., checks written, purchases made, books checked out of the library, time logged on computer, long distance telephone calls ~ or at least recordable by an observer (behavior in contrived experimental nettings, perhaps television watching, study the, class attendance, etc.~--domains which may or may not be representative of those that figure in survey concerns and domains and which may or may not be typical in terms of difficulty of. recall or estimation. For now, we have chosen to ignore accuracy per se, and to focus on confidence intervals--i. e., to compare subjects' certainty (or, to be more precise, the range of their uncertainty) about the frequency and magnitude of their own past behaviors or outcomes with their certainty about frequencies for parallel future responses. Furthermore, we compare confidence intermurals regarding self-estimates and predictions with parallel estimates and predictions for other people. In other words, we are comparing estimates or recollections about one's past behavior, about

67 which one might have a basis for considerable certainty, with three types of estimate that one would expect to be highly uncertain and heavily Theory basely rather than data based. In our single research effort, Stanford students were asked to make estimates about a variety of pant and future responses either for themselves (self-estimate) or for their roommates (roommate estimates). Items included number of checks written (or to be written) in a 30-day period, money spent in restaurants, hours spent watching television, number of long distance telephone calls, and so forth. For each item they made a best guess and then bracketed that best guess with an upper and lower confidence limit. In [act, they furnished two different confidence limits: 50 percent limits (such that they thought the probability of the upper limit being too low was .25 (or 25 percent) and that the probability of the lower limit being too high was also .25 (or 25 percent) and 80 percent limits ~ such that the probabilities of the limits being too low or too high were each NO percent). They also rated the ease or difficulty of making each estimate, and indicated how "surprisedn they would be if the "right answers was not contained within the confidence limits that had been specified (although these items shall not be dealt with in thin brief report). The research design made use of both within- and between-~ubJect comparisons--with self versus other a between-subject factor and past versus future a within-subJect factor (order of past vs. future was counterbalanced , as was the order of specific items). The results of this pilot effort (see Table 1) can be summarized succinctly. Confidence limits for predictions of the future were only TABLE ~ Relative Width of 50 Percent Confidence Intervals for Estimates of Past Behavior and Predictions of Future Behavior Estimate of Prediction of Past Behavior Future Behavior Combined Judgment about self ~ 00 ~ 3 ~ ~ ~ 5 Judgment about roommate 1 1 1 124 117 Combined ~ 05 ~ 28 Note: All intervals were transformed to reflect magnitude relative to interval for past behavior of self. Means reported summarize results for 1-2 behavioral estimates and predictions. A total of 25 subjects offered confidence intervals for self only (both pant and future behavior) and an equal number offered confidence intervals for roommate only (both past and future behavior).

68 modestly wider than confidence limits for estimates about the past regardiens of whether it was the self or ones roommate who was the target. Furthermore, confidence internals about one's roommate were no wider, overall, than confidence levels about oneself; for estimates about the past, the confidence interval was slightly wider for roommate than self; for predictions about the future, the reverse wan true. None of theme main effects or interaction effec~c^, moreover, appear to be statistically significant (although this may change somewhat when data from a second cohort of subjects in added to our analysis. It is particularly noteworthy that the confidence intervals for ostensively data-based estimates about one's pant behavior were only marginally narrower than for the ostensively theory-based estimates about the future responses of one's roommates. Such data certainly prompt one to wonder exactly how data-based estimates about ones past really are! They also encourage the type of speculation offered earlier--i. e., that the accuracy of any inferences we might want to make about that behavior would be facilitated by procedures that facilitated recall or encouraged more accurate estimation strategies on the part of the respondent. Finally, the data comparing self-est imates and roommate estimates are interesting in their own right, beyond any relevance to concerns of survey methodology. People apparently believe that they can mare as accurate and confident estimates about other people' ~ responses as they can about their own, particularly when it is future rather than past responses that are the subject of such estimates. It is obviously tempting to find out whether such relative immodesty regarding one's social predictions, and modesty regarding self-prediction, is ''ustified ~ Just as it is tempting to find out whether one's ability to predict the future is really as good, and one's estimates about the pant are really as bad, as suggested by the confidence limits offered in our study). The need for follow-up research is e~rident--research in which accuracy, and therefore calibration of such confidence intervals, can be assessed directly.

69 OUTREACH ACTIVITIES In the months that have elapsed since the January 1984 meeting in Baltimore, several of the seminar participants have been active in publicizing the activities and outputs of CASH and in seeking to encourage others to work in this exciting cross-disciplinary field. The principal method of outreach has been the presentation and publication of papers, but other means are also being used. On January 26, 1984, Norman Bradburn attended a seminar, Problems of Measuring Behavior,. sponsored by the Economic and Social Research Council in London, England, and presented a paper, Potential contributions of cognitive sciences to survey questionnaire design. ~ In February Bradburn made presentations on the same subject to three organizations in West Germany: the Zentrum fur Umtragen und Methodische Analyze (Z0M4, in Mannheim, the Institut fur Demo-~kopie in Allenabach, and the Max Planck Society in Munich. In July 1984 Roger Tourangeau attended a ZUMA seminar entitled Asocial Information Processing and Surrey Methodology and presented a paper, "Question order and context effects. n At the 39th Annual Conference of. the American Association for Public Opinion Research held on May 17-20, 1984, at Lake Lawn Lodge, Dele~ran, Wisconsin, Judith Tanur chaired a session on contributions of cognitive psychology to survey research that included a paper entitled Ran information processing approach to recall in surveys. by Norman Bradburn and a paper entitled Attitude measurement: a cognitive perspectives by Roger Tourangeau. The discussant was Elizabeth Martin of the Bureau of Social Science Research. At the annual meeting of the American Statistical Association held on August ~ 3- ~ 6, ~ 984, in Philadelphia, Pennsylvania, a session on cognitive aspects of survey methodology was sponsored by the Section on Survey Research Methods and cosponsored by the Social Statistics and Statistical Education Sect ions . Organized and chaired by Judith Tanur, the session included a paper by Roger Tourangeau entitled Interchanges between cognitive science and survey methodology and a paper authored by David C. Fathi, Jonathan W. Schooler, and Elizabeth F. Lortus, presented by Elizabeth Loftus, entitled Moving survey problems into the cognitive psychology laboratory. The discussants were two of the CASH guests at St. Michael~, Dr. Jacob J. Feldman of the National Center for Health Statistics and Professor Phillip J. Stone of. Harvard University. A paper entitled Recognitive psychology meets the national survey. by Elizabeth F. Lofts, Stephen E. Fienberg, and Judith M. Tanur has been prepared in response to an invitation from the ~rtr~ and is expected to appear ~ n December 1984. The editor of the ~ _~ has invited members of the CASH group to prepare papers. A paper entitled nCognitive aspects of health surveys for public information and policy. is being prepared by Stephen E. Fienberg, Elizabeth F. Loftus, and Judith M. Tanur. A companion piece being prepared by Monroe Sirkin and Judith Lessler stems from their laboratory-based research project at the National Center for Health Statistics. The editor of the ~v in inviting several

JO Health Statistics. The editor of the ~a~y is inviting several discussants for these papers. In response to a preliminary proposal presented at its board meeting in June 1984, the Social Science Research Council (SSRC) is organizing a working group on cognition and survey research in order to prepare a detailed plan for the activities of a possible SSRC committee that would bear thesame name. Cochaired by Robert Abelson and Judith Tanur, the working group includes Roy D'Andrade, Stephen Fienberg, Robert Groves, Robin Hogarth, Don Kinder, and Elizabeth LofLus. The staff person responsible for this effort at SSRC is Robert Pearson, a guest at CASM's Balt imore meet ing. Theme are the outreach activities that the editors have been able to identify as this report goes to press; there may well be others that have escaped their attention. It seems reasonable to predict, on the basis of. this record, that we can look forward to a continuing round of relevant reports and discussions an further results emerge from the cross-disciplinary research programs and activities genera~ced by the CASH project .

Next: APPENDIX A BACKGROUND PAPERS »
Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines Get This Book
×
Buy Paperback | $50.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF
  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!