The Dynamics of Disability Measuring and Monitoring Disability for Social Security Programs (2002) / Chapter Skim
Currently Skimming:
Methodological Issues in the Measurement of Work Disability
Methodological Issues in the Measurement of Work Disability
Pages 211-240
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 211... ...
The Social Security Administration (SSA) is currently reengineering its disability claims process for providing benefits to blind and disabled 1This paper was originally prepared for the committee workshop titled "Workshop on Survey Measurement of Work Disability: Challenges for Survey Design and Method" held on May 27-28, 1999 in Washington, D.C.
|
From page 212... ...
For both the conduct of the DES and the monitoring of the pool of potential beneficiaries through the use of various data collection efforts, it is critical to understand the measurement error properties associated with the identification of persons with disabilities as a function of the essential survey conditions under which the data have been and will be collected. The extent to which alternative instruments designed to measure persons with disabilities map to various eligibility criteria under consideration by SSA is also important.
|
From page 213... ...
As a first step to achieving Ws goal, a common language and framework needs to be established for Me enumeration and assessment of the various sources of error that affect the survey measurement process. The chapter draws from several empirical investigations to provide evidence as to the extent of knowledge concerning the error properties associated with various approaches to the measurement of functional limitations and work disability.
|
From page 214... ...
The concept of variable error requires the possibility of repeating the survey, with changes in the units of replication, that is, the particular set of respondents, interviewers, supervisors, coding, editing, and data entry staff. Errors of Nonobservation Within the framework of survey methodology, both variable error and bias are further characterized in terms of errors of nonobservation and errors of observation.
|
From page 215... ...
Nonresponse Error Nonresponse error can arise from failure to obtain any information from the persons selected to be measured (unit nonresponse) or from failure to obtain complete information from all respondents to a particular question (item nonresponse)
|
From page 216... ...
. This section briefly reviews the theoretical framework and empirical findings related to the various sources of measurement error in surveys.
|
From page 217... ...
Respondent as Source of Measurement Error Once the respondent comprehends the question, he or she must retrieve the relevant information from memory, make a judgment as to whether the retrieved information matches the requested information, and communicate a response. Much of the measurement error literature has focused on Me retrieval stage of the question-answering process, classifying the lack of reporting of an event as retrieval failure on the part of the respondent and comparing the characteristics of events that are reported with those that are not reported.
|
From page 218... ...
The findings suggest that proxy reports of functional limitations tend to be higher than self-reports; the research is inconclusive as to whether the discrepancy is a function of overreporting on the part of proxy informants, underreporting on the part of self-respondents, or both.
|
From page 219... ...
Other Essential Survey Conditions as Sources of Measurement Error Any data collection effort involves decisions concerning the features that define the overall design of the survey, referred to here as the "essential survey conditions." In addition to the sample design and the wording of individual questions and response options, these decisions include the following: . whether to use interviewers or to collect information via some form of self-administered questionnaire; · the means for selecting and training interviewers (if applicable)
|
From page 220... ...
For example, as noted above, interviewer variance is one source of variability that can be eliminated through the use of a selfadministered questionnaire. However, the use of an interviewer may aid in the measurement process by providing the respondent with clarifying Information or by probing insufficient responses.
|
From page 221... ...
The focus for psychometrics is on variable errors; from the perspective of classical true score theory, all questions produce unbiased estimates, but not necessarily valid estimates, of the construct of interest. The confusion arises in that both statistics and psychometrics use the terms validity and reliability to sometimes refer to very similar concepts and to sometimes refer to concepts that are quite different.
|
From page 222... ...
If repeated measurement resulted in consistent reports by all respondents, test-retest measures would indicate a high degree of reliability, not dissimilar to Me conclusion drawn by statisticians. Within survey research, the conduct of a reinterview under the same essential survey conditions as the original interview is an example of a test-retest assessment of reliability.
|
From page 223... ...
Impairments or disabilities may limit a person's ability to participate in the survey process or limit access to the individual. The essential survey design features of a data collection effort can facilitate or limit access and participation of persons with disabilities.
|
From page 224... ...
Many of the current measures of disability used in federal data collection efforts have not been subjected to testing methods common to new questions and questionnaires, for example, cognitive interviewing and behavior coding. Cognitive interviewing encompasses several techniques designed to elicit information about the respondent's comprehension of the question, the strategies by which the respondent attempts to retrieve information from memory, judgments as to whether the retrieved information meets the perceived goals of the question, and the formulation of responses.
|
From page 225... ...
for a cognitive evaluation of questions from the Survey of Income and Program Participation and the National Health Interview Survey concerning discrepancies in print reading disability statistics.
|
From page 226... ...
Ibe actor-observer difference suggests that actors draw on situational information to explain behavior at any given time, whereas observers use stable disposition properties of the actor to understand behavior Cones and Nisbett, 1971~. To the extent that proxy reporters view disabilities as stable as opposed to dynamic characteristics, one would anticipate rliscrepancies between self-reports and proxy reports.
|
From page 227... ...
. EMPIRICAL EVIDENCE CONCERNING MEASUREMENT OF DISABILITY ERROR To date, most investigations with respect to Me error properties associated with the measurement of persons win disabilities or the measurement of persons with work disabilities have focused on errors of observation, ignoring differences in estimates due to coverage error and nonresponse error.
|
From page 228... ...
by using different sets of ADL items and across different modes.7 They conclude that the measurements of functional limitations with respect to counts of ADLs, indications of the use of assistive devices or personal help, and indications of any difficulty are all subject to large amounts of measurement error, of which a substantial portion is random error. Similar to other empirical work (e.g., Mathiowetz and Lair, 1994)
|
From page 229... ...
SOURCE: McNeil, 1993. TABLE 2 Self-Care Limitations: Distributions to Census Question 19b and Content Reinterview Survey Question 34b, Persons 16 to 64 Years of Age, United States, 1990 Content Reinterview Survey: Difficulty Going Outside Census Long Form: Difficulty Talcing Care of Personal Needs Yes No Total Yes 69 346 415 No 120 13,856 13,976 Total 189 14,202 14,391 NOTE: The prevalence rate based on census: 2.9 percent, of which 16.6 percent were consistent responses.
|
From page 230... ...
of those who report a self-care limitation at the time of the census fail to report a self-care limitation in the CRS. Comparison of the percentage of persons with mobility and self-care limitations from the two surveys is confounded by differences in the essential survey conditions under which the data were collected and that most likely contribute to the discrepancies evident in the data.
|
From page 231... ...
These empirical findings illustrate some of the error properties associated with the measurement of functional limitations and sensory impairments. The research indicates that despite psychometric measures that indicate a relatively high degree of reliability, survey applications offer several examples of low levels of reliability, even under conditions in which the essential survey conditions are held constant.
|
From page 232... ...
Not dissimilar to the assessment of functional limitations, work disability is measured in data collection efforts that vary with respect to the essential survey conditions, the specific wording of questions, the number of questions asked, and the determination of severity, duration, and the use of assistive devices or environmental barriers.
|
From page 233... ...
raises questions concerning the validity of the work disability measures currently in use, several empirical investigations raise questions about the reliability of these measures, not unlike the findings with respect to the measurement of functional limitations and sensory impairments. Once again, it can be seen that differences in the wording of the questions, the context in which they are asked, the nature of the respondent, and other essential survey conditions, including the data collection organization and the sponsorship of the survey, may contribute to differences in estimates of the working-age disabled population.
|
From page 234... ...
sponsoring the study. Although the wording of the questions is quite similar across the various surveys, there are some minor differences in specific wording (e.g., differences with respect to the emphasis on a health condition)
|
From page 235... ...
Although these variations in estimates derived from different surveys suggest instability in the estimates of the proportion of persons with work disabilities as a function of the wording of the question, me nature of the respondent, and the essential survey conditions under which the measurement was taken, they provide little information about measurement
|
From page 236... ...
QUESTION WORDING ISSUES RELATED TO SELECTED MEASURES OF WORK DISABILITY Jette and Badley point out the conceptual problems inherent in many questions designed to measure persons with work disabilities, including the failure of most questions to enumerate the separate elements related to the role of work. That failure is evident in most work disability screening questions designed to be administered to the general adult population.
|
From page 237... ...
For example, if an individual is currently employed in an environment that accommodates a health condition, the respondent must determine whether the person should be considered as having difficulty working, even though the present employment situation presents no difficulty to the person. The NHIS asks two questions concerning work limitations: Does any impairment or health problem NOW keep from working at a job or business?
|
From page 238... ...
Such tee -- es ~111 Ed ~ He underst~d~g of He validity of He questions ~d, Coup He refinement of He Jordan of questions hopefully 1-rove He reb~1~ of He items. Sadly documenting Hat v=1ad~ ~ ~ essenha1 survey condH10ns of He measurement process commutes to deferent estates of persons w1+ work dis~1~hes is not sufficient He markup enacts of various factors need to be measured Ad He -act needs to be reduced Coup He use of aHe=adve design Cadres.
|
From page 239... ...
eds. Measurement Errors in Surveys.
|
From page 240... ...
1993. Content Reinterview Survey: Accuracy of Data for Selected Population and Housing Characteristics as Measured by Reinterview.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.