Psychological Testing in the Service of Disability Determination (2015)

Summary¹

BACKGROUND

In 2012, the U.S. Social Security Administration (SSA) provided benefits to nearly 15 million disabled adults and children through two disability programs. The majority of beneficiaries, 8.8 million, received benefits through the Social Security Disability Insurance (SSDI) program for disabled individuals, and their dependent family members, who have worked and contributed to the Social Security trust funds. The remaining beneficiaries (4.9 million adults and 1.3 million children) received benefits through the Supplemental Security Income (SSI) program, which is a means-tested program based on income and financial assets for adults aged 65 years or older and disabled adults and children.

SSA disability determinations are based on the medical evidence and all evidence considered relevant by the examiners in an applicant’s case record. Physical or mental impairments must be established by objective medical evidence consisting of medical signs and laboratory findings, which may include psychological tests and other standardized test results. SSA establishes the presence of a medically determinable impairment in individuals with mental disorders other than intellectual disability through the use of standard diagnostic criteria, which include symptoms and signs. Evidence for these mental impairment claims, as well as for many other categories of claims, such as those for certain musculoskeletal and connective tissue

_____________

¹ This summary does not include references. Citations to support text, conclusions, and recommendations made herein are provided in the body of the report.

conditions, relies less on standard laboratory tests than for some other categories of impairment.

SSA maintains a list of criteria for specific conditions that an applicant with one or more of those conditions must meet in order to receive disability benefits based solely on medical criteria. SSA currently requires psychological test results, specifically intelligence test results, in the listing criteria for intellectual disability in children and adults and in the criteria for cerebral palsy, convulsive epilepsy, and meningomyelocele and related disorders. SSA questions the value of purchasing psychological testing in cases involving mental disorders, other than for intellectual disability, and it does not require testing either to establish or to assess the severity of other mental disorders.

As noted, SSA indicates that objective medical evidence may include the results of standardized psychological tests. Given the great variety of psychological tests, some are more objective than others. Whether a psychological test is appropriately considered objective has much to do with the process of scoring. For example, unstructured measures that call for open-ended responding rely on professional judgment and interpretation in scoring; thus, such measures are considered less than objective. In contrast, standardized psychological tests and measures, such as those discussed in the report, are structured and objectively scored. In the case of non-cognitive self-report measures, the respondent generally answers questions regarding typical behavior by choosing from a set of predetermined answers. With cognitive tests, the respondent answers questions or solves problems, which usually have correct answers, as well as he or she possibly can. Such measures generally provide a set of normative data (i.e., norms), or scores derived from groups of people for whom the measure is designed (i.e., the designated population), to which an individual’s responses or performance can be compared. Therefore, standardized psychological tests and measures rely less on clinical judgment and are considered to be more objective than those that depend on subjective scoring. Unlike measurements such as weight or blood pressure, standardized psychological tests require the individual’s cooperation with respect to self-report or performance on a task. The inclusion of validity testing in the test or test battery allows for greater confidence in the test results. Standardized psychological tests that are appropriately administered and interpreted can be considered objective evidence.

As illustrated in Figure S-1, standardized psychological testing is one component of a full psychological assessment. Standardized psychological tests can be divided into measures of typical behavior and tests of maximal performance. Measures of typical behavior, such as personality, interests, values, and attitudes, may be referred to as non-cognitive measures. Tests of maximal performance ask people to answer questions and solve problems

FIGURE S-1 Components of psychological assessment.

NOTE: Performance validity tests do not measure cognition but are used in conjunction with performance-based cognitive tests to examine whether the examinee is exerting sufficient effort to perform well and responding to the best of his or her capability. Similarly, symptom validity tests do not measure non-cognitive status but are used to examine whether a person is providing an accurate report of his or her actual symptom experience. Because cognitive tests frequently are performance based and non-cognitive measures generally involve self-report, performance validity tests and symptom validity tests are shown as being associated with these types of tests.

as well as they possibly can. Because tests of maximal performance typically involve cognitive performance, they are often referred to as cognitive tests. It is through these two lenses—non-cognitive measures and cognitive tests—that the committee examined psychological testing for the purpose of disability evaluation in this report. Intelligence tests and neuropsychological tests are examples of cognitive tests, while depression, anxiety, or personality inventories are examples of non-cognitive measures. Cognitive tests tend to be performance based, and non-cognitive measures tend to be based on self-report. Validity testing is an area of psychological testing. Performance validity tests (PVTs) provide information about an individual’s effort on tests of maximal performance, such as cognitive tests. Symptom validity tests (SVTs) provide information about the consistency and accuracy of an individual’s self-report of symptoms he or she is experiencing.

There are differences of opinion on the use of validity tests and their value for work disability evaluations. Current SSA policy precludes the purchase of validity tests as part of a consultative examination to supplement an applicant’s medical evidence record, although applicants and their representatives sometimes submit validity test results in support of their claims.

Professional organizations of neuropsychologists and psychologists have issued position statements and guidance advocating for the use of validity tests in clinical and medicolegal contexts, and several have challenged SSA’s institutional prohibition on ordering such tests. A September 2013 report from SSA’s Office of the Inspector General concluded that although SSA does not allow the purchase of validity tests, “medical literature, national neuropsychological organizations, other federal agencies, and private disability insurance providers support the use of [validity tests] in determining disability claims.”

It is within this context that SSA asked the Institute of Medicine (IOM) to convene a committee of relevant experts (e.g., adult and pediatric neuropsychology, psychology, psychiatry, disability medicine, behavioral economics, and economics) to review selected psychological tests, including validity tests, and to provide guidance on the use of such testing in the adjudication of claims submitted to the SSA Disability Programs (see Box S-1 for the statement of task). In carrying out this task, the Committee on Psychological Testing, Including Validity Testing, for Social Security Administration Disability Determinations was asked to address several specific topics, including testing norms, the administration of relevant tests and the qualifications for administering them, the interpretation and reporting of test results, and economic considerations.

COMMITTEE’S APPROACH TO ITS CHARGE

As part of its information-gathering process, the committee conducted an extensive review of the literature pertaining to the use of psychological tests, including PVTs and SVTs, in disability determinations. The committee supplemented its review of the literature with two public workshops to hear from neuropsychologists with expertise in performance validity and symptom validity testing in adults and children, the use of psychological and validity tests in culturally diverse populations, and the use of such tests in non-SSA disability determination contexts (e.g., private disability insurance programs, Canadian auto insurance, U.S. military disability or return-to-duty decisions, veterans’ disability compensation). The committee also heard from SSA and Disability Determination Services (DDS) representatives about the SSA disability determination process and its current policies surrounding the use of psychological and validity testing. The committee commissioned two papers to provide additional critical analysis in areas relevant to the committee’s work. The committee’s work was further informed by previous IOM and National Research Council reports focused on different aspects of the SSA disability determination process.

COMMITTEE’S RECOMMENDATIONS

The committee identified three elements of SSA’s disability determination process in which psychological testing could play a role: (1) identification of a “medically determinable impairment,” (2) evaluation of functional capacity for work, and (3) assessment of the validity of applicants’ psychological test results or the consistency of applicants’ statements about self-reported symptoms. Although this report addresses all three elements, the committee focuses on the second and the third, for which questions about the use of psychological tests are more complex. As indicated in the following section, the committee found that the results of standardized psychological testing do provide information of value to each of the three elements.

Role of Psychological Testing in Social Security Administration Disability Programs

There currently is great variability in allowance rates for both SSI and SSDI among states that are not fully accounted for by differences in the populations of applicants. In addition, there is great variability in the disability determination appeal rulings among administrative law judges within and across states. Each state DDS agency, within the confines of SSA policy, issues its own rules regarding the tests that may be purchased as part of a consultative examination. Aside from the use of intelligence tests as

described in the listings for intellectual disability and certain neurological impairments, SSA does not require or specify the purchase of any type of (or individual) psychological test. SSA provides general guidance that good psychological tests are valid and reliable and have appropriate normative data. For this reason, there is variation among states about when and which standardized psychological tests can be purchased, with the exception of SVTs and PVTs, which are precluded from purchase by SSA except in rare cases such as a court order.

Although there currently are no data on the rates of false positives and false negatives in SSA disability determinations, systematic use of standardized psychological testing for a broader set of physical and mental impairments than is current practice is expected to improve the accuracy and consistency of disability determinations for applicants who allege cognitive impairment or whose allegation of functional impairment is based solely on self-report. The results of standardized cognitive and non-cognitive psychological tests that are appropriately administered, interpreted, and validated can provide objective evidence to help identify and document the presence and severity of medically determinable mental impairments at Step 2 of SSA’s disability determination process. In addition, standardized cognitive test results can provide objective evidence to help identify and assess the severity of work-related cognitive functional impairment relevant to disability evaluations at the listing level (Step 3) and to mental residual functional capacity (Steps 4 and 5).

Current data on the prevalence of inconsistent reporting of symptoms or performing below one’s capability on cognitive tests are very imprecise. In the context of SSA disability applicants, neither scenario rules out disability, but both suggest the need for additional assessment of the alleged impairment with the goal of making an accurate determination of disability. When a disability claim is based primarily on an applicant’s self-report of symptoms and self-reported statements about their intensity, persistence, and limiting effects, SSA relies on an assessment of the consistency of the self-report with all of the evidence in the applicant’s medical evidence record.

Although SSA’s current policy precludes the purchase of SVTs and PVTs, these tests provide information about the validity of standardized non-cognitive and cognitive test results when administered as part of the test or test battery and therefore are an important addition to the medical evidence record in such cases. It is important that SVTs and PVTs only be administered in the context of a larger test battery and only be used to interpret information from that battery. Validity tests do not provide information about whether or not the individual is, in fact, disabled.

Standardized Non-Cognitive Psychological Measures and Symptom Validity Tests

The use of standardized non-cognitive psychological measures is essential to the determination of all cases in which an applicant’s allegation of non-cognitive functional impairment meets each of three requirements:

• The applicant alleges a mental disorder (i.e., schizophrenic, paranoid, and other psychotic disorders; affective disorders; anxiety-related disorders; and personality disorders) unaccompanied by cognitive complaints or a disorder with somatic symptoms that are disproportionate to demonstrable medical morbidity (i.e., somatoform disorders, multisystem illnesses, and chronic idiopathic pain conditions).
• The presence and severity of impairment and associated functional limitations are based largely on applicant self-report.
• Objective medical evidence or longitudinal medical records sufficient to make a disability determination do not accompany the claim.

In certain instances, cognitive concerns may accompany the applicant’s allegations, in which case cognitive testing, as discussed below, may be more appropriate. The committee recognizes there are a few chronic conditions (e.g., schizophrenia, chronic idiopathic pain, multisystem illnesses) that may generate potentially disabling, non-cognitive functional impairments but may not be accompanied by objective medical evidence. In such cases, the evidence provided by longitudinal medical records may be sufficient to substantiate the allegation.

Assessment of symptom validity, including the use of SVTs, analysis of internal data consistency, and other corroborative evidence, helps the evaluator to interpret the accuracy of an individual’s self-report of behavior, experiences, or symptoms and responses on standardized non-cognitive psychological measures. For this reason, it is important to include an assessment of symptom validity when non-cognitive psychological measures are administered. Evidence of inconsistent self-report based on symptom validity measures is cause for concern with regard to self-reported symptoms but does not provide information about whether or not the individual is, in fact, disabled. A lack of validity on symptom validity testing alone is insufficient grounds for denying a disability claim, although additional information would be required to assess the applicant’s allegation of disability.

• All non-cognitive psychological evaluations should include a statement of evidence of the validity of the results, which could include symptom validity test results, analysis of internal data consistency (e.g., item response theory), and other corroborative evidence as well as discussion of the test norms relative to the individual being assessed.
• For cases in which validation is not achieved, SSA should pursue additional evidence of the applicant’s allegation.

The committee intends standardized non-cognitive psychological tests to include measures of behavior, affect, personality, and psychopathology. By objective medical evidence in this and the following recommendation, the committee means medical signs and/or laboratory or test results that constitute clear objective medical evidence of a significant mental disorder and related functional impairment of sufficient severity to make a disability determination. An example would be a severe brain injury associated with significant functional deficits (e.g., minimally conscious state). By longitudinal medical records the committee means a documented history of a significant mental disorder or a chronic condition such as chronic idiopathic pain or multisystem illness and related functional impairment of sufficient severity and duration to make a disability determination. An example would be a well-documented history of repeated hospitalizations and treatments for a diagnosed mental disorder, such as an affective or personality disorder.

The committee intends the “statement of evidence of the validity of the results” specified in this and the following recommendation to reflect objective evidence that goes beyond the clinical opinion of the examiner. In addition to analysis of the results of SVTs or PVTs administered at the time of the testing and analysis of internal data consistency, evidence could include a pattern of test results that is inconsistent with the alleged condition, observed behavior, documented history, and the like. It is important to note that a finding of inconsistency between the test results and the areas specified is more informative than a finding of consistency would be.

The committee’s recommendation here and in the following recommendation that SSA “pursue additional evidence of the applicant’s allegation” for cases in which validation is not achieved means that the test results in those cases are an insufficient basis to make a determination regarding disability status.

Standardized Cognitive Tests and Performance Validity Tests

Standardized cognitive test results are essential to the determination of all cases in which an applicant’s allegation of cognitive impairment is not accompanied by objective medical evidence. The results of cognitive tests are affected by the effort put forth by the test-taker. If an individual has not given his or her best effort in taking the test, the results will not provide an accurate picture of the person’s neuropsychological or cognitive functioning. Performance validity indicators, which include PVTs, analysis of internal data consistency, and other corroborative evidence, help the evaluator to interpret the validity of an individual’s neuropsychological or cognitive test results. For this reason, it is important to include an assessment of performance validity when cognitive testing is administered. It also is important that validity be assessed throughout the cognitive evaluation.

A PVT only provides information about the validity of an individual’s cognitive test results that are obtained during the same evaluation. Evidence of invalid performance based on PVT results pertains only to the cognitive test results obtained and does not provide information about whether or not the individual is, in fact, disabled. A lack of validity on performance validity testing alone is insufficient grounds for denying a disability claim. In such cases, additional information is required to assess the applicant’s allegation of disability.

• All cognitive evaluations should include a statement of evidence of the validity of the results, which could include performance validity test results, analysis of internal data consistency (e.g., item response theory), and other corroborative evidence as well as discussion of the test norms relative to the individual being assessed.
• For cases in which validation is not achieved, SSA should pursue additional evidence of the applicant’s allegation.

Qualifications for Test Administration and Interpretation

Use of standardized procedures for the administration of standardized non-cognitive and cognitive psychological tests enables application of normative data to the individual being evaluated. Without standardized administration, the test-taker’s performance may not accurately reflect his or her ability. It is important that any person administering cognitive or neuropsychological tests be well trained in the administration protocols for those particular tests, possess the interpersonal skills necessary to build rapport with the test-taker, and understand important psychometric properties, including validity and reliability, as well as factors that could emerge during testing to place either at risk.

Interpretation of standardized psychological test results is more than a report of the standardized test scores; it requires assigning meaning to the scores within the individual context of the specific examinee. As such, interpretation of test results requires a higher level of clinical training than does the administration alone of some psychological tests. Licensed psychologists and neuropsychologists are the specialists qualified to interpret the results of most standardized psychological and neuropsychological tests. Under close supervision and direction of licensed psychologists and neuropsychologists, it is standard practice for psychometrists or technicians with specialized training to administer and score tests. Test manuals specify the qualifications necessary for administration, scoring, and interpretation of the test or measure. It is important as well that the individual responsible for making the disability determination (disability examiner or administrative law judge) have the training and experience to understand and evaluate the report provided by the psychologist or neuropsychologist.

• “Qualified” means that the specialist must be currently licensed or certified to administer, score, and interpret psychological tests and have the training and experience to administer the test and interpret the results.
• This recommendation applies not only to standardized psychological testing that may be ordered in the course of a disability evaluation, but also to standardized psychological testing already in an applicant’s medical evidence of record if the results are considered as part of the disability determination.

Economic Considerations

Systematic use of standardized psychological testing in SSA disability evaluations for a broader set of physical and mental impairments than is current practice will have financial implications. The average cost of testing services varies by the type of testing (e.g., psychological, neuropsychological), by the type of provider (e.g., psychologist or physician, technician), and by geographic area. The variation in pricing implies that the expected costs to SSA of requiring psychological testing will depend on exactly which tests are required, the qualifications mandated for testing providers, and the geographical location of the providers most in demand. Estimating the exact cost of broad use of psychological testing by SSA will require more detailed data on the exact implementation strategy.

At present, there do not appear to be any independently conducted studies regarding the accuracy of the disability determination process as implemented by DDS offices. Some published estimates of billions of dollars in potential cost savings to SSA associated with the use of symptom validity testing and performance validity testing are based on assumptions that if violated would substantially lower the estimated cost savings. Potential cost savings associated with testing vary considerably based on the assumptions about who it is applied to and how many individuals it detects and thus rejects for disability benefits. A full financial cost-benefit analysis of psychological testing will require SSA to collect additional data both before and after the implementation of the recommendations of this report.

Evaluation and Research

Based on its examination of the literature and dialogues with experts in a variety of areas, including psychological and neuropsychological testing, performance validity testing and symptom validity testing, and the disability evaluation process both within SSA and in other arenas, the committee recognizes many questions remain with regard to the use of standardized psychological testing in the disability determination process.

As part of its assessment of the use of standardized psychological tests for the disability evaluation process, the committee was asked to discuss the costs and cost-effectiveness of requiring a single test or a combination of tests. This report provides an initial framework for evaluating the economic costs and highlights the types of data that will be needed to accurately determine the financial impact of implementing the committee’s first two recommendations. The following conclusions and recommendation relate to this enterprise.

Conclusions

• Accurate assessments of the net financial impact of psychological testing as recommended by the committee will require information on the current accuracy of DDS decisions and how the accuracy is affected by the increased use of standardized psychological testing.
• The absence of data on the rates of false positives and false negatives in current SSA disability determinations precludes any assessment of their accuracy and consistency.
• There currently is great variability in allowance rates for both SSI and SSDI among states that are not fully accounted for by differences in the populations of applicants. There also is great variability in the disability determination appeal rulings among administrative law judges within and across states. Although it is not possible to know definitively whether the large share of unexplained variation in state filing, award, and allowance rates is driven by variability in the federal disability determination process, there is some evidence that states differ in how they manage claims.
• In light of this unexplained variability, systematic use of standardized psychological testing as recommended by the committee is expected to improve the accuracy and consistency of disability determinations.

• Accuracy refers to the rates of false negatives and false positives in SSA’s disability determinations.
• Consistency means that adjudicators presented with the same evidence for comparable cases come to the same conclusion.

Recognizing that the costs and benefits of implementing the committee’s recommendations go beyond the financial, the committee recommends that SSA evaluate the effect of implementing the committee’s recommendations on its disability determination process using a number of different measures.

• Number of backlogged cases;
• Efficiency of throughput or time to determination;
• Number of requests for appeals;
• Adherence to recommended evaluations;
• Effect on accuracy and consistency of disability determinations; and
• Effect on state-to-state variation in disability allowance rates and on appeal rulings among administrative law judges.

Over the course of the project, the committee identified two areas in particular in which it expects that the results of further research would help to inform disability determination processes as indicated in the following conclusions and recommendation.

Conclusions

• Additional research is needed on the use of SVTs and PVTs in populations representative of the pool of disability applicants, including in terms of gender, ethnicity, race, primary language, educational level, medical condition, and the like. In particular, additional research on the development of appropriate criterion or cutoff scores for PVTs and SVTs in these populations for the purposes of disability evaluation would be beneficial.
• The committee’s task was to evaluate the usefulness of psychological testing in the disability determination process, as reflected in the foregoing recommendations. However, the committee recognizes that just as systematic use of standardized psychological testing is expected to improve the accuracy and consistency of disability determinations for applicants who allege cognitive impairment or whose allegation of functional impairment is based solely on self-report, the use of other standardized assessment tools also may be expected to improve the accuracy of disability determinations. The value of standardized assessment tools, including psychological tests, to assessments of individuals’ work-related functional capacity is an area that would benefit from further research.

• The effects of standardized psychological testing on the accuracy and consistency of disability determinations;
• The use of PVTs and SVTs with disability applicants; and
• The use of psychological tests, including PVTs and SVTs, in different populations with regard to fairness for members of all gender, ethnic, racial, language, educational levels, and other protected groups.