Reporting Test Results for Students with Disabilities and English-Language Learners: Summary of a Workshop (2002)

Peggy Carr, associate commissioner for assessment at the National Center for Education Statistics, and Jim Carlson, assistant director for psychometrics at the National Assessment Governing Board (NAGB), made the opening presentations, providing historical context about the inclusion of students with special needs in NAEP and laying out what they hoped to learn from the days’ interactions. Carlson began by describing a series of resolutions through which NAGB established a plan for conducting research on the effects of including students with disabilities and English-language learners in the assessment. In these resolutions, the Board articulated dual priorities of including students who can “meaningfully take part” in the assessment while also maintaining the integrity of the trend data that are considered a key component of NAEP. According to Peggy Carr, the resolution and research plan provided “a bridge to the future” in which NAEP would be more inclusive, and “a bridge to the past” in which NAEP would continue to provide meaningful trend information. One of the chief concerns was that new policies and procedures would not interfere with the ability to report trends in the important subjects both for the nation and for the states.

In her presentation, Carr described the research plan implemented with the 1996 mathematics assessment. This plan called for data to be collected for three samples, referred to as S1, S2, and S3. The S1 sample maintained the status quo, in which administration procedures were handled in the same way as in the early 1990s. In the early 1990s, a student with an

individual education plan (IEP) could be excluded from the assessment if he or she was mainstreamed less than 50 percent of the time in academic subjects or was judged to be incapable of participating meaningfully in the assessment (U.S. DoEd, 1994). Any students identified by school officials as “limited English proficient” could be excluded if he or she was “a native speaker of language other than English,” had been enrolled “in an English-speaking school for less than two years,” and was “judged to be incapable of taking part in the assessment” (U.S. DoEd, 1994: pg. 126).

In the S2 sample, revisions were made to the criteria given to schools for determining whether to include students with special needs, but no accommodations or adaptations were offered. For S2, students with IEPs were to be included unless

Students designated as limited English proficient by school officials and

In S3, the revised inclusion criteria were used, and accommodations were made available for students with disabilities and English-language learners. These students were allowed to take the test with the accommodations that they routinely received in their state or district assessments, as long as the accommodations were approved for use on NAEP. NAEP-approved accommodations for the 1996 administrations included extended time; individual or small group administration; a large-print version of the test; transcription, oral reading, or signing of directions; and use of bilingual dictionaries in mathematics. Final decisions about which accommodations to provide to students in S3 were made by school authorities. The criteria for the three samples are summarized in Box 2–1.

Analyses of the 1996 data revealed no differences in participation rates between the S1 and S2 samples. Thus, the S1 criteria were discontinued, and research was based on samples of schools that applied either the S2 or

the S3 criteria. The research continued with the 1998 national and state NAEP reading assessment and the 2000 assessments (mathematics and science at the national level in grades four, eight, and twelve and at the state level in grades four and eight; reading at the national level in grade four). The accommodations permitted were similar to those allowed in 1996, and a bilingual booklet was offered in mathematics at grades four and eight. Reading aloud passages or questions on the reading assessment was explicitly prohibited. Alternative language versions and bilingual glossaries were not permitted on the reading or science assessments. Findings from studies in 1996, 1998, and 2000 are described in detail in Chapter 6.

Based on the research findings and other considerations, NAGB passed the following resolution in 2001 (NAGB, 2001: pg. 43):

Through this policy NAGB adopted the criteria applied in the S3

sample as the official procedures (i.e., permitted accommodations will be provided to students who need them).

There are a number of unanswered questions about the comparability of scores from standard and nonstandard (accommodated) administrations and the effects of changes in inclusion policies on NAEP’s trend information. Although an accommodation is intended to correct for the disability, there is a risk that the accommodation over- or undercorrects in a way that further distorts a student’s performance and undermines validity. Thus, it cannot simply be assumed that scores from standard and nonstandard administrations are comparable. Adopting the procedures used for the S3 sample represents a significant change in NAEP’s inclusion policy, since special needs students who required accommodations were not included in the pre-1996 assessments. The change in inclusion policy could mean that results from the pre-1996 assessments are not comparable to results based on the inclusion policy used for S3 (National Institute of Statistical Sciences, 2000).

One of NAEP’s chief objectives is to provide information about trends in U.S. students’ educational achievement, but changes in policy regarding who participates in NAEP and how the test is administered can have an impact on the comparability of trend data. Carlson and Carr both emphasized that they hoped that the day’s discussions would provide them with a better understanding of the effects of accommodations on test performance and assist them as they work with others to formulate and refine NAEP’s reporting policies.