Keeping Score for All The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments (2004) / Chapter Skim
Currently Skimming:
4 Factors That Affect the Accuracy of NAEP's Estimates of Achievement
4 Factors That Affect the Accuracy of NAEP's Estimates of Achievement
Pages 62-84
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 62... ...
We explore the variability in state policies for identifying students with disabilities and English language learners and the variability in state policies regarding allowable accommodations on state assessments, and we consider ways in which local decision making affects the integrity of NAEP samples and its results.
|
From page 63... ...
However, in the case of both students with disabilities and English language learners, which students ultimately remain in the sample depends in part on decisions made at the local level. These decisions are discussed in greater detail below.
|
From page 64... ...
First, as they are developing the sample, NAEP officials make no attempt to identify students with disabilities or English language learners themselves; rather, the percentages of those students who end up in the sample reflect decisions that have already been made at the school level; these decisions are guided by state policies, which vary. Second, NAEP officials leave it to school-level staff, who are knowledgeable about students' educational functioning levels, to determine whether selected students who have a disability or are English language learners can meaningfully participate.
|
From page 65... ...
Finally, a key objective for NAEP is to characterize the achievement of the school-age population in the United States, yet the extent to which NAEP results are representative of the entire population depends on the locally made decisions that affect the samples. Identifying and Classifying Students with Disabilities and English Language Learners Determining which students should be classified as disabled in some way or as an English language learner is thus critical to ensuring that these groups of students are adequately represented, but making these classifications is far more complicated than many people recognize.
|
From page 66... ...
For example, financial pressures on school districts and a lack of adequate federal and state support may make local officials reluctant to refer students for special education services even when they seem to meet relevant eligibility criteria (National Research Council, 1997a, p.
|
From page 67... ...
Hence the category includes a broad range of students whose level of fluency in English, literacy in their native language, previous academic experiences, and socioeconomic status all vary significantly. Below we present results from several analyses of state policies with regard to identification of English language learners.
|
From page 68... ...
. In addition, some states base the identification on information gathered from enrollment records, home language surveys, interviews, observations, and teacher referrals, while others identify students as English language learners from their performance on tests designed to measure "English proficiency" (National Research Council, 2000b)
|
From page 69... ...
Jurisdictions typically do not track English language learners' progress once they stop receiving educational supports, although they may be far from fluent. Moreover, students who are no longer identified as needing educational supports would not ordinarily receive testing accommodations either.
|
From page 70... ...
The extent to which state policies will continue to vary remains to be seen. Policies on Accommodation NAEP results are also affected by the ways in which students with disabilities and English language learners are accommodated when they participate in NAEP.
|
From page 71... ...
The No Child Left Behind Act has meant that far fewer English language learners can be excluded from assessments, and that accommodations and alternate assessments will be used for many students who might formerly have been excluded. The variation in policies for accommodating these students around the country is similar to that evident for students with disabilities.
|
From page 72... ...
Signing of directions 48 Noa Oral reading of directions 48 Not specified Audio taped directions or questions 29 No Repeating of directions 47 Yes Explanation of directions 38 Yes Interpretation of directions 28 Not specified Short segment testing booklets 14 Not specified Equipment: Use of magnifying glass 47 Not specified Amplification No info Not specified Light/acoustics No info Yes Calculator No info Only on calculator use Templates to reduce visual field 38 Not specified Response Format: Use of scribe 48 Yes Write in test booklet 44 Yes Use template for recording answers 29 Not specified Point to response, answer orally 41 Yes Use sign language 42 Noa Use typewriter/computer/word processor 41 Yes Use of Braille writer 42 Yes Answers recorded on audio tape 32 No Scheduling/Timing: Extended time 46 Yes More breaks 46 Yes Extending sessions over multiple days 37 No Altered time of day that test is given 41 Not specified Setting: Individual administration 47 Yes Small group 47 Yes Separate room 47 Yes Alone in study carrel 43 Yes At home with supervision 27 Not specified In special education class 46 Not specified Other: Out of level testing 15 No Use of word lists or dictionaries 25 No Spell checker 16 No aNot provided by NAEP, but school, district, or state may provide after fulfilling NAEP security requirements. SOURCES: Annual Survey of State Student Assessment Programs 2000-2001 (Council of Chief State School Officers, 2002)
|
From page 73... ...
Use of technology 13 Not specified SOURCES: Annual Survey of State Student Assessment Programs 2000-2001 (Council of Chief State School Officers, 2002) ; Available: http://nces.ed.gov/nationsreportcard/about/inclusion.asp# accom_table.
|
From page 74... ...
As has been noted, there is currently no legislation parallel to IDEA to guide decision making about accommodations for English language learners. When a student in either group requires an accommodation that is not on the approved list for NAEP, the student is generally excluded from the assessment.
|
From page 75... ...
First, students with disabilities may be excluded because schools exclusively devoted to special education students are not included in the sampling. Second, students with disabilities and English language learners may be excluded because the test is not administered to students who, in the judgment of school personnel, cannot meaningfully participate.3 3That is, students with disabilities who would require an accommodation that is not allowed on NAEP or an alternate assessment, as well as English language learners who do not meet NAEP's rules for inclusion.
|
From page 76... ...
As we have seen, the decisions made by school personnel in identifying students as having disabilities or being English language learners vary both within states and across states, but there is no way to measure this variance or its effect on the sample. Nevertheless, it is very likely that students are excluded from NAEP according to criteria that are not uniform.
|
From page 77... ...
The number of students with disabilities (column 1, row 1) was obtained from the 24th Annual Report to TABLE 4-4 Comparisons of the Percentages of Students with Disabilities and English Language Learners in the United Statesa for the 2000-2001 School Year with Those in the NAEP Samples for 2002 Reading and 2003 Mathematics (1)
|
From page 78... ...
The national demographics on students with disabilities and English language learners include counts of students for outlying areas (American Samoa, Guam, Northern Marianas, Puerto Rico, Virgin Islands) , which are not all always included in the NAEP sample, and the way in which the data were reported would not allow disentangling these numbers for all of the columns on this table.5 Furthermore, the differences in the estimated proportions of students with disabilities and English language learners sampled in NAEP and existing in the United States could be attributable to differences in the way students are counted, the way the data are reported, or both; neither source for the estimated proportions should be considered infallible.
|
From page 79... ...
We further note that while national data may be useful in evaluating the representativeness of national NAEP results, state-level demographics would be needed to evaluate the representativeness of the state NAEP results. Tables 3-4 and 3-5 presented data by state on the percentages of students with disabilities and English language learners, respectively, who participated in NAEP's 1998 and 2002 reading assessments.
|
From page 80... ...
80 a Year and NAEP in School 11 14 9 13 9 10 11 8 11 12 10 13 10 12 11 14 11 (9) Percentage (Identified)
|
From page 81... ...
81 continued 9 11 12 10 14 8 11 12 10 11 10 15 14 18 12 12 13 8 14 12 13 17 8 9 9 9 15 4 (13)
|
From page 82... ...
82 a and NAEP Education in of 9 11 13 13 14 ent (9) Percentage (Identified)
|
From page 83... ...
The integrity of the samples is of paramount importance for data regarding students with disabilities and English language learners, but it also affects the validity of NAEP data about the population as a whole and other subgroups as well. At the same time there exists the possibility that greater attention to the data regarding students with disabilities and English language learners provided by NAEP, and the factors that complicate interpretation of these data, may raise difficult questions for NAEP.
|
From page 84... ...
This definition should serve as a guide for decision making and the formulation of regulations regarding inclusion, exclusion, and reporting. RECOMMENDATION 4-4: NAEP officials should evaluate the extent to which their estimates of the percentages of students with disabilities and English language learners in a state are comparable to similar data collected and reported by states to the extent feasible given the data that are available.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.