Keeping Score for All The Effects of Inclusion and Accommodation Policies on Large-Scale Educational Assessments (2004) / Chapter Skim
Currently Skimming:
6 Articulating Validation Arguments
6 Articulating Validation Arguments
Pages 103-122
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 103... ...
To investigate the extent to which accommodations for students with disabilities and English language learners may affect the validity of inferences based on scores from NAEP and other assessments, one must begin with a close look at the validation arguments that underpin these scores in general. Research can most effectively investigate the effects of accommodations if it is based on a well articulated validation argument that explicitly specifies the claims underlying the assessments and the inferences the assessments are designed to support, and also if it explicitly specifies possible counterclaims and competing inferences.
|
From page 104... ...
Such an argument would provide a basis for conducting validation research that systematically investigates the effects of different accommodations on the performance of students with disabilities and English language learners. Such a validation argument would also inform assessment design and development, since the effects of different alterations in task characteristics and in test administration conditions caused by accommodations could be better understood using this approach.
|
From page 105... ...
(a visually impaired student taking the NAEP reading assessment) and one that the committee developed (an English language learner taking the NAEP reading assessment)
|
From page 106... ...
identified several key ancillary skills that are required to respond to NAEP items, as shown in Tables 6-1 and 6-2. These TABLE 6-1 Target and Ancillary Skills Required to Respond to Items on NAEP's Reading Assessment Knowledge and/or Skill Classification Comprehend written text Target skill Know vocabulary Ancillary skill Decode text Not specified as target skill Reading fluency Not specified as target skill See the item Ancillary skill Hear the directions Ancillary skill TABLE 6-2 Target and Ancillary Skills Required to Respond to Items on NAEP's Mathematics Assessment Knowledge and/or Skill Classification Mathematical reasoning Target skill Know content vocabulary Target skill Perform computations Target skill Comprehend written text Ancillary skill Know noncontent vocabulary Ancillary skill See the item Ancillary skill Hear the instructions Ancillary skill
|
From page 107... ...
In a reading assessment task, for example, it may be possible for a test-taker to answer some of the tasks correctly simply by guessing or as a consequence of prior familiarity with the reading passage or its subject matter, despite poor reading skills. In this case, these potential alternative explanations for good performance weaken the claim or inference about the test-taker's reading ability.
|
From page 108... ...
· Alternative explanations are rival hypotheses that might account for the observed performance on an assessment task. (Alternative explanations are often related to ancillary skills and knowledge.)
|
From page 109... ...
Validation Argument for Performance on NAEP Fourth Grade Reading Assessment Task. There are a number of possible alternative explanations for 1 This example is designed only to illustrate the principle; no such claim would actually be made on the basis of only a single test item.
|
From page 110... ...
Staying warm wasn't just a bedtime problem. On winter rides, colonial travelers covered themselves with animal skins and warm blankets.
|
From page 111... ...
[Aspect: Developing Interpretation Percent Full Comprehension: 37%]
|
From page 112... ...
is understood, and put this understanding to use. FIGURE 6-2 Validation argument for performance on NAEP fourth grade reading assessment task.
|
From page 113... ...
If decoding and fluency are considered ancillary skills, then the read-aloud accommodation would simply provide Sue with easier access to the passage and questions so she could more accurately demonstrate her reading comprehension skills. Example 2: Tina, an English Language Learner Taking a NAEP Reading Assessment A validity argument in support of inferences to be made on the basis of scores for the "Brick to Cuddle Up To" would work the same way for English language learners as for students with disabilities.
|
From page 114... ...
the task requires the test-taker to connect ideas from the text to his or her background knowledge and experiences. We have looked at the way in which an argument can support the claims or inferences an educational assessment was designed to support, but, as we have seen, there are a number of potential alternative explanations for any assessment results.2 Variations in either the attributes of test-takers or the characteristics of assessment tasks can in some cases account for variations in performance that are unrelated to what is actually being measured -- the variation among test-takers in what they actually know and can do.
|
From page 115... ...
Alternative Explanation inaT have background knowledge required successfully complete task. Rebuttal Tina native American and experienced winter soapstones, other experiences are the reading to the as fourth inaT the or the and in grade reading for and from such fect and ef and and The was ideas her fourth level and 2: a requested reasonable recognize unless with is .
|
From page 116... ...
The fourth grade NAEP reading task discussed earlier can be used to demonstrate how one or more alternative explanations might work and the nature of the rebuttal data that might support them. Suppose Tina is an English language learner whose family's cultural heritage is not North American and who is not familiar with some of the concepts presented in the passage, such as winter, fireplaces, and soapstones.
|
From page 117... ...
To demonstrate how this type of information can be used in developing a validation argument, the same example can be tied to NAEP's documentation of the content and skills targeted by its fourth grade reading assessment. The NAEP Reading Framework explicitly recognizes that background knowledge is a factor in test performance but describes this factor as one that contributes to item difficulty.
|
From page 118... ...
118 are as or they ignore focus The s score NAEP to scoring 3: language to such and 3: ina'T errors features low sometimes errors incorrect scorers tasks. a by or when responses and of when Data are trained NAEP written answers.
|
From page 119... ...
present a framework of task characteristics that may be useful in describing the ways in which the characteristics and conditions of assessment tasks are altered for the purpose of accommodating students with disabilities and English language learners. This framework includes characteristics of the setting, the rubric, the input, and the expected response, as well as the relationships between input and response.
|
From page 120... ...
To weaken the second alternative explanation, that Tina's poor performance on reading assessment is the result of poor writing skills (which are not being assessed) , either or both of two characteristics of the expected response -- language or mode of response -- could be altered.
|
From page 121... ...
We have explained that a variety of test-takers' characteristics, such as disabilities, insufficient proficiency in English, or lack of cultural knowledge, can constitute alternative explanations for their performance on assessments. We have also discussed the ways specific accommodations can be described in terms of specific aspects of the assessment tasks and administration procedures.
|
From page 122... ...
This research should apply a variety of approaches and types of evidence, such as analyses of test content, test-takers' cognitive processes, criterion-related evidence, and other studies deemed appropriate. RECOMMENDATION 6-3: NAEP officials should conduct empirical research to specifically evaluate the extent to which the validation argument that underlies each NAEP assessment and the inferences the assessment was designed to support are affected by the use of particular accommodations.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.