Designing a Market Basket for NAEP Summary of a Workshop (2000) / Chapter Skim
Currently Skimming:
4 The Perspectives of NAEP's Sponsors and Contractors
4 The Perspectives of NAEP's Sponsors and Contractors
Pages 19-29
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 19... ...
What is the status of research and development work on the market basket and short form? What are the Board's plans for pursuing work on the market basket/short form with regard to the 2000 assessment and beyond?
|
From page 20... ...
The short form would provide additional data collection opportunities to state-NAEP users that are not part of the standard NAEP schedule, such as testing in off years or in other subjects not assessed at the state level. Truby described how some people envision using a short form: If short forms were developed and kept secure, they could provide flexibility to states and any jurisdiction below the state level that were interested in using NAEP for surveying student achievement in subjects, grades, and times that were not part of the regular state-NAEP schedule.
|
From page 21... ...
This study involves preparation of NAEP short forms (scenario two in Figure 11. Details of the study are described below (see section entitled "NAEP's Year 2000 Market Basket Study: What Do We Expect to Learn?
|
From page 22... ...
Item response theory scales items according to the probability of a correct answer, given the proficiency level of the examinee and the item's discriminating power, difficulty, and susceptibility to examinee guessing. It relies on assumptions that, if met, result in proficiency estimates that, theoretically, are not dependent on the particular subset of items administered and that yield item parameter estimates that are relatively independent of the group of students taking the items.
|
From page 23... ...
Unlike the use of average percents correct in the early days of NAEP, the use of IRT-based projections of expected percent correct on a market basket of items enables prediction of performance on other items from the same framework that did not happen to be included in NAEP's assessment instrument. Kolstad believes that focus groups and empirical studies should be conducted to verify that the market-basket metric expected percent correct is indeed simpler for consumers to understand.
|
From page 24... ...
For instance, one arbitrary form may focus on multiple-choice items while another may primarily use constructed response items. Mislevy drew distinctions among three reporting metrics: the observed score metric, the true score metric, and the latent trait metric.
|
From page 25... ...
While this process involves a complicated transformation from observed scores, it has the advantage that, when IRT assumptions are met, the distributions are not content specific. Further, the latent trait distributions could be transformed to an expected percent correct metric.
|
From page 26... ...
, where a short form is released to states and districts for their use and scores are to be derived quickly and are intended to be comparable to main NAEP. In his paper, Mislevy systematically laid out the issues that need to be resolved before decisions are made on data gathering and data reporting models for the market basket.
|
From page 27... ...
According to Mazzeo, the test developers hope that the study will serve as a learning experience regarding the construction of alternate short forms. Whereas creating intact test forms is a standard part of most testing programs, this is not the case with NAEP.
|
From page 28... ...
One of the studies planned involves conducting separate analyses of the data using methods appropriate for arbitrary forms, methods appropriate for congeneric forms, and methods appropriate for parallel forms. Each of these sets of analyses will produce results in an observed score metric as well as a true score metric.
|
From page 29... ...
The year-2000 study will entail evaluation of the potential benefit of using longer market baskets. According to Mazzeo, the 31-item short forms were chosen out of consideration for school and student burden, increasing difficulties in obtaining school participation in NAEP, and the conviction that, "to be effective, a publicly released market basket of items should be of modest size." Other decisions regarding test length could also be made, such as Darrell Bock's domain score reporting approach.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.