Skip to main content

Currently Skimming:

3 Technical Issues in Test Development
Pages 32-42

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.


From page 32...
... · Designs and Item Calibration Plan for the 1999 Pilot Test (American Institutes for Research, 1998f) · Designs and Item Calibration Plans for Including NAEP Item Blocks in the 1999 Pilot Test of the VNT (American Institutes for Research, 1998e)
From page 33...
... (American Institutes for Research, 1999g) · Selected Item Response Theory Scoring Options for Estimating Trait Values (American Institutes for Research, 1999h)
From page 34...
... Reporting of confidence band information will allow parents to see the precision with which their children are placed into the various achievement levels without having to grapple with classification error probabilities. We believe that parents will finc3 information about measurement uncertainty more useful anc3 unclerstanciable if it is reported by means of confidence bands rather than as probabilistic statements about the achievement levels.
From page 35...
... It is likely, for example, that the final VNT accuracy targets will not yield the same amount of information at each achievement level. PILOT TEST PLANS Forms Design Key features of the pilot test forms design are the use of school clusters, the use of hybrid forms, NAEP anchor forms, and item calibration procedures.
From page 36...
... Another advantage of the hybrid design is that it will allow intact NAEP blocks to be combined with VNT half- test blocks, which will provide a basis for comparing VNT anc3 NAEP item c3 ifficulties anc3 putting the VNT item parameters on the NAEP scale. To the extent that the NAEP blocks cover the content domain, it also will allow an assessment of the extent to which the VNT anc3 NAEP measure the same areas of knowledge.
From page 37...
... We strongly support the inclusion of NAEP blocks in the pilot test design to provide data on the feasibility of a common calibration of VNT and NAEP items as a means of linking the two scales. item Calibration Item calibration refers to the procedures used for estimating item parameters or characteristics of items such as difficulty level.
From page 38...
... Constructed- response items may need to be piloted at a higher rate than multiplechoice items in order to produce sufficient items for the field test forms. The materials we reviewed slid not specify the expected survival rates for the various item types nor slid they discuss the rationale for determining item production or item survival rates.
From page 39...
... The contractor has proposed to use the Mantel- Hanszel method for the pilot test data anc3 methods based on item response theory for the field test data. The sampling plan will allow for comparisons based on race/ ethnicity (African Americans anc3 whites, Hispanics anc3 whites)
From page 40...
... 40 ~ - ~ j/ l l coca - icy // ~ - ice / ~ - hi ooze / : / : / \ ·~ / \ .~.
From page 41...
... Recommendation 3.1 Given that test items and answer sheets will be provided to students, parents, and teachers, test forms should be designed to support scoring using a straightfor ward, total correct raw score approach. Recommendation 3.2 Achievement level reporting should be supplemented with reporting using a standardized numeric scale, and confidence bands on this scale, rather than probabi fistic statements, should be provided regarding the likelihood of classification errors.
From page 42...
... Recommendation 3.6 A target test information function should be decided on and set. Although accuracy at all levels is important, accuracy at the lower boundaries of the basic and proficient levels appears most critical.


This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More information on Chapter Skim is available.