Skip to main content

Currently Skimming:

4 VNT Pilot and Field Test Plans
Pages 35-44

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.


From page 35...
... . PILOT TEST PLANS Our second workshop, in April 1998, reviewed plans for conducting a pilot test of VNT items anc3 plans for subsequent field test of VNT test forms (see Appendix B for the list of participants)
From page 36...
... For mathematics, a total of 360 items will be included in the six operational forms built for the field test. The number of mathematics items to be piloted, i,080, is three times this number.
From page 37...
... In abolition to these "conventional" statistics, item response theory (IRT) parameters will be estimated for each item for use in pre- equating alternative forms anc3 for estimating test information functions that give the expected accuracy of a form at different score levels.1 A simple random sample of 800 students would lead to 95 percent confidence bouncis for proportions of less than .035.
From page 38...
... Estimates from the pilot test will not provide the basis for normative information to be reported to examiners, nor for the final equating of scores from alternative forms, each of which would be a significant use. Rather, item parameter estimates from the pilot test will be used to support construction of forms that are roughly equal in difficulty and accuracy so that form calibrations based on subsequent field test results will be feasible.
From page 39...
... If hybrid forms are not used, there is no overlap across forms: the only way to link the parameter estimates for different forms within a given school cluster is through the assumption that random assignment of forms to students eliminates the need for further ad justment. Item calibrations woulc3 be performed separately for each form, setting the underlying performance scale to have a mean of 0 anc3 a stanciarc3 deviation of 1 (or any other ciesirec3 constants)
From page 40...
... a. Summary and Conclusions Given the goal of assembling six operational forms from the pilot test item pool and current plans for review and revision of items prior to the pilot test, we find the number of items to be piloted to be appropriate, reflecting relatively conservative assumptions about item survival rates.
From page 41...
... If pilot test results lead to elimination of a larger than expected number of items, NAGB will have to consider back- up options for constructing and field testing test forms. Alternatives might include: reducing the number of new test forms to be incluciec3 in the first operational administration; relaxing the item screening criteria to allow greater use of statistically marginal items; finding aciclitional sources of items (e.g., NAEP or commercial item pools)
From page 42...
... NAGB anc3 its contractor have not yet publicly cleciclec3 whether IRT pattern scoring or observed total correct scores should be used as the basis for achievement- level classifications. Sample Size The current fielc3 test plan calls for sample sizes of i,000 for each intact anc3 i,000 for each hybrid version of the research anc3 public release forms anc3 the forms targeted for operational use in Years 2 anc3 3.
From page 43...
... Samples of 2,500 test takers per form are used to develop an initial equating of new forms of the Armed Services Vocational Aptitude Battery (ASVAB) , for instance.
From page 44...
... The accuracy with which students are assigned to achievement levels will clepenc3 directly and perhaps heavily on equating and linking accuracy, so further accuracy goals are needled. Summary and Conclusions The field test plans that we reviewed lacked sufficient rationales for several elements of the proposed design, including the use of hybrid forms, the use of equating clusters, and differential sample sizes for different forms.


This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More information on Chapter Skim is available.