Evaluation of the Voluntary National Tests, Year 2 Interim Report (1999) / Chapter Skim
Currently Skimming:
2 Item Quality and Readiness
2 Item Quality and Readiness
Pages 15-31
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 15... ...
Our process for evaluating potential VNT pilot test items was to identify samples of completed items anc3 ask both committee members anc3 aciclitional outside experts to rate the quality of these items. The evaluation involved two key questions: · Are the completed items judged to be as good as they can be prior to the collection and analysis of pilot test clata?
|
From page 16... ...
Finally, committee members anc3 a panel of aciclitional reacling anc3 mathematics assessment experts reviewed anc3 rated samples of 120 mathematics items anc3 90 reacling items. The committee's review of item quality slid not include separate consideration of potential ethnic or gender bias.
|
From page 17... ...
The committee will comment further on the item development process in our final report. ITEM REVIEW PROCESS Sampling Completed Items On the basis of revised item status information, we drew a revised sample of items, seeking to identify a sample that closely represented the content anc3 format requirements for an operational test
|
From page 18... ...
18 Ct o · _
|
From page 19... ...
19 o ~En ~Go ~ ~ Cal Go Cal ~ Go Go O En Go cat En Cal o oo o o o Cal ~ Cal tn ~ 0 tn o o o o o o ~ ~tn Go ~ ~ O cat ~ ~ En Go ~ cat ~ ~ Go Go ~Cal ~ ~ ~ ~ ~ tn cat tn 0 tn c ~oo ~ ~ ~ ~ C~ o oooo ~ ~ oooo oo ~ ~ oo ~ o tn oo ~ ~ ~ C~ o CD CD o o ._ ._ ~ ~ ~ v~ ~ ~ E ~ ~ .
|
From page 20... ...
Of course, these items JO not represent the balance of the current VNT items, as these items are still uncler development. Expert Panel Our overall conclusions about item quality are based primarily on ratings proviclec3 by panels of five math experts anc3 six reacling experts with a variety of backgrounds anc3 perspectives, including class' room teachers, test developers, anc3 disciplinary experts from academic institutions (Box 2~1~.
|
From page 21... ...
no ~ ~ · - at, 'D · - clS ¢ ~ z ct ~ ~ to to En cD ~ ct ~ .
|
From page 22... ...
Second, NAGB has been clear and consistent in its desire to make the VNT as much like NAEP as possible; NAEP items thus provide a very logical comparison sample, much more appropriate than items from other testing programs. We also note that NAEP items provide the basis for a fairly stringent comparison because they have been aciministerec3 to large samples of students, in contrast to the pre~pilot VNT items.
|
From page 23... ...
For mathematics, each booklet containec! three sets of common VNT items, targetec]
|
From page 24... ...
The second part of the rating task involved providing comments to document specific concerns about item quality or specific reasons that an item might be exemplary. Major comment categories were iclentifiec3 in the initial panel discussion, and specific cocles were assigned to each category to facilitate and standardize comment coding by the expert panelists.
|
From page 25... ...
Table 2~6 shows mean quality ratings for VNT anc3 for NAEP reacling anc3 math items anc3 also the percentages of items jucigec3 to have serious, minor, or no problems. Average ratings were 3.4 for VNT mathematics anc3 3.2 for VNT reacling items, both slightly below the 3.5 boundary between minor edits anc3 accept' able as is.
|
From page 26... ...
For both reacling anc3 math items, the most frequent comment overall anc3 particularly for items jucigec3 to require only minor edits was "clistractor quality," for both NAEP anc3 VNT items. In discussing their ratings, the panelists were clear that this code was used when one or possibly more of the incorrect (clistractor)
|
From page 27... ...
Matching VNT llems to the NAEP Achievement-Level Descriptions In the interim Phase ~ evaluation report (National Research Council, 1998:6) , the NRC recommended "that NAGB and its contractors consider efforts now to match candidate VNT items to the NAEP achievement-level descriptions to ensure adequate accuracy in reporting VNT results on the NAEP achievement-level scale." This recommendation was included in the interim report because it was viewed as desirable to consider this matching before final selection of items for inclusion in the pilot test.
|
From page 28... ...
. The main goal in matching VNT items to NAEP achievement levels is to have an adequate distribution of item difficulties to ensure measurement accuracy at key scale points.
|
From page 29... ...
CONCLUSIONS AND RECOMMENDATIONS With the data from item quality rating panels anc3 other information proviciec3 to the committee by NAGB anc3 AIR, the committee iclentifiec3 a number of specific findings about current item quality anc3 about the item development process for this interim report. Further evidence will be weighed as it becomes available anc3 reflected in our final report, to be issued in September.
|
From page 30... ...
As clescribec3 above, the committee and other experts reviewed a sample of items that were really or nearly really for pilot testing. Average quality ratings for these items were near the boundary between "needs minor edits" and "use as is" and were as high as or higher than ratings of samples of released NAEP items.
|
From page 31... ...
Small shortages in the number of pilot test items in some item content and format categories might be tolerated or even planned for in order to accommodate potentially greater rates of item problems in other categories. Recommendation 2.2: Specific issues identified by our item review, such as distractor quality, should be considered in further review of the VNT items by NAGB and its contractor.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.