Evaluation of the Voluntary National Tests Phase 1 (1999) / Chapter Skim
Currently Skimming:
3 Item Development and Review
3 Item Development and Review
Pages 17-34
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 17... ...
Table 3~1 shows the number of items written, in comparison to the number neeciec3 for the pilot anc3 field tests. Current plans for the pilot test require creating 24 forms containing 45 reacling items each anc3 18 forms containing 60 mathematics items each a combined total of 2,160 17
|
From page 18... ...
For the VNT, these figures would be comparable to the combined survival rates from item review and from the pilot test. While we do not have industry information on survival rates through the review process alone, data presented later in this chapter indicate that overall survival rates from the different reviews of the VNT items are running well above the 67 percent needed to meet pilot test requirements.
|
From page 19... ...
Our analysis suggests that if the current item classifications hold up, aciclitional mathematics items will be neeciec3 for the content strands in algebra and functions and probably also in geometry and spatial sense. Only 264 items are currently classified as algebra and functions items; 270 will be neeciec3 for the pilot test.
|
From page 20... ...
Data Analysis, Statistics, and Probability Multiple choice 6108115 94 Gridded response 11839 46 Short constructed response 23643 84 Extended constructed response 007 0 Total 9162204 79 E Algebra and Functions Multiple choice 11198189 105 Gridded response 0014 0 Short constructed response 23653 68 Extended constructed response 1187 257 Total 14252263 96 Total Multiple choice 42756953 79 Gridded response 6108164 66 Short constructed response 10180274 66 Extended constructed response 23655 66 Total 601,0801,446 75 aThe current specifications do not include drawn response items, so the 20 drawn response items developed have been excluded from this table.
|
From page 21... ...
Given the speec3 with which a very large bank of items has already been cievelopec3, there should not be any insurmountable problem in creating the aciclitional items neeciec3 to meet pilot test requirements in each item category. Since the items have not yet been mapped to the NAEP achievement levels, however, we have no basis for determining whether the current distribution of items appropriately reflects the knowledge and skills described for each level or whether a significant number of aciclitional items might be needled to ensure adequate coverage at all levels.
|
From page 22... ...
22 EVALUATION OF THE VOLUNTARY NATIONAL TESTS TABLE 3~5 Number of Reacling Items by AIR Passage Type by Format Plan Number Developed NumberNumber Neededas of Minimum Needed Type and Formatper Formfor Pilot Test7/15/98 to Retain: Percent Short Literary Multiple choice5-6120-144126 105 Short constructed response12435 69 Extended constructed responseOOO O Total6-7144-168161 97 Medium Literary Multiple choice5120176 68 Short constructed response1-224-4856 64 Extended constructed responseOOO O Total6-7144-168232 67 Long Literary Multiple choice8-9192-216399 51 Short constructed response1-224-48104 35 Extended constructed response12441 59 Total10-12240-288544 49 Short Informational Multiple choice4-596-120192 56 Short constructed response12457 42 Extended constructed response002 0 Total5-6120-144251 53 Medium Informational Multiple choice9-10216-240332 69 Short constructed response12466 36 Extended constructed response12444 55 Total1 1-12264-288442 62 Intertextuala Multiple choice24875 64 Short constructed response12439 62 Total396114 84 Total Multiple choice33-378401,300 65 Short constructed response7-9192357 54 Extended constructed response24887 55 Total451,0801,744 62 aItems that require students to answer questions based on their reading of two passages pertaining to the same or similar topics.
|
From page 23... ...
The specifications for bias anc3 sensitivity review of mathematics items are identical, except there is no need for NAGB preclearance of reacling passages. At the present time, final NAGB review of the reacling anc3 mathematics items has not yet taken place, anc3 in advance of the pilot test it is not possible to carry out the DIF (differential item functioning)
|
From page 24... ...
NAGB would then have until its November meeting to review the item pool anc3 approve a sufficient number as cancliciates for pilot testing. As ciescribec3 below, these plans were changed by NAGB in light of our interim report in July (National Research Council, 1998)
|
From page 25... ...
25 to , o ~ Q 03 Q'A (n m a' TIC 0 Z Q ~ ~ S E 1 ~P c E I CD ~ ~ ~ ~ ~ C ~ ~ ~ C ~ ~ Al Al Al ~0 2 0 0 0 o At 0 0 a, 'en at ~ Cal In ~ Cal Cal E ~ ~ ~ ~I ~ C0Z ...................................
|
From page 26... ...
The sample of items we selected matched the VNT specifications for the length of a test form as well as for content coverage anc3 item formats. The experts examined anc3 rated a subset of secure test items in their area of expertise.3 Because the items we examined in June were products of the initial stages of item development, we clic3 not expect them to reflect the complete development process.
|
From page 27... ...
First, it is common to find problems with a significant number of items early in the item development process: Why conduct rigorous item reviews if not to weed out items that do not pass muster? The VNT developers, in fact, expected that 15 percent of the items would be eliminated from consideration even before pilot testing, and they also expected that only one- third or one- fourth of the piloted items would be used in the initial forms to be field tested for possible operational use.
|
From page 28... ...
The period of time allocated for NAGB's review of item materials might be reduced correspondingly, to allow for full and complete attention to item revision and quality assurance by the test development contractors. · We recommend that NAGB and its contractors consider efforts now to match candidate VNT items to the NAEP achievement level descriptions to ensure adequate accuracy in reporting VNT results on the NAEP achievement level scale.
|
From page 29... ...
In 4The items included 93 extended constructed response items, 289 short constructed response items, 49 "ridded, 8 drawn, and 145 multiple choice items (American Institutes for Research, Cognitive Lab Report, July 29, 1998~. 5The description of the cognitive labs is based on AIR's procedural report of July 29, 1998.
|
From page 30... ...
Items were grouped in packets of about 14 mathematics items or 10 reacling items. Each item was analyzed, anc3 a written protocol, describing potential paths to correct or incorrect responses.
|
From page 31... ...
We offer the following assessment of the cognitive labs: ~ ~ ~ The promise of cognitive interviewing as a tool for item development is that it will acid to the information obtained from stanciarc3 content anc3 bias review procedures before items are pilot or field tested. Because of the compressed time schedule, however, items were introclucec3 into the cognitive lab sessions in relatively unrefined form, before the other review anc3 revision processes had been completed.
|
From page 32... ...
It should be possible to organize anc3 schedule this activity to provide a cost~benefit analysis of stanciarc3 item review anc3 revision procedures in comparison with processes that include cognitive laboratories. ONGOING REVIEW PROCESS To assess progress in revising anc3 editing VNT items, we visited AIR on July 28 anc3 reviewed file information for a sample of reacling anc3 mathematics items.
|
From page 33... ...
The final stratum consisted of another 30 mathematics items anc3 20 reacling items selected from items that hac3 been aciclec3 to the item bank after our June workshop.6 In all, we sought information on the status of 90 mathematics items anc3 60 reacling items. For each sampled item, we recorciec3 whether a folder was located; if so, the number of item reviews clocumentec3 in the folder; the summary recommendation (accept, revise, drop)
|
From page 34... ...
3 2. The developers should improve and automate procedures for tracking items as they progress through the development cycle so as to provide timely warnings when additional items will be needed and historical information on item survival rates for use in planning future item develop ment cycles.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.