A Pragmatic Future for NAEP Containing Costs and Updating Technologies (2022) / Chapter Skim
Currently Skimming:
Pages 38-47
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 38... ...
In addition, sometimes extra items need to be developed, which can be required, for example, when a new framework requires a new type of item or area or content that was not previously covered.6 Over the next few years, a somewhat higher proportion of new items may be required, if the items in long-term trend NAEP are updated in its transition to digital administration and if the scheduled framework updates the three types of items (roughly the midpoints of the three ranges) produces the weighted average item creation and review cost of $3,700.
|
From page 39... ...
This point was added after a prepublication version of the report was provided to the Institute of Education Sciences, NCES, and NAGB. The correction altered the estimates of item creation and pilot administration costs, as well as the estimates of potential savings to administration costs from local administration and longer testing time.
|
From page 40... ...
Although NAEP includes some traditional selected-response items for which automatic item generation might be applied, those items are more prevalent in long-term trend NAEP, where new items are not generally created. Main NAEP, where new items are needed, often uses more complex item types, which are less amenable to automatic item generation.
|
From page 41... ...
found that the investment in automatic item generation could be worthwhile within a narrow content area if more than 173–247 items were needed. This number of items is feasible for high-stakes assessments, such as admissions tests, that are administered frequently, and where reducing the exposure of items is necessary for security purposes.
|
From page 42... ...
NAEP's use of automated processes of item generation could then evolve as the state of the art in automatic item generation evolves. RECOMMENDATION 4-2: The National Assessment Governing Board and the National Center for Education Statistics should move toward using more structured processes for item development to both 16 For examples of the detailed achievement-level descriptions, see, for mathematics, https:// nces.ed.gov/nationsreportcard/mathematics/achieve.aspx; for science, https://nces.ed.gov/ nationsreportcard/science/achieve.aspx; and for reading, https://nces.ed.gov/nationsreportcard/ reading/achieve.aspx.
|
From page 43... ...
The choice of item types is also influenced by factors other than the cognitive and content areas to be assessed, such as testing time and development, administration, and scoring costs. Changing the mix of item types could potentially change NAEP's average costs for item creation, pilot testing, test administration, and scoring.
|
From page 44... ...
The costs considered should include item development (both item creation and pilot administration) , administra tion time, and scoring.
|
From page 45... ...
It then outlines the program's new vision for test administration, followed by a description of the experience with local administration during the era of voluntary state participation in NAEP (prior to the No Child Left Behind Act of 2001)
|
From page 46... ...
. In addition, roughly 23 percent of the $21.9 million average annual cost for pilot testing is supported by the sampling and data collection contract for administration of the pilot test, representing another $5.0 million in administration costs each year.4 VISION FOR A DEVICE-AGNOSTIC, CONTACTLESS NAEP The current administration model for NAEP, which uses professionally trained NAEP staff and contractors to administer the assessment, minimizes the participation burden for local schools and helps ensure quality, 2 NCES response to Q33: The sampling and data collection contract covers the following activities: "Selects samples; prepares sampling weights; administers assessments and collects data for pilot and field tests, operational assessments, and special studies; and ships completed assessment materials to the scoring sites.
|
From page 47... ...
NCES recognizes that it may have to provide equipment and proctors to some schools for a number of years and that the exact timing that will be feasible for all schools for this transition is uncertain. LOCAL ADMINISTRATION IN THE PAPER-BASED ERA Although NCES's plans for local test administration represent a change from NAEP's current approach, the program had extensive experience with local administration during the 1990s.6 Prior to 2002, local education employees proctored NAEP assessments in the trial state assessment portion of the program.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.