A Pragmatic Future for NAEP Containing Costs and Updating Technologies (2022) / Chapter Skim
Currently Skimming:
7 Item Scoring
7 Item Scoring
Pages 67-74
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 67... ...
The scoring and dissemination contract, also referred to as the materials, distribution, processing, and scoring contract, includes the following activities: "Prepares and packages all assessment and auxiliary materials; distributes assessment booklets and materials to the test administrators for each school; receives the materials from the schools; with [item development] and [design, analysis and reporting]
|
From page 68... ...
5 NCES response to Q69e. The six contract activities related to management and reporting are listed as follows: administrative reports; quality control; contractor meetings; information collections requests for Office of Management and Budget approval; technical documentation web page; and NAGB attendance, preparation, and support.
|
From page 69... ...
. The incorporation of automated scoring into NAEP offers a number of likely benefits, including faster scoring, improved score consistency within and across administrations, higher-quality scoring of items when combined with human scoring, increased information about student responses, and potentially cost savings.
|
From page 70... ...
Careful planning related to technical oversight, public acceptance, and validation of its effects would be critical to the successful implementation of automated scoring. NAEP, given its national significance, is uniquely suited to leverage industry and academic expertise to lead the United States as an exemplar in how to incorporate automated scoring into an assessment program.
|
From page 71... ...
Items in the mandated reading and mathematics assessments with state and urban district samples are at the top of the range of response counts: items used in assessments with national samples are at the bottom of this range. In most implementations, automated-scoring models are trained for every item, and so increasing items increases costs.15 The NAEP response counts per item are near the threshold for achieving cost savings from automated scoring, which is typically around 30,000 responses; it depends on the cost savings from hand scoring and the overall number of items automatically scored.
|
From page 72... ...
, rate of agreement between automated scores and human scores at the item level and the test level, and other measures of quality and to determine whether they vary with the training data used. These comparisons will need to be conducted for the full group of test takers and for test takers grouped by race and ethnicity, gender, English-learner status, disability status, family socioeconomic status, and other characteristics of interest.16 Fairness is a particularly important issue to consider in evaluations, given that research has documented disparities related to machine learning and automated scoring (Corbett-Davies and Goel, 2018; Hutchinson and Mitchell, 2019)
|
From page 73... ...
The NCES response to Q78 provides further detail about this work: The proof of concept will cost $80,000 and will "evaluate the use of automated scoring to score 2017 release NAEP grade 4 and 8 reading items." A field test for $1–1.5 million will carry out a "duplicate ‘Shadow Score' of 2019 NAEP Math & Reading items" using "the entire corpus of 285 constructed response mathematics and reading items." In addition, NCES referred to ongoing special studies involving human double scoring ($400,000–600,000 each)
|
From page 74... ...
RECOMMENDATION 7-1: The National Center for Education Statis tics (NCES) should continue its work to implement automated scoring on the reading and mathematics assessments for grades 4 and 8, with the item types that current scoring engines can score accurately and consistently.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.