The National Academies Press

Currently Skimming:

Pages 72-75

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.

From page 72... ... ," Williamson outlined several group differences to examine: differences in the associations between automated and human scores across groups at the task, task type, and reported score levels; differences in the generalizability of automated scores by group; differences in the predictive ability of automated scoring; and difference in relation to the decisions made based on the scores.17 In evaluating fairness, it is also important to examine whether humans are introducing bias and, if so, to introduce methods to correct the bias, such as improved training and monitoring. Finally, while the concepts of machine learning and automated scoring are becoming increasingly familiar to the public, there is still considerable skepticism. Read the entire page →
From page 73... ... Currently, 40 to 50 percent of the reading items and 25 percent of the mathematics items are hand scored.19 NCES estimates that automated scoring can be used for 70 percent of the hand-scored reading items and 40 percent of the hand-scored mathematics items.20 These figures are being empirically tested in the open challenge that is being conducted as this report is finalized and will be examined in future research studies.21 For the items that use automated scoring, hand scoring will continue for about 5 to 10 percent of responses to monitor the performance of automated scoring.22 NCES estimates that automated scoring will cut the cost of hand scoring in half for the reading and mathematics assessments in grades 4 and 8 starting in fiscal 2024,23 which would save approximately $2.5 million in scoring costs every 2 years. This reduction of $1.25 million in the annual average scoring cost represents 0.7 percent of NAEP's budget. Read the entire page →
From page 74... ... RECOMMENDATION 7-1: The National Center for Education Statis tics (NCES) should continue its work to implement automated scoring on the reading and mathematics assessments for grades 4 and 8, with the item types that current scoring engines can score accurately and consistently. Read the entire page →
From page 75... ... NCES response to Q33 says that the design, analysis, and re porting contract includes the following activities: "designs all pilot and field tests, operational assessments, and special studies; analyzes data ensuring reporting of valid results; proposes and prepares psychometric and statistical analyses compatible with previous NAEP methodologies; specifies data needed to meet the goals for reporting; and prepares reports." 2 NCES response to Q32. Read the entire page →

From page 72...

... ," Williamson outlined several group differences to examine: differences in the associations between automated and human scores across groups at the task, task type, and reported score levels; differences in the generalizability of automated scores by group; differences in the predictive ability of automated scoring; and difference in relation to the decisions made based on the scores.17 In evaluating fairness, it is also important to examine whether humans are introducing bias and, if so, to introduce methods to correct the bias, such as improved training and monitoring. Finally, while the concepts of machine learning and automated scoring are becoming increasingly familiar to the public, there is still considerable skepticism.

Read the entire page →

From page 73...

... Currently, 40 to 50 percent of the reading items and 25 percent of the mathematics items are hand scored.19 NCES estimates that automated scoring can be used for 70 percent of the hand-scored reading items and 40 percent of the hand-scored mathematics items.20 These figures are being empirically tested in the open challenge that is being conducted as this report is finalized and will be examined in future research studies.21 For the items that use automated scoring, hand scoring will continue for about 5 to 10 percent of responses to monitor the performance of automated scoring.22 NCES estimates that automated scoring will cut the cost of hand scoring in half for the reading and mathematics assessments in grades 4 and 8 starting in fiscal 2024,23 which would save approximately $2.5 million in scoring costs every 2 years. This reduction of $1.25 million in the annual average scoring cost represents 0.7 percent of NAEP's budget.

Read the entire page →

From page 74...

... RECOMMENDATION 7-1: The National Center for Education Statis tics (NCES) should continue its work to implement automated scoring on the reading and mathematics assessments for grades 4 and 8, with the item types that current scoring engines can score accurately and consistently.

Read the entire page →

From page 75...

... NCES response to Q33 says that the design, analysis, and re porting contract includes the following activities: "designs all pilot and field tests, operational assessments, and special studies; analyzes data ensuring reporting of valid results; proposes and prepares psychometric and statistical analyses compatible with previous NAEP methodologies; specifies data needed to meet the goals for reporting; and prepares reports." 2 NCES response to Q32.

Read the entire page →

This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More information on Chapter Skim is available.