A Pragmatic Future for NAEP Containing Costs and Updating Technologies (2022) / Chapter Skim
Currently Skimming:
11 Summary: A New Path for NAEP
11 Summary: A New Path for NAEP
Pages 93-108
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 93... ...
The work of this panel is to contribute to that exploration by providing analysis and recommendations for the next phase of NAEP. This chapter summarizes our recommendations: taken together, they chart an ambitious yet practical way for the National Center for Education Statistics and the National Assessment Governing Board to plan for NAEP's future.
|
From page 94... ...
Congress, the National Assessment Governing Board, and the National Center for Education Statistics should then consider the value of a modernized and continued long-term trend NAEP in comparison with other program priorities. If continued, long-term trend NAEP should be renamed to better distinguish it from the trend data provided by main NAEP.
|
From page 95... ...
The downside of not actively considering assessments for combined subjects is illustrated by the cost pressures that force some subjects to be assessed infrequently, such as writing, or to have been effectively eliminated, such as economics and geography. RECOMMENDATION 3-3: The National Assessment Governing Board should give high priority to consideration of integrating non mandated subjects that are currently assessed separately (such as sci ence and technology and engineering literacy)
|
From page 96... ...
Although NAEP includes some of these types of items, their numbers are insufficient to justify the costs of implementing automatic item generation. In addition, it is difficult to use automatic item generation for the scenario-based and other complex item types that predominate in main NAEP.
|
From page 97... ...
Changing the mix of item types could potentially change NAEP's average costs for item creation, pilot testing, test administration, and scoring. The average costs of the three item types are $1,750 for selected-response items, $2,500 for constructed-response items, and $13,000 for items that are part of scenario-based tasks.
|
From page 98... ...
MODERNIZING NAEP ADMINISTRATION The current NAEP administration model relies on professionally trained NAEP staff and contractors who travel to schools to administer the assessment. When NAEP transitioned to digitally based assessment, NCES provided the technology needed for students to test.
|
From page 99... ...
NCES has recently estimated the cost savings associated with this change of $56 million from 2026 to 2030. The panel's analysis suggests that the potential savings may be substantially larger, perhaps as large as an annual average savings of roughly $30.8 million, or 18.7 percent of NAEP's current budget.
|
From page 100... ...
Reducing sample sizes is one way to reduce costs, but it needs to be done in a way that does not degrade the quality of valued comparisons and trends. Procedures called statistical power analyses are used to estimate the sample size needed to detect performance differences that are judged to be policy relevant, and they can guide NAEP in its decision making about reductions.
|
From page 101... ...
However, the practical problems can be addressed in multistage adaptive testing, in which the adaptation occurs over groups of items and the first stage is limited to items that can be automatically scored, though this may prevent the use of some item types and may omit consideration of some subscales in the adaptation. The coarse adaptation that is possible is unlikely to result in substantial efficiencies across the full population, but it could improve estimates for some groups.
|
From page 102... ...
RECOMMENDATION 6-4: Efforts to coordinate NAEP test admin istration with the international assessment programs sponsored by the National Center for Education Statistics should not be used as a strategy to reduce costs. USING AUTOMATED ITEM SCORING Automated scoring offers the potential to modestly reduce the cost of hand scoring NAEP's constructed-response items, with an estimated annual savings of about $1.25 million per year, which is 0.7 percent of NAEP's budget. Automated scoring is the use of statistical and computational methods to model scores assigned by human raters.
|
From page 103... ...
However, automated scoring models require human monitoring to examine performance, and models may need recalibration. Automated scoring also offers the potential for collecting additional diagnostic information about student responses beyond a score, with data about spelling, coherence, syntactic variation, and other linguistic features, providing more insight about student knowledge and skills.
|
From page 104... ...
To support the full range of innovations -- those that should be implemented now and those that will become compelling in the next decade or even further in the future -- NAEP needs a robust technology platform that is flexible enough to incorporate innovations as they become ready for application. These innovations span the full chain of the NAEP program, including test design, item and test development, test administration, analysis of results, and reporting.
|
From page 105... ...
should ensure that there is adequate internal and external expertise related to enterprise software development to support and oversee the development of Next-Gen eNAEP for both the NCES staff and the staff working for the platform development contractor. This software expertise is substantially different than expertise related to psychometrics and statistics.
|
From page 106... ...
RECOMMENDATION 10-1: The National Assessment Governing Board and the National Center for Education Statistics should commis sion an independent audit of the program management and decision making processes and costs in the NAEP program, with a charge and sufficient access to review the program's costs in detail. That audit should include proposed ways to streamline these processes.
|
From page 107... ...
SUMMARY: A NEW PATH FOR NAEP 107 technology and be mindful of the costs of its past practices and upcoming decisions. The analysis and recommendations in this report are offered as a way for NAEP to evolve to serve its important purposes for policy makers and the public well into the 2030s.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.