11

Summary: A New Path for NAEP

NAEP is unique in the information it provides, but it is also very expensive and increasingly so. NAEP assessments are roughly twice as expensive as the assessments of the Program for International Student Assessment (PISA) and a full order of magnitude more expensive than almost all high-stakes state assessments. While the increases in NAEP’s costs have been accompanied by important expansions in the information made available to users, the program’s high cost raises concern about its long-term viability.

Encouraged by recent innovations in assessment technology that are increasingly used in state K–12 testing and other large-scale assessment programs, NAEP’s leaders are exploring their feasibility for use with NAEP. Of interest is the extent to which these innovations might reduce costs while maintaining or enhancing technical quality. The work of this panel is to contribute to that exploration by providing analysis and recommendations for the next phase of NAEP. This chapter summarizes our recommendations: taken together, they chart an ambitious yet practical way for the National Center for Education Statistics and the National Assessment Governing Board to plan for NAEP’s future.

CLARIFYING AND DETAILING NAEP’S COSTS

To carry out its task, the panel needed to obtain information about NAEP’s costs, in order to put the potential value of possible cost savings in context. As detailed throughout the report, particularly in Chapter 2, and despite cooperation from NCES, the panel could not obtain a clear picture

of the overall budget for NAEP and how it is spent for the program’s different functions.

RECOMMENDATION 2-1: The National Center for Education Statistics and the National Assessment Governing Board should develop clear, consistent, and complete descriptions of current spending on the major components of NAEP, including contract structure, contractual spending, and direct spending on government staff and other costs. These cost descriptions should be used to inform major decisions about the program to ensure that their long-term budgetary impact is supportable.

CHANGING THE WAY TRENDS ARE MONITORED AND REPORTED

In the core subjects of mathematics and reading, NAEP has two assessment programs for measuring educational progress. One program, long-term trend NAEP, tracks trends since the 1970s and uses some test questions that are largely unchanged since NAEP’s beginnings. The other program, main NAEP, has tracked achievement since 1990. Main NAEP’s testing frameworks are reviewed and refreshed every 10 years or so.

NAEP currently reports both long-term trend data and main NAEP data for reading and mathematics. While this can be confusing to users, long-term trend NAEP complements the information provided by main NAEP and brings a useful balance to the NAEP portfolio. In addition, in light of the COVID-19 pandemic, long-term trend NAEP will provide a useful gauge to measure trends before and after the pandemic and provide information about the possible inequities that marked instruction during that period. However, long-term trend NAEP needs to be modernized to maintain its relevance and it is an open question whether its ongoing value will justify the costs of modernization.

RECOMMENDATION 3-1: The National Center for Education Statistics should prepare a detailed plan and budget for the modernization of long-term trend NAEP, including the costs of creating post-hoc assessment frameworks, bridging between paper and digital assessment, maintaining trends, and ongoing costs after the bridge. Congress, the National Assessment Governing Board, and the National Center for Education Statistics should then consider the value of a modernized and continued long-term trend NAEP in comparison with other program priorities. If continued, long-term trend NAEP should be renamed to better distinguish it from the trend data provided by main NAEP.

For the more comprehensive trend information provided by main NAEP, the program could benefit from small, more frequent changes to the assessment frameworks, potentially for every administration. There are three ways to revise the process. First, more frequent framework updates could encourage the identification of smaller changes that are needed. Second, the use of a standing framework committee with rotating membership, rather than the appointment of a new committee for each framework update, could serve to establish a group with a commitment to continuity and evolution. Third, better integrating the work of the framework and item development committees would allow content experts and item authors to iteratively and seamlessly inform each other’s work.

RECOMMENDATION 3-2: The National Assessment Governing Board (NAGB) and the National Center for Education Statistics (NCES) should work both independently and collaboratively to implement smaller and more frequent framework updates. This work should include consideration of the possibility of broadening the remit of the standing subject-matter committees that already exist to include responsibility for gradual framework updates, participation in item model development, and working directly with both NAGB and NCES.

INTEGRATING ASSESSMENTS FOR SUBJECTS WITH OVERLAPPING CONTENT

Since its beginning, NAEP has assessed subjects separately from one another. However, modern educational practice, as illustrated by the recent assessments adopted by many states, offers compelling arguments for combining assessments that test complementary subject matter, such as reading and writing or science and technology and engineering literacy. Such combined assessments could continue to report subscores for the subjects that have heretofore been assessed separately. Some items might do double duty and contribute to both subscores.

The upfront investment costs to develop the combined assessments might be substantial, but they may be outweighed by the savings realized from reducing the number of assessments. The downside of not actively considering assessments for combined subjects is illustrated by the cost pressures that force some subjects to be assessed infrequently, such as writing, or to have been effectively eliminated, such as economics and geography.

RECOMMENDATION 3-3: The National Assessment Governing Board should give high priority to consideration of integrating non-mandated subjects that are currently assessed separately (such as science and technology and engineering literacy), as well as the possibility

of integrated pairs of subjects that include a mandated subject, such as reading and writing. This consideration should examine the possibility of preserving separate subject subscores in an integrated assessment that could maintain trends, along with potential benefits related to efficiency and cost, closer alignment with student learning, and synergy across subjects that has been found by research.

UPDATING THE ITEM DEVELOPMENT PROCESS

The estimated average annual cost for the NAEP item development contract is $16.3 million, which is 9.3 percent of NAEP’s budget. Item development for NAEP is extremely expensive, and the reasons for these costs are unclear. NAEP develops about 600 new items a year at an average cost per item of roughly $3,700 for item creation and $36,500 for pilot testing. These costs are much higher than the item development costs of other testing programs on a per item basis; nevertheless, they represent only 2.5 percent of NAEP’s budget. The lack of clarity about the remaining spending in this contract is concerning since it represents 6.8 percent of NAEP’s budget.

RECOMMENDATION 4-1: The National Center for Education Statistics should examine the costs and scope of work in the item development contract that are not directly related to item development and pilot administration and explore possibilities for changes that would reduce costs.

Automated and Structured Item Development

Automatic item generation refers to the use of artificial intelligence and computer-based algorithms to automate some or all of the work of item development. This approach can lead to cost savings for assessments that predominantly use traditional multiple-choice item formats, particularly when the number of items is large. Although NAEP includes some of these types of items, their numbers are insufficient to justify the costs of implementing automatic item generation. In addition, it is difficult to use automatic item generation for the scenario-based and other complex item types that predominate in main NAEP. Though long-term trend NAEP uses larger numbers of traditional multiple-choice questions, this program develops few new items. As automatic item generation technologies evolve, however, it may be worthwhile to revisit their applicability for NAEP.

Although the current state of the art in automatic item generation has limited applicability to NAEP, recent advancements in the use of assessment design and engineering principles could be beneficial. Previous expert

panels have recommended evidence-centered design principles for NAEP, and NCES is carrying out work along these lines. Previous panels have also suggested that item development begin with NAEP’s achievement-level descriptions and cut scores and that these should drive the evidence claims that describe students’ knowledge and skills and define their proficiency levels. The task models that result and guide item development will benefit the work of both the subject-matter experts who work with NAGB on framework development and item review and the NCES contractors who develop the items. The quality control and pilot testing efforts that now focus on individual items could shift to the task models and bring efficiencies to item development.

RECOMMENDATION 4-2: The National Assessment Governing Board and the National Center for Education Statistics should move toward using more structured processes for item development to both decrease costs and improve quality. This work should include drawing from the detailed achievement-level descriptions to specify intended inferences and claims, better integrating the work of framework development and item creation, and carrying out critical aspects of review and quality control at the level of task models rather than at the level of individual items.

Changing the Mix of Item Types

NAEP currently uses a range of item types, including selected-response items, constructed-response items, and scenario-based tasks. Item types are typically aligned with cognitive and content specifications such that more complex item types are used to assess more complex skills. This alignment need not be the case, however. Research and practice demonstrate that selected-response or simple constructed-response items can be used to assess cognitively complex material.

Changing the mix of item types could potentially change NAEP’s average costs for item creation, pilot testing, test administration, and scoring. The average costs of the three item types are $1,750 for selected-response items, $2,500 for constructed-response items, and $13,000 for items that are part of scenario-based tasks. Given these differences, increasing the proportion of scenario-based items would increase item development costs, and increasing the proportion of selected-response items would decrease item development costs. There are likely similar relationships with respect to test administration and scoring costs.

RECOMMENDATION 4-3: The National Assessment Governing Board should commission an analysis of the value and cost of different

item types when multiple item types can measure the construct of interest. A full range of potential item types should be included in this analysis. The analysis should develop a framework for considering the tradeoff between value and cost. The value considered should include both the item’s contribution to a score and its signal about the relevant components of the construct. The costs considered should include item development (both item creation and pilot administration), administration time, and scoring.

MODERNIZING NAEP ADMINISTRATION

The current NAEP administration model relies on professionally trained NAEP staff and contractors who travel to schools to administer the assessment. When NAEP transitioned to digitally based assessment, NCES provided the technology needed for students to test. This was intended to reduce the burden on schools while maintaining the level of standardization that is deemed essential for NAEP. It also helped ensure that the assessment could be given in all schools, even those with limited bandwidth and technology resources. This model is laborious, expensive, and unusual in comparison with the administration approach used for typical state assessments. Because it represents about 28.6 percent of NAEP’s budget, test administration presents one of the clearest opportunities for cost savings.

NCES has outlined a plan to transition to locally based administration in which school staff would serve as proctors and students would use the school’s equipment for the NAEP assessment. NCES refers to this change as a transition to “contactless administration” because NAEP staff would no longer be directly in charge of administering the test. NCES is also considering an intermediate “reduced contact” model in which NAEP staff would support test administration virtually without being physically present in each school where the test is given. NCES recognizes that it may have to provide equipment and proctors to some schools.

RECOMMENDATION 5-1: The National Center for Education Statistics (NCES) should continue to develop its plan to administer NAEP using local school staff as proctors with online assessment delivery on local school computers, with development and bridge studies as needed to understand the feasibility and effects of this change in different contexts. This new model should be accompanied by adequate training and support of school staff, including tailored support for schools with more limited resources that may need NCES to provide proctors and equipment. NCES should also explore the use of flexible administration windows to allow schools to develop plans that accommodate local constraints on available equipment and consider appropriate ways to

compensate local schools for their contributions to the administration, especially during the transition to this new model.

The move to local test administration assumes that NAEP will develop some minimum specifications for the equipment, operating systems, and connectivity that are needed. Despite this initial effort at standardization, there is likely to be considerable variability among the devices that meet the minimum specifications. Some level of variability is unavoidable: it reflects both the practical reality of using local devices and the necessary customization that allows students to use devices that are familiar to them. Accounting for this variability will be important in the analysis of the assessment results. To carry out these analyses, the program will need to collect detailed information from the testing sites about the equipment and operating systems that are used.

RECOMMENDATION 5-2: Since a key component of moving to local administration will be the development of minimum requirements for equipment, operating systems, and connectivity, information about local devices, bandwidth, and administration conditions will have to be included in the data collection. Analysts should use statistical techniques that account for the effects of differences in devices and other local conditions to produce estimates that generalize across those differences. The National Center for Education Statistics should explore the use of random effects and other statistical techniques to produce estimates that reflect generalization across devices.

NCES plans to begin a transition to local administration of the reading and mathematics assessments in 2026. NCES has recently estimated the cost savings associated with this change of $56 million from 2026 to 2030. The panel’s analysis suggests that the potential savings may be substantially larger, perhaps as large as an annual average savings of roughly $30.8 million, or 18.7 percent of NAEP’s current budget.

RECOMMENDATION 5-3: The National Center for Education Statistics (NCES) should review its estimates of the potential cost savings from local administration of the mandated assessments in reading and mathematics in grades 4 and 8. The estimated savings are unexpectedly small when local administration would largely eliminate the large current costs for traveling proctors and equipment, even after considering any offsetting additional costs for training and technological infrastructure. NCES should also consider the use of the local administration model for reducing costs of all other assessments, as well as the costs for the pilot administration of new items.

Testing Two Unrelated Subjects for Each Student

Another way of decreasing administration costs is to gather more information from each sampled student by increasing the number of questions each one answers. NCES is considering a plan for administering two subjects to each student for the mandated assessments in reading and mathematics, a plan that we endorse.

Currently, each student takes two blocks of test questions in one subject—either reading or mathematics—and is given 30 minutes to respond to each block. The proposed change is to add an additional 30 minutes to the testing time for each student, for a total of 90 minutes. By increasing the testing time per student, the student sample could be reduced by a third, which would reduce administration costs, at least in the near term.

RECOMMENDATION 6-1: The National Center for Education Statistics (NCES) should continue to develop its plan to administer NAEP in longer sessions that allow for 90 minutes for the testing of cognitive items for each student. NCES should explore other models for using longer tests, in addition to its current plan. The decision to use longer tests should be based primarily on their potential to reduce testing burden by reducing the number of sampled students and to understand dependencies in proficiency across subjects, rather than being based on any long-term cost savings, which would be minimal with local test administration.

Revisiting the Sample Sizes Needed to Achieve NAEP’s Purposes

Given NAEP’s mission to track performance gaps, it is important that sample sizes are large enough for analyses to detect these differences. Besides simple two-way comparisons, such as differences in reading achievement across time or between Black and White students, NAEP also provides more complex multiway comparisons, such as cross tabulations that compare performance for students grouped by race, ethnicity, and gender or by race, ethnicity, gender, and family socioeconomic status, by state. The process of subdividing the full sample by multiple dimensions can create “cells” with samples sizes too small to report or make inferences about.

Reducing sample sizes is one way to reduce costs, but it needs to be done in a way that does not degrade the quality of valued comparisons and trends. Procedures called statistical power analyses are used to estimate the sample size needed to detect performance differences that are judged to be policy relevant, and they can guide NAEP in its decision making about reductions.

RECOMMENDATION 6-2: The National Center for Education Statistics should commission an analysis of the tradeoff between NAEP’s sample sizes and its statistical power in detecting differences in performance, including trends and gaps, and its ability to achieve minimum cell sizes for reporting on subpopulations. In particular, this analysis should consider the stated purposes of the National Assessment Governing Board to measure not only average scores, but also differences over time and between targeted subpopulations, and it should provide evidence about the level of precision required for these results to be meaningful to educators and policy makers. Evidence about meaningful levels of statistical power and minimum cell sizes for subpopulations should be directly related to the implications for NAEP’s sample sizes and associated administration costs.

Adaptive Testing

Computer-adaptive testing has been effectively used in large-scale testing since the mid-1990s. A typical adaptive test uses a student’s performance on one question to assign the next question at the right level of difficulty. With each response, the computer-based algorithm updates its estimate of the student’s proficiency level and selects the next question for the student to answer. Items are given to students until their proficiency can be estimated with a predetermined level of precision.

Because of the requirements of NAEP’s frameworks, computer-adaptive testing at the item level and across all subscales is not practical. However, the practical problems can be addressed in multistage adaptive testing, in which the adaptation occurs over groups of items and the first stage is limited to items that can be automatically scored, though this may prevent the use of some item types and may omit consideration of some subscales in the adaptation. The coarse adaptation that is possible is unlikely to result in substantial efficiencies across the full population, but it could improve estimates for some groups.

RECOMMENDATION 6-3: The National Center for Education Statistics (NCES) should not pursue adaptive testing for NAEP as a way of saving costs, but the agency should continue to investigate its use for its potential to improve the precision of statistical estimates and the test-taking experiences for low-performing students. NCES should also consider that no single approach to adaptive testing may fit all subjects and that some changes to assessment frameworks may be necessary to facilitate adaptive administration.

Coordinating Resources with NCES’s International Assessments

Coordinating the administration of one or more NAEP assessments with the administration of other NCES-administered assessments such as the Trends in International Mathematics and Science Study (TIMSS) or the Program for International Student Assessment (PISA) would potentially allow several assessment programs to share in administration costs; however, substantial difficulties would be involved. At a minimum, a coordinated approach would require that two or more assessments from different programs be administered in the same schools at the same time under roughly comparable conditions. Despite the potential cost savings, and the possibility such coordination would offer of establishing stronger statistical links across the assessments, the practical difficulties of coordination would be prohibitive and any net cost savings would be reduced as more schools are able to administer NAEP successfully with local proctors and equipment. Greater efficiencies across assessments would be possible if the content were shared or commingled, but that would entail even more practical difficulties.

RECOMMENDATION 6-4: Efforts to coordinate NAEP test administration with the international assessment programs sponsored by the National Center for Education Statistics should not be used as a strategy to reduce costs.

USING AUTOMATED ITEM SCORING

Automated scoring offers the potential to modestly reduce the cost of hand scoring NAEP’s constructed-response items, with an estimated annual savings of about $1.25 million per year, which is 0.7 percent of NAEP’s budget. Automated scoring is the use of statistical and computational methods to model scores assigned by human raters. Automated scoring has been widely adopted in K–12 assessment, licensure, and certification programs and is one of the most recognized applications of machine learning in educational measurement.

Automated-scoring models have displayed comparable performance relative to humans when scoring short and long essays and constructed-response items in reading comprehension and mathematics. They have also been successfully applied to mathematical expressions and equations entered using an equation editor or by graphing items using a graph interface. NAEP already has conducted proof-of-concept studies on automated scoring and, as this report was being finalized, was conducting a challenge to evaluate the performance of the latest scoring engines on reading assessment items.

The incorporation of automated scoring into NAEP would offer a number of likely benefits, including faster scoring, improved score consistency within and across administrations, higher-quality scoring of items when combined with human scoring, and increased information about student responses; it would also potentially offer cost savings. Importantly, automated scoring models do not drift and can help ensure that the scoring rubrics are applied consistently across years to support the centrality of trend to NAEP’s mission. However, automated scoring models require human monitoring to examine performance, and models may need recalibration. Automated scoring also offers the potential for collecting additional diagnostic information about student responses beyond a score, with data about spelling, coherence, syntactic variation, and other linguistic features, providing more insight about student knowledge and skills.

RECOMMENDATION 7-1: The National Center for Education Statistics (NCES) should continue its work to implement automated scoring on the reading and mathematics assessments for grades 4 and 8, with the item types that current scoring engines can score accurately and consistently. NCES should also consider the use of automated scoring on other assessments administered to state-level samples. In addition to benefiting from modest net reductions in costs, NCES should work to leverage the potential of automated scoring to improve the speed of reporting, increase the information provided about open-ended responses, and increase the consistency and fairness of scoring over time.

ADOPTING INNOVATIVE ANALYSIS AND REPORTING

Arguably, all of NAEP’s impact is mediated through analysis and reporting, which include not only score reports and related data and analyses, but also the frameworks, innovative example items, advanced psychometrics, and other assessment practices. NAEP score reports have regularly provided clear, high-level overviews of NAEP results. Reports that go a step further and analyze relationships between NAEP data and data from other sources are consistently among the most popular reports produced with NAEP data. Because NCES and NAGB are prohibited from using NAEP data to make policy recommendations, it is important to encourage others to use NAEP data to perform these essential analyses. Although there have been recent improvements, NCES is slow in making NAEP data available to others and the infrastructure for sharing the data is limited. The panel estimates the average annual analysis and reporting budget for NAEP at $17.6 million, which is 10.0 percent of the overall budget.

RECOMMENDATION 8-1: The National Center for Education Statistics should devote a greater percentage of its budget for innovative analysis and reporting that will increase the use and understanding of NAEP’s data, including finding ways to make the raw data available more quickly to researchers, improving the usability and sophistication of the NAEP Data Explorer, making process data more easily accessible, and expanding the availability and use of important contextual variables.

DEVELOPING A NEXT-GENERATION TECHNOLOGY PLATFORM

Exploratory research in psychometrics using data science, machine learning, and artificial intelligence provides a vision of a future in which assessments are much more flexible, adaptable, and integrated into a student’s learning experiences than is currently the case. To support the full range of innovations—those that should be implemented now and those that will become compelling in the next decade or even further in the future—NAEP needs a robust technology platform that is flexible enough to incorporate innovations as they become ready for application. These innovations span the full chain of the NAEP program, including test design, item and test development, test administration, analysis of results, and reporting.

NAEP currently does not have a platform with such a contemporary data architecture. Instead, NAEP data are held in separate “silos,” each administered by a different Alliance contractor. This arrangement is slow and would impede efforts to implement the proposed innovations discussed in this report.

The platform and technology approach of the current eNAEP system is almost a decade old and is based in a customized application that requires dedicated tablet computers, dedicated internet routers, and technical staff at every school site in which NAEP is administered. Developing the next generation of this assessment platform is necessary to administer NAEP on local computers.

NCES and its Alliance partners are working on a new comprehensive, multicomponent system called Next-Generation “Next-Gen” eNAEP. The system will include an assessment delivery platform application for students, as well as an assessment delivery engine and an item authoring system. NCES is currently developing a system that is custom built for NAEP. However, it is possible that the new system could use existing off-the-shelf components—or components that may become available as technology advances. Given the prevalence of online testing, some components may already exist that could be used in the Next-Gen system and potentially result in cost savings.

The costs of creating an entirely new Next-Gen system are considerable, and the panel was not provided information on the costs for the planned work. The estimated average annual cost of the platform development contract is $19.2 million, which is 11.0 percent of the overall NAEP budget. In addition, the program pays $10.2 million annually for the web support contract.

RECOMMENDATION 9-1: The National Center for Education Statistics (NCES) should regularly evaluate the software built by vendors or available in open-source libraries for its potential to meet the requirements of the different components of Next-Gen eNAEP. To support the viability of local administration of NAEP, the ease of installing, managing, and troubleshooting test delivery software should be a strong consideration in selecting the software to be used. Given the substantial ongoing expense associated with developing and maintaining a proprietary platform, Next-Gen eNAEP components should be custom built only if there are clearly large net benefits from doing so that have been identified by rigorous analysis. This decision should be made on a component basis, not as a single decision to build or buy all components. NCES should immediately carry out an evaluation with respect to any components of Next-Gen eNAEP that have not already been substantially developed, and then periodically thereafter. The platform development contract should provide the right incentives to make the best decision between building and buying each component.

Next-Gen eNAEP is an ambitious enterprise software development project that requires special expertise that is not typical for many of the staff members at NCES or its contractors.

RECOMMENDATION 9-2: The National Center for Education Statistics (NCES) should ensure that there is adequate internal and external expertise related to enterprise software development to support and oversee the development of Next-Gen eNAEP for both the NCES staff and the staff working for the platform development contractor. This software expertise is substantially different than expertise related to psychometrics and statistics.

RECOMMENDATION 9-3: The National Center for Education Statistics should seek expert guidance from enterprise application developers and educational technologists who understand assessment technology platforms to evaluate the reasonability of the projected costs for the development of Next-Gen eNAEP.

TAKING A SYSTEMIC APPROACH TO DESIGNING ASSESSMENT PROGRAMS¹

The panel estimates that management, planning, support, and oversight functions represent at least 28.7 percent of the total budget for NAEP, more than $50.3 million on average per year. This total includes costs related to federal employees and support contracts, with likely substantial additional costs for these functions in the program’s non-support contracts. These costs are very large, both in absolute terms and as a percentage of the overall NAEP budget. In consequence, meaningful cost reduction for the NAEP program will need to include a consideration of potential reduction of these costs.

RECOMMENDATION 10-1: The National Assessment Governing Board and the National Center for Education Statistics should commission an independent audit of the program management and decision-making processes and costs in the NAEP program, with a charge and sufficient access to review the program’s costs in detail. That audit should include proposed ways to streamline these processes.

With respect to the types of strategic innovation that are at the center of the panel’s charge, the research activities that support innovation are one of the functions that is included under the label of program management, planning, support, and oversight. The absence of a coordinated structure for these activities limits the ability of the program to focus on and leverage innovation. Such research might include research related to evaluation, validity, innovative assessment items, and new assessment technologies.

RECOMMENDATION 10-2: The National Center for Education Statistics should increase the visibility and coherence of NAEP’s research activities to help NAEP’s stakeholders, as well as other assessment programs, understand the innovations the program is investigating and the lessons it is learning. The NAEP research program should have an identifiable budget and program of activities.

A VISION FOR THE FUTURE

NAEP has been and can continue to be an invaluable resource for the nation to understand the learning of U.S. students over time. To make that possible, however, NAEP must adapt to the evolving landscape of

___________________

¹ After a prepublication version of the report was provided to the Institute of Education Sciences, NCES, and NAGB, this section was edited to reflect a broader range of costs and to revise the description and estimate of the costs associated with the non-support contracts.

technology and be mindful of the costs of its past practices and upcoming decisions. The analysis and recommendations in this report are offered as a way for NAEP to evolve to serve its important purposes for policy makers and the public well into the 2030s.

This page intentionally left blank.