National Academies Press: OpenBook
« Previous: Executive Summary
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2022. A Pragmatic Future for NAEP: Containing Costs and Updating Technologies. Washington, DC: The National Academies Press. doi: 10.17226/26427.
×

1

Introduction

The National Assessment of Educational Progress (NAEP) is a congressionally mandated program administered by the National Center for Education Statistics (NCES) of the Institute of Education Sciences (IES) in the U.S. Department of Education. Policy for NAEP is set by the independent National Assessment Governing Board (NAGB). Known as “The Nation’s Report Card,” NAEP provides an assessment of what 4th, 8th, and 12th graders in the United States know and can do in reading, mathematics, science, writing, and other academic subjects. For reading and mathematics, NAEP also provides separate measures for 9-, 13-, and 17-year-olds. For over half a century, the NAEP program has been an essential resource that helps educators and policy makers understand important outcomes in U.S. education. NAEP has also played a crucial role in carrying out the policy priorities reflected in the Elementary and Secondary Education Act.

CHARGE TO THE PANEL

To build on NAEP’s past successes and ensure its continued leadership and viability into the future, IES asked the Committee on National Statistics of the National Academies of Sciences, Engineering, and Medicine (the National Academies) to consider innovations that have the potential to reduce the program’s costs while maintaining or enhancing its technical

Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2022. A Pragmatic Future for NAEP: Containing Costs and Updating Technologies. Washington, DC: The National Academies Press. doi: 10.17226/26427.
×

quality and informative value.1 The request specifically focused on a set of computer-based innovations that have been successfully used in other large-scale assessments: see Box 1-1 for the full statement of task.

In response to the request from IES, the National Academies formed the Panel on Opportunities for the National Assessment of Educational Progress in an Age of AI and Pervasive Computation: A Pragmatic Vision. The

___________________

1 IES concurrently commissioned two other studies from the National Academies. One addresses NCES’s portfolio of activities and products, operations, staffing, and use of contractors, focusing on the Center’s statistical programs. The second addresses the future of education research at IES, including critical problems where new research is needed; new methods or approaches for conducting research; and new types of research training investments.

Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2022. A Pragmatic Future for NAEP: Containing Costs and Updating Technologies. Washington, DC: The National Academies Press. doi: 10.17226/26427.
×

panel includes members with expertise in psychometrics and educational measurement, new technology-based assessment approaches, statistics and data science, education policy and research, NAEP, and other large-scale assessment programs. Given the pragmatic nature of the request, the panel membership was designed to focus on experts with knowledge about the use of technology-based approaches in educational contexts rather than artificial intelligence (AI) experts who carry out basic research or who work on AI applications outside of education.

The IES request was accompanied by a sense of urgency from the leaders who are responsible for guiding the NAEP program. NAEP costs have increased substantially over the past two decades. Although these increases have been accompanied by important expansions in the information provided by NAEP, there is a growing sense that the high cost of NAEP is threatening the viability of the program. In this context, the promise offered by digital approaches that could reengineer the process of assessment design, development, administration, and reporting is highly attractive. At the same time, however, the program’s leaders are skeptical about past promises of technological benefits that went unfulfilled: this skepticism led to the important caveat in the IES request that the National Academies consider a pragmatic vision for innovations in the NAEP program. IES asked for guidance about innovations that have a demonstrated potential to provide improvements in the next few years.

While focusing on the possibility for substantial cost reduction, the IES request highlights the importance of balancing cost reduction with the competing objectives of technical quality and informative value. This set of three objectives closely parallels the 2025 vision of NAGB for the program, which highlights utility, frequency, and efficiency.2

The statement of task calls out three specific computer-based innovations—automatic item generation, local test administration,3 and computer-adaptive testing—that suggest the kinds of promising changes that IES wanted the panel to consider. However, the request also underlines the importance of considering other major changes that might also show a large promise of reducing costs while maintaining the quality and the informative value of the program. The request specifically mentions one such non-technological change, involving the potential elimination of substantive overlaps across assessments. The request also references possible programmatic changes to support innovation, reflecting the importance of considering any concrete changes that might be needed in the structure of the NAEP program or contracting structure to support innovation.

___________________

2 See https://www.nagb.gov/content/dam/nagb/en/documents/who-we-are/2020_NAGBStrategic-Vision_FINAL.pdf.

3 Described as “remote” test administration in the statement of task.

Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2022. A Pragmatic Future for NAEP: Containing Costs and Updating Technologies. Washington, DC: The National Academies Press. doi: 10.17226/26427.
×

THE PANEL’S APPROACH

The topics listed for consideration in the statement of task are not new and have been considered for adoption in recent years by many large-scale testing programs, including NAEP. As a result, the panel was designed to include members with expertise about the relevant technological innovations, as well as members with experience in implementing such innovations in other large-scale assessment programs and members with deep knowledge of NAEP itself. In addition to the knowledge and expertise of its members, the panel began its work by soliciting comments on the statement of task from 16 additional experts.

In its work, the panel reviewed key aspects of the research literature about the application of computer technology to assessment and recent experience in other large-scale assessment programs. Most importantly, it also reviewed extensive information about NAEP itself. This information included descriptions of the program’s structure, research carried out to consider possible changes in the program, the program’s current plans for innovation, and information about the program’s costs. The panel received documents provided by NCES, as well as the agency’s responses to a series of questions that arose in the course of the panel’s work.4 The NCES materials and responses to the panel questions provided a basis for considering the general promise of the various innovations, including those used in assessment programs other than NAEP.

In developing a way to consider the potential innovations that might be relevant, the panel considered the overall goals of the NAEP program, the sequence of topics addressed during assessment development and validation, and the types of innovations that are being used successfully in other large-scale assessment programs. The panel broadened the list of major innovations to review to include automated scoring and the technological infrastructure necessary to support the full range of assessment program processes, including the program’s oversight and management processes. These program processes and underlying infrastructure can be easily forgotten but are nonetheless critical to NAEP’s overall costs and efficiency.

As the panel deepened its understanding of the cost structure of NAEP, it became clear that the innovations described in the statement of task would not be sufficient to significantly affect NAEP’s high costs. In response to this finding, the panel decided to broaden its approach to consider the

___________________

4 The documents and responses received from NCES are available on request from the project’s Public Access File, along with the other documents provided by NCES. Many of the citations in this report are to NCES responses to specific numbered questions from the panel, which are available in the Public Access File. Those panel questions are labeled “Q” in the report. The process for obtaining information from a project’s Public Access File is provided at the following link: https://www8.nationalacademies.org/pa/information.aspx.

Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2022. A Pragmatic Future for NAEP: Containing Costs and Updating Technologies. Washington, DC: The National Academies Press. doi: 10.17226/26427.
×

overall structure of NAEP’s costs. As a result of this decision to provide a more comprehensive, though still limited, picture of NAEP’s costs, there are several topics in the report for which the panel lacks the necessary expertise and data to provide a satisfying analysis of the relevant cost drivers. In these cases, we limit ourselves to a brief discussion of the cost structure as we understand it, a discussion of relevant observations given our expertise related to assessment and technology, and a recommendation for further work by people with more information and appropriate expertise to examine the specific issues we cannot address.

Reports about the use of technology in assessment often focus on the ways technology allows innovative item types or the analysis of detailed process data related to test takers’ responses. Given the cost focus of the request to the panel, these innovations offered by technology are addressed only in the context of high-level comments about the information that NAEP provides and the importance of continuing to extend and improve that information. These benefits clearly relate to the three-way tradeoff that the statement of task asks the panel to consider—cost, quality, information—but they do not relate to the project’s primary focus on potential cost reductions.

In the course of considering innovations that might reduce costs, the panel concluded that some of them were not promising for cost reduction but were promising for other reasons. In the context of the three-way tradeoff in the statement of task, this means that these innovations are potentially useful for improving the technical quality or the information provided by the program, but not for reducing cost. If the panel had adopted a rigid cost focus for the report, we might have declined to mention these other benefits; instead, we have chosen to briefly discuss them, while noting that they are not promising for cost reduction.

The panel decided not to devote a section of the report to the possibility of saving money by eliminating assessments, although substantive overlap across assessments certainly exists. It is obvious that it is possible to save money on assessments by eliminating them. Assessments could be eliminated by either reducing the frequency of specific NAEP assessments5 or eliminating an assessment when a specific NAEP assessment overlaps with a specific international assessment.6 The panel decided to defer to the political processes that have led to a commitment to provide assessment results for a specific range of domains, grade levels, and frequencies. However, within

___________________

5 For example, NAEP’s reading and mathematics assessments could be reduced to a 4-year frequency instead of the current 2-year frequency.

6 For example, NCES support of the Progress in International Reading Literacy Study (PIRLS), which tests reading in 4th graders, could be eliminated as duplicative of NAEP’s 4th-grade reading tests, or vice versa.

Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2022. A Pragmatic Future for NAEP: Containing Costs and Updating Technologies. Washington, DC: The National Academies Press. doi: 10.17226/26427.
×

the overall structure of these commitments, the panel did consider some ways of reconfiguring the current assessment structure to combine assessments in ways that arguably could provide the same (or better) information for policy makers at less cost.

The study’s timing during the pandemic highlighted issues in education related to inequities in access across the system, challenges in the use of technology, and the need for more timely and responsive measures of achievement. These issues have been persistent in education, but the pandemic placed the need for continued improvement in stark relief. As appropriate, we comment on these issues in the context of discussing potential innovations.

While acknowledging the many technical questions that must be addressed in assessment, the panel endeavored to respond to the sponsor’s request to write a short and broadly accessible report. NAEP has a large and diverse group of stakeholders who are interested in the program’s future direction. The key opportunities and constraints that affect that direction can be understood without mastering the details of the various technical issues. Similarly, the key issues with NAEP’s cost structure can be broadly understood without reviewing detailed accounting records. The report references the necessary supporting documents but focuses on a set of central arguments about the program that can be understood by NAEP’s many stakeholders.

The next chapter provides an overview of NAEP’s structure, goals, cost, and administration. Chapter 3 considers two ways that the content and administration of the different NAEP assessments might be reconfigured to save money while providing equivalent or improved information. Chapter 4 addresses item development and the opportunities for potential cost savings, including the possibilities for automated or more structured item generation. Chapters 5 and 6 address the substantial costs related to the administration of NAEP, with Chapter 5 addressing the program’s plans to administer NAEP using local proctors and equipment and Chapter 6 offering other potential innovations to reduce administration costs. Chapter 7 discusses scoring and the possibility of reducing costs through automated scoring. Chapter 8 discusses the costs of the analysis and reporting of NAEP results. Chapter 9 describes NAEP’s investment in the technology platform, eNAEP, that is essential to NCES’s plans to decrease assessment administration costs and is expected to be able to support a number of other technology-based innovations. Chapter 10 describes NAEP’s overall program management, planning, support, and oversight costs. Chapter 11 summarizes the report’s arguments and recommendations.

Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2022. A Pragmatic Future for NAEP: Containing Costs and Updating Technologies. Washington, DC: The National Academies Press. doi: 10.17226/26427.
×
Page 5
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2022. A Pragmatic Future for NAEP: Containing Costs and Updating Technologies. Washington, DC: The National Academies Press. doi: 10.17226/26427.
×
Page 6
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2022. A Pragmatic Future for NAEP: Containing Costs and Updating Technologies. Washington, DC: The National Academies Press. doi: 10.17226/26427.
×
Page 7
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2022. A Pragmatic Future for NAEP: Containing Costs and Updating Technologies. Washington, DC: The National Academies Press. doi: 10.17226/26427.
×
Page 8
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2022. A Pragmatic Future for NAEP: Containing Costs and Updating Technologies. Washington, DC: The National Academies Press. doi: 10.17226/26427.
×
Page 9
Suggested Citation:"1 Introduction." National Academies of Sciences, Engineering, and Medicine. 2022. A Pragmatic Future for NAEP: Containing Costs and Updating Technologies. Washington, DC: The National Academies Press. doi: 10.17226/26427.
×
Page 10
Next: 2 NAEP Overview: Structure, Goals, and Costs »
A Pragmatic Future for NAEP: Containing Costs and Updating Technologies Get This Book
×
Buy Paperback | $20.00 Buy Ebook | $16.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The National Assessment of Educational Progress (NAEP) - often called "The Nation's Report Card" - is the largest nationally representative and continuing assessment of what students in public and private schools in the United States know and can do in various subjects and has provided policy makers and the public with invaluable information on U.S. students for more than 50 years.

Unique in the information it provides, NAEP is the nation's only mechanism for tracking student achievement over time and comparing trends across states and districts for all students and important student groups (e.g., by race, sex, English learner status, disability status, family poverty status). While the program helps educators, policymakers, and the public understand these educational outcomes, the program has incurred substantially increased costs in recent years and now costs about $175.2 million per year.

A Pragmatic Future for NAEP: Containing Costs and Updating Technologies recommends changes to bolster the future success of the program by identifying areas where federal administrators could take advantage of savings, such as new technological tools and platforms as well as efforts to use local administration and deployment for the tests. Additionally, the report recommends areas where the program should clearly communicate about spending and undertake efforts to streamline management. The report also provides recommendations to increase the visibility and coherence of NAEP's research activities.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!