3

Possible Structural Changes

There are a variety of ways that NAEP’s costs could be lowered if the program reduced the number of assessments or the frequency of administrations. With an average cost of $31.8 million per assessment, a reduction in the number of assessments could clearly save money. However, the panel did not consider the options of simply eliminating subjects or reducing the frequency of assessments as cost saving measures. The statement of task from the Institute of Education Sciences urged the panel to suggest options that would save money without impinging on the valuable information NAEP currently provides to its policy makers and the public. Decisions about when to test, what to test, and who to test are complex and involve many different entities and stakeholders. The panel recognizes that NAEP has existing commitments to provide assessment results for a specific range of domains, grade levels, and frequencies; we decided that remaking those decisions would exceed the statement of task.

There are less intrusive possibilities, however. In this chapter, we propose two types of structural change as possible avenues for decreasing costs but that are more relevant for other goals: the frameworks and their role in measuring trends, and the composition of assessments.

CHANGING THE WAY TRENDS ARE MONITORED AND REPORTED

NAEP assesses trend information for reading and mathematics through both main NAEP and long-term trend NAEP. Main NAEP uses test items

that are regularly updated to reflect new educational approaches and contexts. It is the source for trends in reading and mathematics achievement in grades 4, 8, and 12. Long-term trend NAEP uses test items that have been largely unchanged for decades and reports on trends in reading and mathematics achievement for ages 9, 13, and 17.

The intuitive, simple approach to monitoring progress is to offer the same assessment every time to subsequent cohorts of students. Al Beaton captured this in an oft-cited mantra: “When measuring change, do not change the measure.” However, his next two lines are equally important, “Precise implementation of this dictum is, of course, impossible in actual practice. In fact, NAEP has modified its measurement instruments by rearranging and reformatting assessment exercises since it began measuring trends” (Beaton, 1990, p. 10). The reasons for minor rearrangements can be technical (minimizing exposure of items, maximizing item information), practical (selecting items to accommodate pages, screens, or modes), or substantive (improving alignment of items to frameworks). Historically, the most fundamental challenges to reporting trends have been framework updates, but even without those updates, it has not been possible to maintain an unbroken trend line even for long-term trend NAEP. Even with unchanging items, the meaning and effective difficulty of those items will evolve over time as educational practices and the larger society change around them.

In particular, the shift toward the use of technology throughout education—and the regular changes that occur in that technology and instruction that uses it—makes it effectively impossible to keep delivery modes the same over time. Giving today’s students paper-and-pencil tests will not mean the same thing as it did 20 or even 5 years ago. Similarly, using increasingly dated or unfamiliar technology will result in the same kind of problem. Keeping assessments fixed cannot guarantee trend maintenance when so much else is changing.

Maintaining two programs within NAEP for tracking trends in reading and mathematics achievement is expensive, although the long-term trend assessments are relatively cheap because they use only national-level samples and typically have no costs for item development. Yet maintaining two programs for trend measurement is potentially confusing, particularly when the programs produce two similar but not identical estimates of educational progress. It is therefore reasonable to reevaluate the contribution of long-term trend NAEP to the overall program.

The Case for Reassessing Long-Term Trend NAEP

In 2017, NAGB convened a symposium on options for the future of long-term trend NAEP oriented around a focal paper by a former member

of the NAGB, Edward Haertel (2016).¹ At the time, the program was an appealing target for budget cuts due to inadequate funding, waning public attention to its results, outdated content, a lack of state results, and increasing distance from the then-previous administration in 2012. Although the symposium raised important concerns about the program, including its dated items and content,² it also provided a constructive rationale for preserving and improving the assessment. Preservation and improvement of long-term trend NAEP are appealing for at least three reasons:

1. The assessment adds 20 years to the trend data available from main NAEP, extending the trend line through the 1970s and 1980s, a period of substantial educational progress and achievement gap closure (NCES, 2013).
2. The assessment measures progress in age-based cohorts (9-, 13-, and 17-year-old students). As age distributions can change within grades over time, age-based cohorts are a useful contrast to main NAEP.
3. The assessment represents a relatively inexpensive reference point for main NAEP trends, which can provide a useful comparison in the event of unusual technical or national circumstances.

As one clear example of the utility of long-term trend NAEP, its most recent administration in 2020 managed to secure results for both 9- and 13-year-old students just before the COVID-19 pandemic closed U.S. schools in March of that year. Now, NAGB has redirected resources to offer the assessment to 9-year-old students again in 2022 and 13-year-old students in 2023. The results will be one of the best estimates available of the cumulative effects of the pandemic on national educational achievement.

Instead of eliminating or even deprioritizing long-term trend NAEP, it could instead be brought “up to code” as the NAGB symposium authors

___________________

¹ See https://www.nagb.gov/news-and-events/news-releases/2017/2017-long-term-trendsymposium.html.

² For example, Ina Mullis noted: “[T]he passages and items in the LTT [long-term trend] reading assessments are unlikely to be considered valid and robust assessments of reading. The LTTs assess straightforward comprehension of short pieces of text that are not authentic in the world of 2017, but are carefully replicated to retain their dated features. Reading comprehension is assessed almost wholly by multiple-choice questions. The LTT assessments will become increasingly irrelevant as students perform greater amounts of their reading online, and reading assessments move into the digital age.” Furthermore, in mathematics, she noted: “[T]he LTTs emphasize knowledge and skill much more than problem solving, making them essentially basic skills assessments, with some of the content outdated.” See https://www.nagb.gov/content/dam/nagb/en/documents/newsroom/naep-releases/naep-long-term-trend-symposium/Content%20of%20LTT%20Compared%20to%20Main%20NAEP_Ina%20Mullis%20021317_FINAL.pdf.

suggested. This updating could include creation of frameworks that describe the content of the assessments that make clear what long-term trend NAEP measures. Other ideas were offered at the 2017 symposium.³ If pursued, this effort would need to include a bridge study for transition to a digitally based assessment to minimize cost and increase relevance, as Mullis, Kolstad, and Haertel suggested in the NAGB symposium.⁴ In addition, it would be wise to undertake a renaming effort to minimize ongoing confusion between long-term trend and main NAEP and the trend information they provide.

RECOMMENDATION 3-1: The National Center for Education Statistics should prepare a detailed plan and budget for the modernization of long-term trend NAEP, including the costs of creating post-hoc assessment frameworks, bridging between paper and digital assessment, maintaining trends, and ongoing costs after the bridge. Congress, the National Assessment Governing Board, and the National Center for Education Statistics should then consider the value of a modernized and continued long-term trend NAEP in comparison with other program priorities. If continued, long-term trend NAEP should be renamed to better distinguish it from the trend data provided by main NAEP.

Improving the Way Main NAEP Measures Trends

Current policy on framework updates holds that NAGB will review the relevance of assessments and their frameworks for main NAEP at least once every 10 years.⁵ Moreover, NAGB can initiate a major update, even as the board is required, in its view, to balance needs for stable reporting of student achievement trends. However, each time frameworks are updated for main NAEP, the stability of its trend measurement is threatened.

Given the importance of trend data in the main NAEP program, the program could benefit from smaller changes to the assessment frameworks that are less likely to break the trend lines. Three changes to the process could encourage needed changes without breaking the trend line, as occurred, for example, with the framework updates for 2009 science and 2011 writing:⁶

___________________

³ See https://www.nagb.gov/news-and-events/news-releases/2017/2017-long-term-trendsymposium.html.

⁴ NAGB response to Q76. There are currently no frameworks for the long-term trend assessments.

⁵ Available https://www.nagb.gov/content/dam/nagb/en/documents/policies/frameworkdevelopment.pdf.

⁶ The frameworks for both of these assessments state that a new trend line will be started, given the change in the conceptualization of the construct (NAGB, personal communication, January 20, 2022). See https://www.nagb.gov/content/dam/nagb/en/documents/publications/frameworks/science/2009-science-framework.pdf; also see https://www.nagb.gov/content/dam/nagb/en/documents/publications/frameworks/writing/2011-writing-framework.pdf.

1. More frequent framework updates—potentially for every administration—could encourage the identification of smaller changes.
2. The use of a standing framework committee with rotating membership—rather than the appointment of a new committee for each framework update—could establish a group with a commitment to continuity and evolution.
3. The work of the framework and item development committees could be better integrated so that content experts and item authors iteratively and seamlessly inform each other’s work, with content experts providing feedback to item authors on the intent of the framework and item authors providing feedback to content experts on constraints with the feasibility of items.

Recommendations for standing subject-matter panels date not only to the 2012 report on the future of NAEP (NCES, 2012), but also to the evaluation of NAEP by the National Academy of Education (Glaser, Linn, and Bohrnstedt, 1997). In addition, the recent review of NAEP’s achievement levels by the National Academies of Sciences, Engineering, and Medicine (NASEM, 2017) recommended regular reviews and updates of the achievement-level descriptors and their alignment with the frameworks and the assessments themselves. These recommendations remain largely unaddressed,⁷ and NAEP’s trends have faced threats at regular intervals since, most recently in a proposed revision to the 2026 NAEP reading framework that required substantial revisions of its own to avoid perceived and potential threats to maintaining trend information (Jacobson, 2021). Standing panels with term limits and a rotating structure can help to ensure that NAEP can achieve its titular purpose.

In addition to helping ensure the maintenance of trend lines for main NAEP, the use of standing framework committees to update NAEP’s frameworks could also have some cost implications, both by lowering costs associated with protecting trends when proposed framework updates are drastic and by potentially using the existing subject-matter committees to update the frameworks rather than appointing standalone framework update committees. This change would require some institutional innovation—and close collaboration between NAGB and NCES—but the benefit for protecting NAEP trend data could be substantial.

___________________

⁷ NAGB is conducting studies to review and revise the achievement-level descriptors for reading and mathematics in response to the 2017 recommendations, which were ongoing when this report was being finalized (NAGB, personal communication, March 15, 2022). This point was added after a prepublication version of the report was provided to the Institute of Education Sciences, NCES, and NAGB, which did not acknowledge the ongoing studies.

RECOMMENDATION 3-2: The National Assessment Governing Board (NAGB) and the National Center for Education Statistics (NCES) should work both independently and collaboratively to implement smaller and more frequent framework updates. This work should include consideration of the possibility of broadening the remit of the standing subject-matter committees that already exist to include responsibility for gradual framework updates, participation in item model development, and working directly with both NAGB and NCES.

INTEGRATING ASSESSMENTS FOR SUBJECTS WITH OVERLAPPING CONTENT

Since its beginning, NAEP has assessed subjects separately from one another. Assessments are given in single subjects, such as mathematics and reading, rather than in subjects that might naturally occur in combination. Our statement of task asked us to consider potential cost savings related to “substantive overlaps between NAEP assessments”; the possibility of combining assessments in complementary subject areas is the second way the panel considered interpreting that request, after considering the overlapping trend information in reading and mathematics.

The panel considered several subject pairings. For all of them, we assume that an integrated assessment would allow the reporting of separate subscales for the separate subjects, allowing the separate subject results to continue to be reported where those are relevant.

Current practice in the states is one reasonable proxy to use as an indicator of current perspectives about meaningful groupings of educational subjects.⁸ A high-level consideration of trends in state assessment practices suggests three potential subject groupings that might be relevant for NAEP: reading and writing; science and engineering; and history, civics, economics, and geography.

Reading and Writing

States are held accountable to the terms of the Every Student Succeeds Act (ESSA), which requires them to administer “a set of high-quality student academic assessments in mathematics, reading or language arts, and science” (ESSA, Sec. 111(b)(2)(A), p. 2).⁹ Some states administer reading assessments only, and others administer language arts assessments (often called English language arts), which may include

___________________

⁸ Some other sources to consider for ideas about potentially meaningful groupings of educational subjects would include international assessments and NAGB’s work on postsecondary preparedness.

⁹ Available: https://www2.ed.gov/documents/essa-act-of-1965.pdf.

components of reading, writing, and other domains. Most states do not administer standalone tests in reading and writing, as NAEP does.

Science and Engineering

ESSA also requires states to assess science at least once in each of three grade spans: 3–5, 6–9, and 10–12. Many states base their science assessment program on the Next Generation Science Standards (NGSS) or a state-developed variation of those standards. Within the NGSS, scientific and engineering practices are intertwined, as noted in the document’s executive summary: “Scientific and Engineering Practices and Crosscutting Concepts are designed to be taught in context—not in a vacuum” (cited in Next Generation Science Standards [NGSS] Lead States, 2013, p. 1). In contrast, NAEP has separate assessments of science and what it calls technology and engineering literacy. NAEP’s science framework focuses on knowledge and skills in three areas: physical sciences, life sciences, and Earth and space sciences. The framework also lists four practices: “identify science principles, use science principles, use scientific inquiry, and use technological design” (NAGB, 2019a, p. 12). NAEP’s engineering technology and engineering literacy assessment focuses on three areas: technology and society; design and systems; and information and communication technology (NAGB, 2018, p. xvii). Some concepts appear in the frameworks for both assessments.¹⁰

History, Civics, Economics, and Geography

These four subjects comprise the broad category of social studies. A recent survey cited in a new NAEP validity studies panel report (O’Malley and Norton, 2022) showed that of the 35 states that responded, at least 18 states assess social studies. Seven of the 35 states reported that they assess all four social studies content areas within one test, while two states test some but not all four areas within one test. In 15 of these states, civics and U.S. history are included in the assessment. Two others have variations across grade levels. NAEP has traditionally assessed all four as separate assessments, though the current assessment schedule shows no plans to assess economics and geography through 2030.¹¹

Two other potential subject groupings are not reflected in current state assessment practice as combined assessments but involve substantive relationships across assessments that may be meaningful to reflect in NAEP: reading with science or history, and mathematics and science. NAEP’s new reading framework (NAGB, 2021) proposes three subscales that would

___________________

¹⁰ The NAEP validity studies panel is currently studying these overlapping concepts and the possibility of combining assessments.

¹¹ See Table 2-1 and https://www.nagb.gov/about-naep/assessment-schedule.html.

report reading performance within and across three disciplinary contexts, including science and social studies. The pairing of mathematics and science is reflected in the Trends in International Mathematics and Science Study.

With respect to potential cost savings from combining assessments the primary opportunities lie with the three subject combinations—language arts, science and engineering, and social studies—that are already reflected in current state assessments. There are several relevant considerations with respect to the net benefit of such combinations: need for new frameworks, assessment schedule, sample size, and preserving subjects.

In all cases, it would be necessary to develop new frameworks as the first step in developing a combined assessment, which has practical implications. Since the reading framework has just been revised, it would not be an opportune time to consider a combined assessment for reading and writing. However, civics and U.S. history are scheduled to have updated frameworks in time for the 2030 assessment, a timing that would potentially allow this combination to be considered.

A coordination of the assessment schedule for two or more subjects with overlapping or complementary content allows for the possibility of increasing coordination across them. This is the case for civics and U.S. history and also for science and technology and engineering literacy. In contrast, economics and geography are no longer on the assessment schedule (through 2030), and writing is not on the schedule until 2030, giving limited opportunities for considering any coordination.

In terms of sample size, the most money would be saved by combining two large assessments that include state and urban district samples because of the possibility of eliminating an assessment with a high cost for test administration to a large sample. However, none of the likely combinations fall into this category. Thus, any potential cost savings would likely relate to the smaller cost savings associated with a reducing an assessment that has only a national-level sample.

The assessment schedule illustrates the cost limitations that force some subjects to be assessed minimally (writing) or not at all (economics and geography). For these subjects, cost savings have been realized by eliminating entire assessments.

NAEP’s framework committees are tasked with updating the perspectives on educational goals within individual subjects, and by design, those committees work within the confines of an individual subject. This narrow focus is illustrated by a statement in the new reading framework adopted by NAGB in August 2021 that expressly precludes such consideration: “The 2026 NAEP Reading Assessment will continue NAEP’s longstanding focus on reading comprehension, rather than foundational skills or writing” (NAGB, 2021, p. 13).

At various times, NAGB has noted the importance of considering the possibility of assessments that combine several subjects (see, e.g., NAGB, 1996, p. 5; 2017, p. 13; 2019a, p. 7). In addition, as noted above, there is currently some activity focused on considering the possibility of integrating the science and the technology and engineering literacy assessments. However, even the brief review above suggests there might also be strong arguments for integrating other subjects, and we note the integration across disciplinary contexts already reflected in the new reading framework. Such combined assessments could continue to report subscores for the subjects that are currently assessed with separate assessments.

Although there would be upfront investment costs to develop combined assessments, they could result in cost savings from reducing the number of assessments. The cost savings are likely to be small in most cases because at least one of the assessments in each pairing is given infrequently and usually to only a national sample. However, even the small cost savings from reducing these assessments are sufficient to substantially limit their presence in the assessment schedule. One downside of not actively considering the possibility of integrating assessments is illustrated by the cost pressures that force some subjects to be assessed infrequently or to be effectively eliminated.

RECOMMENDATION 3-3: The National Assessment Governing Board should give high priority to consideration of integrating non-mandated subjects that are currently assessed separately (such as science and technology and engineering literacy), as well as the possibility of integrated pairs of subjects that include a mandated subject, such as reading and writing. This consideration should examine the possibility of preserving separate subject subscores in an integrated assessment that could maintain trends, along with potential benefits related to efficiency and cost, closer alignment with student learning, and synergy across subjects that has been found by research.

This page intentionally left blank.