8

Analysis and Reporting

This chapter addresses innovations related to analysis and reporting. It begins by describing the relevant costs and then discusses several aspects of innovative analysis and reporting for NAEP. Arguably, all of NAEP’s impact is mediated through its analysis and reporting procedures. As the primary mechanisms for communicating with the public, NAEP’s reports need to clearly convey the results of an assessment. They also need to be released in a timely way and help the public understand the current achievement of U.S. students, as well as the trends in educational progress over time. This chapter focuses on score reporting and the supporting data and analyses.

CURRENT COSTS

The Alliance contract for design, analysis, and reporting has an estimated annual average cost of $17.6 million.¹ With an average of 5.5 assessments per year, the design, analysis, and reporting costs average $3.2 million per test. These costs are divided equally between design and analysis (“design, psychometric and statistical analysis”) and reporting.²

Although the NAEP program periodically adopts innovations in design, analysis, and reporting, the approaches used tend to be broadly similar

___________________

¹ See Table 2-2 in Chapter 2. NCES response to Q33 says that the design, analysis, and reporting contract includes the following activities: “designs all pilot and field tests, operational assessments, and special studies; analyzes data ensuring reporting of valid results; proposes and prepares psychometric and statistical analyses compatible with previous NAEP methodologies; specifies data needed to meet the goals for reporting; and prepares reports.”

² NCES response to Q32.

across assessments, changing slowly over time. These costs appear to be higher than comparable costs for other large assessments—which often have much smaller budgets overall—but the panel was not able to obtain good data for comparison.

NCES notes, however, that NAEP has special analysis and reporting costs that many other large assessment programs do not have.³ With respect to design and analysis, NAEP has a mandate to describe trends, which can necessitate additional analyses to understand changes between assessments (such as the transition to digitally based assessments) or unusual periods (such as the first set of post-pandemic results). With respect to reporting, NAEP has a mandate to inform the public and make its results accessible, which includes developing multiple types of reports, carrying out high-profile press release events, and providing ready access to results from confidential data through the NAEP Data Explorer.⁴

INNOVATIVE ANALYSIS AND REPORTING

Substantial innovations in analysis and reporting have taken place over the past several decades, some specific to NAEP and others that are more general. On the analysis side, complex analyses that were previously specially programmed are now largely automated. This includes, for example, the analyses carried out to understand the performance of items and to populate the standardized “report card” reports.⁵ In addition, NAEP pioneered the provision of customized online analyses, with the NAEP Data Explorer, though the platform limits data elements and visualizations, which reduces its potential use for educational research.⁶

Another innovative aspect of data analyses concerns the use of process data, based on the data stream created by student interactions with computer-based testing. Analyses of these data have been a major contributor to new academic disciplines, such as learning analytics and educational data mining (Romero and Ventura, 2020). For example, research on other programs has investigated the extent to which process data predict constructs like student persistence and self-regulated learning (Roll and Winne, 2015). Leading scholars in educational measurement have also identified the potential use of these data to improve understanding of student knowledge (e.g., Mislevy, 2019).

NCES has played a leading role in making NAEP process data available for research and supporting work to develop its use (Center for Process Data, n.d.; Circi et al., 2020; Nation’s Report Card, n.d.). A recent

___________________

³ NCES response to Q77.

⁴ See https://nces.ed.gov/nationsreportcard/tdw/database/data_tool.asp.

⁵ NCES response to Q77.

⁶ Expanding the Data Explorer’s functionality would require addressing issues of confidentiality and student privacy, in addition to other technical issues.

synthesis provides an overview of how process data have been used historically in scoring NAEP items and how they could be used to deepen understanding about test-taker responses (Bergner and von Davier, 2019). Extensive research has been conducted by NAEP Alliance contractors and external researchers using NAEP process data, which has shown that these data can provide important information for forensics, integrity, and higher-order processes, such as scoring and scaling (Provasnik, 2021). Moving forward, interaction data during pilot testing could be used to determine item validity. In addition, feedback could be provided to proctors in real time, enabling them to identify students who might be disengaged and prompt them to return their focus to NAEP.

With respect to reporting, NAEP score reports have regularly provided high-level overviews of NAEP results. Recently, NAEP has used standardization and is considering potential use of artificial intelligence to speed the release of the initial “report card” reports for assessments, as well as shortening the time for making data available from 1 year to 6 months.⁷ Reports that analyze relationships between NAEP data and other variables are consistently among the most popular reports produced by NAEP.⁸ However, there are relatively few such reports, in large part because NAEP’s mission does not explicitly focus on producing such analytical reports to inform policy in the way that the Organisation for Economic Co-operation and Development and some other agencies do.⁹ This constraint underlines the importance to NAEP’s mission of making its data available to others and ensuring that those data provide support for the kinds of analyses that would be useful for policy, particularly with respect to inequities across groups.

Thinking broadly, there are at least three ways that the availability of NAEP data could be improved:

1. Speed: To encourage analyses by outside researchers, NAEP could make its data available more quickly to allow those analyses to take place. One way to do this would be to create a select pool of researchers who have access to NAEP’s raw data under embargo before their release to foster the development of policy-relevant analyses to appear shortly after release.
2. Accessibility: To encourage wider and more innovative use of NAEP data among researchers, journalists, and policy makers,

___________________

⁷ NCES response to Q41.

⁸ NCES response to Q43.

⁹ NCES response to Q45: “NAEP is also administered by a federal statistical agency [NCES] and adheres to accepted policies which prohibit the mixing of official statistics and policy analysis.”

2. the limited functionality of the NAEP Data Explorer could be expanded to make it easier to use and with more sophisticated analytic capabilities.
3. Depth: To support deeper analyses of NAEP’s innovative items, the program could expand the availability and use of its process data. To support deeper analyses of NAEP’s identification of inequities across groups, the program could expand the availability and use of important contextual variables to help identify plausible hypotheses about those inequities.

NAEP exists in a wide ecosystem of assessments and data. Already, advanced technologies—and what used to be advanced but now are commonplace technologies—are being applied widely across consumer and industrial platforms. These technologies raise users’ expectations of what is available and how it appears. For example, people now expect to be able to find things easily through internet searches and to access databases that are interactive, customizable, and potentially even adaptive to the user over time. In this ecosystem, there is substantial scope for innovations in the approach to analysis and reporting that can more effectively use the program’s substantial analysis and reporting budget to improve the insights that are generated from NAEP’s data.

RECOMMENDATION 8-1: The National Center for Education Statistics should devote a greater percentage of its budget for innovative analysis and reporting that will increase the use and understanding of NAEP’s data, including finding ways to make the raw data available more quickly to researchers, improving the usability and sophistication of the NAEP Data Explorer, making process data more easily accessible, and expanding the availability and use of important contextual variables.