National Academies Press: OpenBook

A Pragmatic Future for NAEP: Containing Costs and Updating Technologies (2022)

Chapter: 9 Technological Infrastructure

« Previous: 8 Analysis and Reporting
Suggested Citation:"9 Technological Infrastructure." National Academies of Sciences, Engineering, and Medicine. 2022. A Pragmatic Future for NAEP: Containing Costs and Updating Technologies. Washington, DC: The National Academies Press. doi: 10.17226/26427.
×

9

Technological Infrastructure

This chapter addresses the technological infrastructure that is essential to developing, administering, analyzing, and reporting an assessment in the modern era. This infrastructure is obviously critical to many of the potential innovations discussed in this report. The chapter begins with a short overview of NAEP’s current technology costs and investments and then describes a vision of the functionality that a fully developed technological infrastructure can provide. The last section discusses the development status of Next-Gen eNAEP, which will provide the next generation of the program’s technology platform.

CURRENT COSTS

Technology is covered by two contracts in the NAEP Alliance, one for the web and the other for platform development.1 The estimated annual average cost is $10.2 million for the web contract and $19.2 million for

___________________

1 NCES response to Q33: The “web/technology development, operations and management” contract covers the following activities: “Develops, implements, and supports Internet-related applications and services; identifies and deploys emerging technologies and new products to improve NAEP’s web and other computer-based products and services; monitors compliance with all NCES web requirements, and ensures timely release of quality products and services using Web technologies.” The “NAEP platform development” contract covers the following activities: “Develops NAEP assessment delivery platform utilizing a state-of-the-art, age-appropriate user interface (UI) and overall user experience (UX) that is consistent, intuitive, and accessible across all subjects and grade levels.”

Suggested Citation:"9 Technological Infrastructure." National Academies of Sciences, Engineering, and Medicine. 2022. A Pragmatic Future for NAEP: Containing Costs and Updating Technologies. Washington, DC: The National Academies Press. doi: 10.17226/26427.
×

the platform development contract.2 The platform development contract is roughly evenly divided between development of the new system and maintenance (initially maintenance of the old eNAEP system and later maintenance of the new Next-Gen eNAEP system).

VISION FOR A TECHNOLOGICAL INFRASTRUCTURE FOR NAEP

Exploratory research in psychometrics using data science, machine learning, and artificial intelligence provides a vision of a future in which assessments are much more flexible, adaptable, and integrated into a student’s learning experiences than current assessments (Markowetz et al., 2014; Romero and Ventura, 2020; von Davier et al., 2019). The above chapters have already covered some of the capabilities that could be included in such an approach to assessment, including model-driven and fully automated item generation (Chapter 4), the administration of assessment on a wide variety of devices (Chapter 5), adaptive testing (Chapter 6), automated scoring of constructed responses (Chapter 7), and the analysis of process data (Chapter 8).

To support a full range of innovations—those that should be implemented now and those that will become compelling in a decade or more—NAEP needs a robust technology platform that is flexible enough to incorporate a series of innovations as they become ready for application. These potential innovations span the full chain of the NAEP program, including test design, item and test development, test administration, scoring, analysis of results, and reporting.

A robust data architecture for NAEP needs to integrate data flows, support quality control and other analysis processes, and provide easy and secure access to historical NAEP data. At present, NAEP data are held in separate silos, each administered by a different Alliance contractor, which prevents NCES staff from directly accessing data outside of pre-defined data products. For any new kind of analysis, additional requests and review are required, which impedes new analytical studies and more integrated use of data. This situation can make it prohibitively expensive and slow to implement psychometric innovations that are based on response data, such as adaptive testing, automated scoring, adaptive reporting, and other operational implementations of advanced algorithms. By reducing the manual merging, integration, and analysis of data files, NAEP can become more efficient and reduce the effort needed for administration.

Contemporary data architectures provide an elegant and effective solution to these issues, with secured authentication systems providing access application programming interface endpoints in a standardized manner by any technology application. This new architecture could provide the

___________________

2 See Table 2-2 in Chapter 2.

Suggested Citation:"9 Technological Infrastructure." National Academies of Sciences, Engineering, and Medicine. 2022. A Pragmatic Future for NAEP: Containing Costs and Updating Technologies. Washington, DC: The National Academies Press. doi: 10.17226/26427.
×

foundation for NAEP to successfully integrate innovations from artificial intelligence, assessment engineering, and other advances in ways that have been very difficult to incorporate in the past. If such a system is not pursued, it could block the successful modernization of NAEP and require ongoing large investments to provide ongoing basic functionality and stability.

The field of software development has generally moved over the past several decades toward leveraging the benefits that come from using standards-based approaches and interoperable systems. These approaches have generally been accepted in the field as having substantial benefits for reducing errors and costs related to software development. These common industry practices include

  • Standards-based development: the use of industry standards, such as IMS QTI3 for item structure and xAPI4 for process data format, increases the speed of development and enables integration with other systems and analysis.
  • Cloud-based technology systems: the use of cloud native platforms, such as Amazon Web Services or Microsoft Azure, can provide the foundation for increased interoperability in a distributed system, as well as improved services and reduced cost. This is likely to be true for NAEP, for which the small annual assessment window makes it attractive to have an infrastructure system that can be “turned off” when not in use.
  • Federated architecture: an assessment architecture based on a “system of systems” principle, linked together through these protocols and data security standards, enables new systems and technologies to be added as they become available.

These new and now common practices suggest important staffing considerations in developing such systems. It is unclear whether these considerations are reflected in NCES’s current staffing plans and contract arrangements. The expertise needed to lead a large-scale psychometric software development of this kind is distinctly different from the expertise needed to develop assessment items, administer an assessment, or analyze its results, which all rely on the resulting technology infrastructure. But using the infrastructure is different than creating it. The expertise needed for the software development for the technology architecture includes

  • experience with cloud-based architectures and software development;

___________________

3 See https://www.imsglobal.org/question/index.html.

4 See https://sagroups.ieee.org/9274-1-1/.

Suggested Citation:"9 Technological Infrastructure." National Academies of Sciences, Engineering, and Medicine. 2022. A Pragmatic Future for NAEP: Containing Costs and Updating Technologies. Washington, DC: The National Academies Press. doi: 10.17226/26427.
×
  • psychometric processes and data standards for assessment items; and
  • agile-based and customer-responsive software development.

DEVELOPMENT OF THE NEXT-GEN ENAEP PLATFORM5

The platform and technology approach of the current eNAEP system is almost a decade old and is based on a custom application using dedicated tablet computers, dedicated internet routers, and technical staff at every school that participates in NAEP. The dedicated tablets are only used during the administration of NAEP and other NCES surveys; they are unused for the rest of the year. As described in Chapter 5, this approach reduces school burden, provides absolute consistency in appearance of and interaction with test items, and does not require any reliance on school connectivity. However, it is very expensive and no longer necessary for schools that now routinely administer high-stakes assessments using local computers.

The current contract for NAEP platform development funds the development of Next-Gen eNAEP, a multistage development activity that is planned through 2024. This new system includes not only an assessment delivery platform application for students, but also a library of reusable item components, a “data lake” to store all current and historical NAEP data, and mechanisms for data access with a commitment to extensibility, reusability, and other contemporary software design principles. The current contract goes through 2024 and includes a field test for online administration using NAEP-owned devices in 2023 and a proof-of-concept study for delivering NAEP on school-owned devices in 2024.6 An operational system is a deliverable of the current contract, and Next-Gen eNAEP will be used for both the pilot and operational NAEP administrations in 2024.7 After 2024, the system will be used to deliver NAEP on school-owned devices in a field test in 2025 and then for an operational test in 2026.

The panel was provided documentation that describes the goals, vision, and roadmap to develop Next-Gen eNAEP. However, these materials do not provide the detail needed to understand the underlying technical approaches. In addition, much of the documentation provided focuses on test delivery, so it is unclear how much functionality will initially be provided related to item authoring and test assembly, scoring, reporting, data architectures, and data access.

___________________

5 After a prepublication version of the report was provided to Institute of Education Sciences, NCES, and NAGB, this section was edited to revise the descriptions of the capabilities of the Next-Gen eNAEP platform, the timeline for its use in operational administrations, and the expertise of the NCES staff who oversee its development.

6 NCES responses to panel questions (personal communication, August 11, 2021).

7 NCES response to Q73c.

Suggested Citation:"9 Technological Infrastructure." National Academies of Sciences, Engineering, and Medicine. 2022. A Pragmatic Future for NAEP: Containing Costs and Updating Technologies. Washington, DC: The National Academies Press. doi: 10.17226/26427.
×

An operating assumption from NCES is that item appearance should not change between the current and the new system, out of concern about potential threats to item validity and trend. Therefore, the new system incorporates the presentation layer from the current eNAEP while replacing the back-end data infrastructure, item rendering, and other components. However, as discussed in Chapter 5, some variance is inevitable with the use of multiple machines and browsers, and only a basic level of compatibility can be guaranteed in any software development activity. Recommendation 5-2 (in Chapter 5) proposes that the program should plan to accommodate this inevitable variance by collecting data on the systems used at different sites and then reflecting any differences in the analysis. This approach opens up potential flexibility in the item appearance requirements for Next-Gen eNAEP.

The custom building of enterprise software for a single program is an expensive approach, in terms of both initial development and delivery and later maintenance on an ongoing basis. The amount budgeted in the current contract for this activity through fiscal 2024 (roughly $50 million) is an indicator of the cost implications of this approach. By building a new system internally, NCES does not leverage the cost sharing that occurs in standard commercial development, in which a component developed serves multiple customers who effectively share the cost, whether explicitly or indirectly. NCES informed the panel that no current system met the full requirements when the current direction was defined, and this approach enables the ultimate control in terms of functionality and requirements.8 However, this choice requires that NAEP spend whatever it takes every time a new functionality, defect fix, or other maintenance update is required.

Given the dynamism in the field of software development, it is likely that the options available to NCES for both building and buying the relevant components of eNAEP are substantially different than they were when the decision was made to develop the system in-house. Fortunately, the platform development contract includes research and business analysis of other technologies to “stay continually abreast of the latest trends and innovations in large-scale assessment and education technology.”9 As part of this work, it is important to be clear that many vendors have successfully administered K–12 summative assessments using platforms that are based on the kinds of software tools and standards specified by NAEP.

RECOMMENDATION 9-1: The National Center for Education Statistics (NCES) should regularly evaluate the software built by vendors or available in open-source libraries for its potential to meet the

___________________

8 NCES response to Q73e.

9 NCES response to Q73e.

Suggested Citation:"9 Technological Infrastructure." National Academies of Sciences, Engineering, and Medicine. 2022. A Pragmatic Future for NAEP: Containing Costs and Updating Technologies. Washington, DC: The National Academies Press. doi: 10.17226/26427.
×

requirements of the different components of Next-Gen eNAEP. To support the viability of local administration of NAEP, the ease of installing, managing, and troubleshooting test delivery software should be a strong consideration in selecting the software to be used. Given the substantial ongoing expense associated with developing and maintaining a proprietary platform, Next-Gen eNAEP components should be custom built only if there are clearly large net benefits from doing so that have been identified by rigorous analysis. This decision should be made on a component basis, not as a single decision to build or buy all components. NCES should immediately carry out an evaluation with respect to any components of Next-Gen eNAEP that have not already been substantially developed, and then periodically thereafter. The platform development contract should provide the right incentives to make the best decision between building and buying each component.

The NAEP Alliance member selected to perform this activity is a global leader in psychometric research and development and is not known in the field for development of production enterprise software applications and assessment technology platforms. The planning documents provided to the panel reference best-of-breed approaches to software development using agile planning, cloud-native technologies, extensible and modular code, and other approaches. NCES also reports that the Alliance member has been meeting the deliverable schedule and meeting expectations.10

Although NCES staff have some experience in planning enterprise software development, it will be critical that NCES have additional technical staff expertise to provide oversight and guidance for the development of a project of this magnitude. Staff should have the background required to evaluate alternative technical approaches, review requirements for testing components of the system, and inspect testing of software components. Guidance in cloud-based applications, data infrastructure systems, and psychometrics will be needed to ensure that the system performs as needed and can move NAEP to the next generation. NCES reported to the panel that it is consulting regularly with NAEP’s Digital Transition Advisory Council and the State Education Technology Director’s Association, but it is highly likely that additional technical resources will be needed.11

The technical expertise may be a challenge for NCES given that most staff have either psychometric or statistical backgrounds but do not have this technical background. Given the criticality of this system to achieve program goals and cost reduction, it will be important to find staff who

___________________

10 NCES response to Q73b and to follow-up questions (personal communication, August 11, 2021).

11 NCES response to Q73e.

Suggested Citation:"9 Technological Infrastructure." National Academies of Sciences, Engineering, and Medicine. 2022. A Pragmatic Future for NAEP: Containing Costs and Updating Technologies. Washington, DC: The National Academies Press. doi: 10.17226/26427.
×

can fulfill this role. These resources need to have regular review and input into the development of the software.

RECOMMENDATION 9-2: The National Center for Education Statistics (NCES) should ensure that there is adequate internal and external expertise related to enterprise software development to support and oversee the development of Next-Gen eNAEP for both the NCES staff and for staff working for the platform development contractor. This software expertise is substantially different than expertise related to psychometrics and statistics.

An open question is the appropriateness of the budget for this system. The panel was provided a rough breakdown of the current platform development contract between the cost of maintaining the current system, developing the Next-Gen platform, and future maintenance of the new system.12 Yet the NCES reports indicate that most changes will be enhancements to the existing system. Given the high-level functional descriptions provided and overlap between old and new systems, the panel was not able to evaluate the correspondence between the functional descriptions and the deliverables. In addition, the panel itself does not have sufficient expertise related to enterprise software development to evaluate whether the estimated annual average budget of $19.2 million is appropriate for the development plan.

RECOMMENDATION 9-3: The National Center for Education Statistics should seek expert guidance from enterprise application developers and educational technologists who understand assessment technology platforms to evaluate the reasonability of the projected costs for the development of Next-Gen eNAEP.

___________________

12 NCES response to Q73c.

Suggested Citation:"9 Technological Infrastructure." National Academies of Sciences, Engineering, and Medicine. 2022. A Pragmatic Future for NAEP: Containing Costs and Updating Technologies. Washington, DC: The National Academies Press. doi: 10.17226/26427.
×

This page intentionally left blank.

Suggested Citation:"9 Technological Infrastructure." National Academies of Sciences, Engineering, and Medicine. 2022. A Pragmatic Future for NAEP: Containing Costs and Updating Technologies. Washington, DC: The National Academies Press. doi: 10.17226/26427.
×
Page 79
Suggested Citation:"9 Technological Infrastructure." National Academies of Sciences, Engineering, and Medicine. 2022. A Pragmatic Future for NAEP: Containing Costs and Updating Technologies. Washington, DC: The National Academies Press. doi: 10.17226/26427.
×
Page 80
Suggested Citation:"9 Technological Infrastructure." National Academies of Sciences, Engineering, and Medicine. 2022. A Pragmatic Future for NAEP: Containing Costs and Updating Technologies. Washington, DC: The National Academies Press. doi: 10.17226/26427.
×
Page 81
Suggested Citation:"9 Technological Infrastructure." National Academies of Sciences, Engineering, and Medicine. 2022. A Pragmatic Future for NAEP: Containing Costs and Updating Technologies. Washington, DC: The National Academies Press. doi: 10.17226/26427.
×
Page 82
Suggested Citation:"9 Technological Infrastructure." National Academies of Sciences, Engineering, and Medicine. 2022. A Pragmatic Future for NAEP: Containing Costs and Updating Technologies. Washington, DC: The National Academies Press. doi: 10.17226/26427.
×
Page 83
Suggested Citation:"9 Technological Infrastructure." National Academies of Sciences, Engineering, and Medicine. 2022. A Pragmatic Future for NAEP: Containing Costs and Updating Technologies. Washington, DC: The National Academies Press. doi: 10.17226/26427.
×
Page 84
Suggested Citation:"9 Technological Infrastructure." National Academies of Sciences, Engineering, and Medicine. 2022. A Pragmatic Future for NAEP: Containing Costs and Updating Technologies. Washington, DC: The National Academies Press. doi: 10.17226/26427.
×
Page 85
Suggested Citation:"9 Technological Infrastructure." National Academies of Sciences, Engineering, and Medicine. 2022. A Pragmatic Future for NAEP: Containing Costs and Updating Technologies. Washington, DC: The National Academies Press. doi: 10.17226/26427.
×
Page 86
Next: 10 Program Management, Planning, Support, and Oversight »
A Pragmatic Future for NAEP: Containing Costs and Updating Technologies Get This Book
×
Buy Paperback | $20.00 Buy Ebook | $16.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The National Assessment of Educational Progress (NAEP) - often called "The Nation's Report Card" - is the largest nationally representative and continuing assessment of what students in public and private schools in the United States know and can do in various subjects and has provided policy makers and the public with invaluable information on U.S. students for more than 50 years.

Unique in the information it provides, NAEP is the nation's only mechanism for tracking student achievement over time and comparing trends across states and districts for all students and important student groups (e.g., by race, sex, English learner status, disability status, family poverty status). While the program helps educators, policymakers, and the public understand these educational outcomes, the program has incurred substantially increased costs in recent years and now costs about $175.2 million per year.

A Pragmatic Future for NAEP: Containing Costs and Updating Technologies recommends changes to bolster the future success of the program by identifying areas where federal administrators could take advantage of savings, such as new technological tools and platforms as well as efforts to use local administration and deployment for the tests. Additionally, the report recommends areas where the program should clearly communicate about spending and undertake efforts to streamline management. The report also provides recommendations to increase the visibility and coherence of NAEP's research activities.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!