5
Test Administration: Moving to a Local Model
Test administration for NAEP is expensive. Because it represents about 28.6 percent of NAEP’s budget, test administration presents one of the clearest opportunities for cost savings.1 In this chapter we discuss NCES’s plans to replace the current computer-based delivery model with one that is primarily school based and includes the use of local equipment and internet providers as well as school-based proctoring of the assessment. This approach could produce substantial cost savings, though with potential concerns related to standardization, comparability, equal access, and increased burden for schools. In addition to reducing the costs of regular test administration, local administration could also be used to reduce NAEP’s high costs for pilot testing (see Chapter 4).
This chapter starts with a discussion of the cost of test administration for NAEP. It then outlines the program’s new vision for test administration, followed by a description of the experience with local administration during the era of voluntary state participation in NAEP (prior to the No Child Left Behind Act of 2001). The fourth section addresses the challenges of local administration with computer-based delivery and the flexibility the approach offers. The fifth section considers the way the new local administration model should be reflected in the analysis of NAEP results. The final section discusses the potential for cost savings from local administration.
___________________
1 Using the figures from Table 2-2 (in Chapter 2), $50.2 million is the average annual cost for the data collection and service and support contracts, which is 28.6 percent of NAEP’s current $175.2 million total cost.
The next chapter discusses other ways of reducing test administration costs.
CURRENT COSTS
Test administration is supported by two contracts in the NAEP Alliance, one for sampling and data collection and the other for the support and service center.2 The estimated annual average cost for these contracts is $44.8 million and $5.3 million, respectively. These average yearly costs fall much more heavily in years when the mandated reading and mathematics assessments are carried out, which require the larger samples that support state and urban district results.
The current NAEP assessment model involves sending NAEP-supported staff and devices into sampled schools. The typical cost is roughly $3,500 to $4,500 per sampled school, including the field staff that visit schools, the infrastructure to support that staff, and the devices that are brought to the schools.3 This field work represents an average annual cost of about $36 million for an average yearly sample of 9,000 schools (see discussion in Chapter 2).
In addition, roughly 23 percent of the $21.9 million average annual cost for pilot testing is supported by the sampling and data collection contract for administration of the pilot test, representing another $5.0 million in administration costs each year.4
VISION FOR A DEVICE-AGNOSTIC, CONTACTLESS NAEP
The current administration model for NAEP, which uses professionally trained NAEP staff and contractors to administer the assessment, minimizes the participation burden for local schools and helps ensure quality,
___________________
2 NCES response to Q33: The sampling and data collection contract covers the following activities: “Selects samples; prepares sampling weights; administers assessments and collects data for pilot and field tests, operational assessments, and special studies; and ships completed assessment materials to the scoring sites. Conducts the High School Transcript Study and the Middle School Transcript Study.” NCES response to Q60: Of these activities, sampling accounts for 3 percent of the contract, weighting accounts for 3 percent, assessment field work accounts for 67 percent, transcript studies account for 7 percent, and infrastructure and assessment-related central office activities account for 20 percent. NCES response to Q33: The support and service center contract covers the following activities: “Provides support, training, and resources to state and TUDA [Trial Urban District Assessment] coordinators to ensure the accurate and timely sampling, administration and reporting of NAEP in each state and TUDA district.”
3 NCES response to Q70g.
4 NCES answers to follow-up questions about evidence-centered design task models and item development costs (personal communication, June 24, 2021).
accuracy, and comparability in administration. When NAEP recently moved to computer-based delivery, the use of NCES-provided equipment was intended to reduce the burden on schools while maintaining the level of standardization that is deemed essential for NAEP. It also helped ensure that the assessment could be given in all schools, even those with limited bandwidth and technology resources.
As school staff have recently assumed increasing responsibility for state-sponsored large-scale, high-stakes assessments, which are often administered online using local devices, the current NAEP test administration model seems increasingly outdated and unnecessary. In response to these changes, and in recognition of the large costs associated with NAEP’s current approach to test administration, NCES has outlined a plan to use local staff and devices for administering NAEP. NCES refers to this change as a transition to “contactless administration” because NAEP staff would no longer be directly in charge, turning it over to trained school-based staff.5 NCES is also considering an intermediate “reduced contact” model in which NAEP staff would support test administration either virtually or with fewer in-person staff. NCES recognizes that it may have to provide equipment and proctors to some schools for a number of years and that the exact timing that will be feasible for all schools for this transition is uncertain.
LOCAL ADMINISTRATION IN THE PAPER-BASED ERA
Although NCES’s plans for local test administration represent a change from NAEP’s current approach, the program had extensive experience with local administration during the 1990s.6 Prior to 2002, local education employees proctored NAEP assessments in the trial state assessment portion of the program.
The creation of the trial state assessment in 1990 led to a potentially 15-fold increase in samples, which made the professional administration model untenable given NAEP budgets at the time. However, since participation in the trial was voluntary, states that wanted to participate were asked to contribute in-kind support by supplying staff who could administer the assessment, as well as participate in the necessary training and preparation. Contractor proctors observed and audited 10 percent of the sessions.
The trial state model was used only for subjects and grades for which state results were being reported. Professional proctors continued to conduct
___________________
5 The terms “contactless” and “reduced contact” have acquired other meanings during the COVID-19 pandemic than previous ones. This report follows the convention of “local administration” of NAEP to mean the use of local devices and local school officials as proctors.
6 Descriptions of the operation of the state assessment during these years is based on the experience of panel member Stephen Lazer, who helped lead the work of the Educational Testing Service on NAEP during this period.
all administration in non-state subjects and grades and in private schools. Additionally, the trial state model covered only the trial administrations in a given subject and at a given grade. Since states could choose not to participate in the program, the national results were insulated from non-participation effects by keeping national and state samples strictly separate. In all but the smallest states (where the small number of students precluded two separate samples), there were separate national and state administrations in the same subject at the same grade. Since national samples did not contribute to state results and needed to maintain or allow trend comparisons to years that did not include the trial results, NAEP used professional proctoring for all schools in the national sample.
Technically, the mixed system of the 1990s worked well. However, analysts found a small difference between state and national administration models that persisted throughout the period: in matched samples, performance under the trial state model was slightly higher than in the national model. This difference necessitated an equating step to bring the results of the trial state assessment onto the national scale.
Politically, program officials viewed the “contribution in kind” as acceptable since the program was wholly voluntary. States signed up if they wanted NAEP data. If they did not wish to supply the administrators, they could forego participation in NAEP’s state-level sample.
In 2001, No Child Left Behind changed the situation, with the state NAEP becoming mandatory in reading and mathematics at grades 4 and 8. To avoid having NAEP participation become an unfunded mandate, program officials asked Congress to allocate funds to allow NAEP to expand the national administration model to all schools. Since states could no longer opt out, there was also no longer a need for a separate national sample in reading and mathematics at grades 4 and 8.
CHALLENGES AND FLEXIBILITY WITH LOCAL ADMINISTRATION WITH COMPUTER-BASED DELIVERY
The planned return to local administration in a computer-based era will require local staff to address a set of issues that were absent during the trial state assessment in the 1990s. The process of preparing for assessment administration will require local staff to ensure that appropriate computer equipment and internet connections are available, in addition to helping prepare the student sample, proctoring the assessment administration, and participating in the necessary training. However, computerization over the past two decades has substantially reduced the work that local staff performed in the 1990s to prepare the student sample, read instructions for the assessment, and distribute and collect the assessment books and other assessment materials.
Since the 1990s, NAEP has also moved to include students with disabilities and English learners in its assessments.7 Many of the accommodations to increase inclusiveness for NAEP can be implemented as universal design elements in computer-based assessments, such as providing adjustments for font size or having directions given aloud.8 However, some accommodations, such as providing assessments in Braille or giving instructions in sign language, would require additional support from local and NAEP staff.
NAEP conducted a proof-of-concept study on the use of school-based equipment in 57 Virginia schools;9 it uncovered several problems:
- Communication: staff who were planning administrations had incomplete information about available equipment, network and security configurations, needed setups, and available space.
- Hardware: in some schools, the hardware had low working memory and processing speed, insufficient battery charges, cracked screens, or missing keys.
- Connectivity problems: in some cases, poor connection speeds, lagging and freezing, and access to bandwidth competed with other school demands.
- Technical support: access to technical support was uneven across the studied schools, with some schools needing unavailable help troubleshooting problems and monitoring administration progress.
These difficulties are similar to those seen in other programs that have used or tried to use local equipment to administer large-scale standardized exams (see, e.g., Brown, 2019; Herold, 2016; Strauss, 2020).
At this time, familiarity with the computer technology and its use in assessment continues to advance, particularly with the now-widespread use of computer-based administration for state assessments. The remote arrangements that many schools and districts were able to make during the pandemic to carry out assessment virtually from students’ homes illustrate how far the technology has come, though many barriers remain, and inequities persist (Michel, 2021). In light of the barriers and inequities that were highlighted by experiences during the pandemic, many districts used
___________________
7 See https://nces.ed.gov/nationsreportcard/about/inclusion.asp.
8 See https://nces.ed.gov/nationsreportcard/about/accom_table.aspx.
9 The Virginia proof-of-concept study is described in a PowerPoint presentation by A. Deigan, Exploring eNAEP’s Design. Presented to the NAEP Validity Studies Panel, National Center for Education Statistics, 2021, Feb. 11. The presentation was provided to the panel and is available in the project’s Public Access File. Note that the proof-of-concept study was carried out in a state that has a history of successfully administering its state test online. Other states without such experience may experience greater problems than did Virginia in the local administration of NAEP.
funding from the American Rescue Plan to make further improvements to their technology infrastructure (AASA, 2021).
NCES’s plans for local administration call for school-based staff to conduct readiness checks for available equipment before each administration. NCES will need to create a scalable and efficient process to validate that schools are using equipment that conforms to NAEP requirements. School equipment will need to accommodate NAEP’s innovative item types10 and ensure that test questions and answer options display correctly and load quickly in a consistent manner from student to student and from school to school across the country. These requirements would certainly include detailed specifications for laptop computers and tablets.11 Examples of minimum requirements might cover screen size and resolution, touch screen capabilities, mouse and track pad/ball capabilities, keyboard size and general layout (e.g., not allowing virtual keyboards or enhanced gaming keyboards), memory and processing capabilities, acceptable operating system versions, internet browsers or cloud applications, and bandwidth.12
Furthermore, with a local model, school-based staff will need to provide technical assistance for hardware-related issues, with a school technology coordinator who can serve as the first level of technical support when issues arise during the testing period. Off-site NAEP staff will be needed to address questions that may be out of the scope of school technology coordinators.
The local staff who provide technical support for the equipment and address the software issues will need training. So will the staff who administer the assessments. While traditional models for training have been in person (Hoagwood et al., 2018), the COVID-19 pandemic has accelerated a shift to online training, with improvements in its quality and effectiveness (Lockee, 2021). The use of online training will result in cost saving and simplification compared to the 1990s experience of local administration.
Results from the Virginia proof-of-concept study (Deigan, 2021) indicate that school contexts and access to the necessary equipment differ markedly. NAEP’s local administration plans assume that the model will not be feasible in some places because of limited equipment or other barriers. Since NAEP selects a representative sample of schools to reflect the demographics of the nation, it is important that a high proportion of selected schools and students participate. In cases where the local administration model will be difficult to implement, it will have to be tailored to the local
___________________
11 Given the complications of small devices, the requirement would likely exclude notepads and smartphones.
12 There has been some discussion in the context of eNAEP (see Chapter 9) about developing dedicated cloud-based NAEP test delivery applications to bypass some of the display and interactive limitations of internet browser–based applications.
context, with NAEP-supported equipment and staff provided as needed. The NAEP program may find it efficient to institute a routine process for approving certain schools to automatically receive additional equipment and possibly staff to support the administration. This might include, for example, schools that qualify for free or reduced-price lunch or have large percentages of students who qualify, or schools in rural or remote areas, or other criteria. Augmentation to the NAEP state coordinator program may also be needed to find appropriate ways to support schools.
The additional activities that school staff will need to carry out may also suggest a role for some stipend or other financial support for local administration of NAEP, particularly for schools that are not yet routinely administering their state assessments digitally. Such financial support would help avoid the impression that a shift to local administration is an exercise in cost-shifting from the NAEP program to local schools. At the same time, however, providing substantial support to every participating school could eliminate any net savings from a change to local administration. In addition, the burden for local school staff is likely to decrease over time—with increases in computing power and staff familiarity with digitally based assessment—but it could be politically difficult to end a policy of financial support once established unless it is clearly framed as transitional.
In addition to the challenges described above, the new approach could also potentially provide some flexibility that could help schools in administering NAEP. The current administration approach is designed to minimize costs to the program by using the NAEP-supported proctors and equipment as efficiently as possible. As a result, the current model simultaneously assesses as many sampled students as possible in a school. However, with a local administration model, schools could administer assessments to sampled students over a multiweek window, which would allow a large number of students to be tested on a small number of machines available in a library or media center. This small-group approach to administering would require a somewhat different approach for local proctoring and program auditing. Although this approach would not appreciably affect NAEP program costs, the flexibility could substantially simplify the difficulties some schools may have in administering NAEP by avoiding the need to provide large rooms equipped with many computers that meet NAEP’s requirements.
RECOMMENDATION 5-1: The National Center for Education Statistics (NCES) should continue to develop its plan to administer NAEP using local school staff as proctors with online assessment delivery on local school computers, with development and bridge studies as needed to understand the feasibility and effects of this change in different contexts. This new model should be accompanied by adequate training and support of school staff, including tailored support for schools with
more limited resources that may need NCES to provide proctors and equipment. NCES should also explore the use of flexible administration windows to allow schools to develop plans that accommodate local constraints on available equipment and consider appropriate ways to compensate local schools for their contributions to the administration, especially during the transition to this new model.
RETHINKING STANDARDIZATION WITH LOCAL ADMINISTRATION
NAEP has traditionally taken a strong view regarding standardization by providing both the test materials and proctors to each testing site, and, currently, also the software, computers, and network equipment for administering the assessments digitally. In addition to the expense involved, high levels of standardization may actually have adverse consequences regarding NAEP’s generalizability and utility in the presence of ubiquitous and ongoing technological changes in teaching, learning, and assessment.
Standardization in testing implies that as many of the important conditions of measurement are held constant as practicable. Those conditions are usually introduced as constraints that include administering a fixed-length test form comprised of the same or highly similar test items to all examinees using the same mode of delivery, as well as following a consistent set of item formats, time limits, and test administration instructions (American Educational Research Association, American Psychological Association, and National Council on Measurement in Education, 2014). The two chief motivations for standardization are fairness—operationalized as applying consistent conditions of measurement for everyone—and the need for score comparability over time.13 However, conditions of measurement will undoubtedly change as the constructs evolve and assessment technologies change.
There is also no guarantee that strong standardization that penalizes examinees who are unfamiliar with one or more of the conditions of measurement is actually fairer than universally customizing the testing experience to equally facilitate all examinees. Testing students on familiar technology may allow them to put forth their best performance.14 For example, the provision of a word processing application developed exclusively for NAEP testing cannot guarantee fairness and comparability when the
___________________
13 Score comparability across conditions only holds if test takers are assigned under all conditions or at least randomly assigned to various conditions of measurement. Holding them constant does not allow comparisons across conditions.
14 Note that this point is closely related to the arguments made in support of providing accommodations in assessment.
features, functionality, and interface may differ from what (some) examinees use in their everyday learning activities. Way and Strain-Seymour (2021) summarize the research related to 17 different device-related factors that may affect student performance on NAEP.
As noted above, the move to local test administration assumes that NAEP will develop some minimum specifications for devices, operating systems, network configurations, and connectivity that can be used to administer the assessments. These specifications will allow the assessment to be administered in a relatively common way, while falling short of the complete standardization that NAEP currently enforces by providing its own equipment. There will inevitability still be substantial variability across classes of equipment, operation systems, and network configurations that meet NAEP’s minimum requirements for local test administration. This variability reflects both the practical reality of using local devices and the necessary customization to allow students to use devices that are familiar to them.
To allow NAEP to account for the effects of equipment variability in the analysis of assessment results, the program will need to collect detailed information from the testing sites about the equipment and the operating systems used. This can be done as part of the readiness checks that are performed to ensure that the equipment meets the minimum requirements. The device characteristics can then be used to develop categorical classes of equipment and systems to use during analysis of the assessment results (Luecht, 2005, 2006, 2016). Such analyses would include using item response theory to evaluate item and person data-model misfit and carrying out residual analysis. Influential differences in different classes of equipment could then be incorporated into the assessment modeling and calibration framework, though it would be important to consider the potential effect of any correlations between classes of equipment and student characteristics and contexts. Modeling the variation across equipment classes as random effects would allow the resulting estimates to reflect generalization across devices, which is the construct of interest for NAEP since the frameworks are not focused on device-specific competencies. Furthermore, the program can use the results related to different equipment classes to update the program’s equipment requirements over time and to calibrate results across years as the mix of devices changes.
RECOMMENDATION 5-2: Since a key component of moving to local administration will be the development of minimum requirements for equipment, operating systems, and connectivity, information about local devices and administration conditions will have to be included in the data collection. Analysts should use statistical techniques that account for the effects of differences in devices and other local conditions
to produce estimates that generalize across those differences. The National Center for Education Statistics should explore the use of random effects and other statistical techniques to produce estimates that reflect generalization across devices.
ANTICIPATED COST SAVINGS FROM LOCAL ADMINISTRATION
NCES estimates that development costs for the transition to an online, device-agnostic, contactless model will total $18 million. These development costs involve a series of proofs of concept and field test studies to examine the use of two hardware options—NAEP-provided non-touch screen Chromebooks and school equipment—with reduced field staff. The estimated development costs also support the cost of a bridge study in 2024 to look at the transition between NAEP-provided touch screen Surface Pros and NAEP-provided non–touch screen Chromebooks.15
Initially, NCES expected the new model to save $52 million from 2026 through 2030 for the mandated assessments in reading and mathematics in grades 4 and 8, though that projection has evolved as the development work continues.16 In the latest estimates, as this report was being prepared, NCES projects that the proportion of schools using local administration will grow from 40 percent in 2026 to 67 percent in 2028 and to 80 percent in 2030.17 This growth in the projected proportion of schools using local administration is accompanied by a reduction of per-school administration costs of roughly 20 percent in 2026, 32.5 percent in 2028, and 37.5 percent in 2030, in comparison with the current baseline.18
The NCES estimates were based on an initial sample of 15,500 schools for the mandated assessments, which would produce an administration cost of roughly $62 million in the baseline year of 2024.19 The reductions in administration costs would imply total reductions compared to the baseline starting with $12.4 million in 2026, and then $20.2 million in 2028, and $23.3 million in 2030. These projected savings total about $56 million from local administration.
___________________
15 NCES response to Q57c. NCES does not expect to require bridge studies to move from NAEP-provided non–touch screen Chromebooks to school equipment or to move to reduce contact or contactless administration.
16 NCES response to Q57d and cost driver PowerPoint provided by NCES (personal communication, May 13, 2021).
17 NCES response to Q70a.
18 Cost-per-school figures from NCES (personal communication, November 10, 2021). The current estimated costs per school for administration are $2,700 to $3,700 in 2026, $2,200 to $3,200 in 2028, and $2,000 to $3,000 in 2030. We used the midpoint for each year in calculating the projected reductions.
19 Cost-per-school figures from NCES (personal communication, November 10, 2021).
It is not clear to the panel why local administration is projected to reduce costs by less than half for the schools in which it is used. The panel did not have access to a breakdown between the different types of costs related to the field teams—combining the field staff, infrastructure, and devices that support the schools. However, a local administration model would be expected to substantially reduce all of these major costs, and we do not understand what major new costs could arise that would be half again as large.
NCES notes that training costs could be higher with local administration.20 This is possible, but it seems that this extra cost should be modest, given the likelihood that remote instruction can be used for training.
NCES also notes that auditing would be done in some schools, but that would clearly involve a small portion of the schools and perhaps only a single NAEP staff member rather than a team of several people.21 NCES also notes that increased help-desk support would be required, but again the level of cost required for such support would be expected to be substantially lower than sending teams to each school to administer the assessments.22 Even if NAEP continued to have some staff onsite or available electronically rather than unassisted local administration, fewer NAEP staff members would be used for each school, and there would still be expected savings related to devices.
NCES does not currently expect to provide payments to schools for local administration as reimbursements for costs or incentives for participation.23 Given the increasing familiarity with administering assessments online that is likely over the coming decade, the panel agrees with this position over the long term; however, as noted above, it might be reasonable to consider some stipend for schools during the initial transition to local administration when the approach is relatively new.
After schools gain experience with local administration, the panel expects substantially larger savings are possible than are suggested by current NCES estimates, especially when considering increased familiarity by 2030 with computers in general and computer assessment in particular across the entire education system. In addition, the panel expects that it would be reasonable to extend the local administration model to the full set of assessments, substantially reducing the average annual administration costs of $36 million and the average annual pilot administration costs of $5.0 million.
___________________
20 NCES response to Q57d.
21 NCES response to Q70c; the response suggests that perhaps 10 to 15 percent of schools with contactless administration would be audited.
22 NCES response to Q57d.
23 NCES response to Q57d.
As an initial approximation, the panel estimates that the program can reasonably aim for a percentage reduction in administration costs that is much closer to the percentage of schools using the local administration model. This would suggest an expected annual savings closer to 80 percent of the current administration costs if NCES expects that 80 percent of the schools can use local administration. This estimate in turn suggests an estimated annual savings of roughly $28.8 million for assessment administration and $4.0 million for pilot testing by 2030. The total estimated savings of $32.8 million represents 18.7 percent of the current NAEP budget.
NCES notes24 that the adaptations for assessment administration postCOVID-19 may suggest substantial increases in administration costs that are not yet understood. It is important to note that a widespread use of local administration is likely to reverse these extra cost increases, since special procedures for going into the schools will not be necessary if NAEP staff do not go into the schools. As a result, there may be large new administration costs in the next few years that would be mirrored by equivalent large decreases in administration as the transition to local administration proceeds.
RECOMMENDATION 5-3: The National Center for Education Statistics (NCES) should review its estimates of the potential cost savings from local administration of the mandated assessments in reading and mathematics in grades 4 and 8. The estimated savings are unexpectedly small when local administration would largely eliminate the large current costs for traveling proctors and equipment, even after considering any offsetting additional costs for training and technological infrastructure. NCES should also consider the use of the local administration model for reducing costs of all other assessments, as well as the costs for the pilot administration of new items.
___________________
24 Information from NCES (personal communication, November 10, 2021).
