10
Program Management, Planning, Support, and Oversight1
This chapter addresses NAEP’s overall program management, planning, support, and oversight functions. These functions include a wide array of tasks that are important to the program but not directly related to specific assessment components: defining the program’s direction; overseeing and coordinating the staff, contractors, stakeholder representatives, and experts who guide and implement the program; and carrying out the necessary planning functions to define innovations in the program and coordinate existing work. The planning functions related to innovation often involve some research and development, which are included with the overall management costs. Program management, planning, support, and oversight includes the functions played by NAGB and NCES staff, by NAGB members, by the many participants in NAEP stakeholder and expert advisory groups, and by many Alliance contractors.
Program management, planning, support, and oversight are not an explicit focus of the panel’s statement of task (see Chapter 1), though it is mentioned in its call for “programmatic changes and research needed for NAEP to explore innovations while balancing the competing objectives of cost reduction, technical quality and informative value” [emphasis added]. The panel initially considered these functions in precisely the supporting role suggested by the statement of task. However, after reviewing the program’s overall cost structure, the panel concluded that potential reductions
___________________
1 After a prepublication version of the report was provided to Institute of Education Sciences (IES), NCES, and NAGB, this chapter was edited to reflect a broader range of costs and to revise the description and estimate of the costs associated with the non-support contracts.
in the cost of program management, planning, support, and oversight functions would play a critical role in any cost reduction strategy for NAEP.
The members of the panel were chosen with a view to address a request focused on assessment and potential technological innovation, not a request focused on organizational reengineering. As a result, we have limited our discussion of these issues to areas for which the panel members are familiar with relevant literature. As our recommendation below indicates, this topic needs further attention.
After discussing the costs related to program management, planning, support, and oversight, the chapter briefly discusses the importance of taking a systemic approach to the overall design of the program and considering the role of research and innovation in the program.
CURRENT COSTS
The cost of NAEP program management, planning, support, and oversight is divided across a number of different budgets and contracts and is therefore hard to understand. These costs are of three different types:
- Federal program staff: The direct and indirect costs related to the federal government employees who staff the NAGB and NCES program offices for NAEP are not separately identified in appropriations. Overall, there are roughly 32 federal employees who primarily support NAEP for either NAGB2 or NCES. Table 2-2 (in Chapter 2) provides an estimated annual average cost for these staff of $7.1 million.3
- NAEP support contracts: NAEP includes several different contracts that provide various sorts of management and support functions. In the NAEP contract summaries provided by NCES, these contracts are identified as program management support or support contracts, with estimated annual average costs of $6.2 million and $37.0 million, respectively.4
___________________
2 As a legislatively established independent entity within the Department of Education, NAGB’s annual appropriation is uniquely set up as separate line item within the IES budget. The appropriation does not segregate salaries and expenses from all other costs. The annual appropriation, $7.1 million in fiscal 2021 for example, covers not only salaries and expenses, office rent, and contract costs, but all other aspects of NAGB’s operations as an independent entity. NAGB (personal communication, March 15, 2022).
3 This figure underestimates the cost for NCES staff by including only salaries.
4 NCES response to Q33: The “program support management” contract covers the following activities: “Manages the NAEP program in alignment with Project Management (PM) best practices to ensure proper scheduling, quality control, risk management and communication across contractors and handoffs.” The “support contract” category actually includes several different contracts over the years that have been combined for the historical analysis under
- Costs within the non-support contracts that reflect various aspects of management, planning, and oversight.
The panel was not able to obtain sufficient details of these program-related costs in relation to the various staff and contractor activities. No doubt, some of these costs reflect the complex governance arrangement for the NAEP program, and some reflect the complex design of the Alliance contracts.
The panel also does not have good estimates of the relevant costs across the non-support contracts. Based on the very limited information we have available, these costs are plausibly at least in the range of 10 to 15 percent of the respective budgets.5 Spread across the $125 million of non-support contracts, that range would suggest potential additional costs of $12.5 to $18.8 million.
Overall, the panel roughly estimates the cost of program management, planning, support, and oversight for NAEP from these first two types of costs—federal staff and support contracts—is about $50.3 million or about 28.7 percent of the total budget for NAEP. Related costs in the non-support contracts might add substantially more. While acknowledging the complexity and importance of NAEP, the panel thinks these costs are very large, both in absolute terms and as a percentage of the overall NAEP budget. Meaningful cost reduction for the NAEP program will need to include a consideration of potential reduction of these costs.
TAKING A SYSTEMIC APPROACH TO DESIGNING ASSESSMENT PROGRAMS
Large-scale assessment programs like NAEP are complex endeavors, involving many activities, including test development, psychometrics, and reporting. Each activity is highly technical and operationally challenging. The integration and implementation of all the activities is even more complex and challenging. The assessment community has a groundbreaking stream of research looking at systemic approaches to assessment design, going under such labels as “evidence-centered design” (Mislevy, 2006; Mislevy, Steinberg, and Almond, 2003), “assessment engineering” (Luecht, 2012a, 2012b, 2020a, 2020b; Luecht and Burke, 2020), and, more generically, “principled approaches” (Ferrara et al., 2017). The NAEP program
___________________
the label of “planning and coordination” (NCES response to Q72a) and cover the following activities: “Ensures coordination among Alliance contractors; maintains data for tracking program progress; and provides logistical support for complaints and substantive comments.”
5 The planning portion of the item development contract is in this range (NCES, personal communication, March 15, 2022).
is already capitalizing on some of this literature, and we recommend that more be done along these lines.
However, the complexity of NAEP’s assessment activities is further compounded by the program’s organizational structure: separate federal agencies responsible for the program’s policies and administration, and implementation of the assessment program by a complex team of contractors. This organizational complexity comes from a clear logic: two agencies to play two different roles, and a contract structure that allows separate companies with expertise in different areas to bid on the work in areas for which they have expertise. However, this seemingly logical structure then produces predictable overlaps and inefficiencies in a program in which decisions about one activity can have implications for all the other activities. This produces the set of review bodies and advisory committees noted in Chapter 2. In caricature: everyone who is involved with one part of NAEP needs to review everything else related to NAEP and to keep up with everyone else’s reviews.
The costs related to this organizational structure need to be addressed systemically. The kind of systemic thinking that NCES is applying to assessment design needs to be applied to the complexity of NAEP’s organizational structure. The high cost of NAEP program management, planning, support, and oversight affects every other part of the assessment.
The panel is only slightly familiar with other research literatures that call for systemic thinking for solving organizational problems. For example, Steiber (2014) lists a systems approach as one of the six key management principles for successful, continuously innovating firms. Although this panel is not constituted to conduct a thorough evaluation of the organizational structures that give rise to such costs for program management, planning, support, and oversight, it is clear that the challenge of NAEP’s costs cannot be addressed by looking only at the activities that directly relate to developing, administering, and reporting on the assessments.
RECOMMENDATION 10-1: The National Assessment Governing Board and the National Center for Education Statistics should commission an independent audit of the program management and decision-making processes and costs in the NAEP program, with a charge and sufficient access to review the program’s costs in detail. That audit should include proposed ways to streamline these processes.
The activities that are grouped together under program management, planning, support, and oversight include many of the research activities that are carried out to support the planning process. This research provides the basis for launching the development of new innovations and monitoring their progress and impact as they are implemented. Examples include
studies related to specific topics addressed in this report, such as computer-based delivery, automated item generation, automated scoring, and the development of eNAEP.
Strikingly, however, there is no way for the panel, or anyone else, to see an integrated summary of these research activities with respect to both their cost and their coverage and vision. This absence is particularly noteworthy given the role NAEP seeks to play as an exemplar for other assessment programs: those other assessment programs could benefit substantially from understanding the kinds of innovations NAEP has identified to explore and the lessons the program has learned from those explorations. And the absence is particularly striking in considering the task that faced this panel to provide advice on valuable innovations for NAEP to consider for the years ahead.
RECOMMENDATION 10-2: The National Center for Education Statistics should increase the visibility and coherence of NAEP’s research activities to help NAEP’s stakeholders, as well as other assessment programs, understand the innovations the program is investigating and the lessons it is learning. The NAEP research program should have an identifiable budget and program of activities.
Such a research program might include research related to evaluation, validity, innovative assessment items, and new assessment technologies.
This page intentionally left blank.
