Assessing the Role of K-12 Academic Standards in States: Workshop Summary (2008)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

2 Estimating Costs W ith the complexity of implementing a standards-based account- ability system in mind, the participants turned to the ques- tion of how one might estimate the costs of such a system to states. The committee had identified five framing questions to guide this discussion: 1. What are the major activities states undertake to develop and maintain a standards-based K-12 education system? What is the nature of the costs to states associated with each of these major activities? 2. What are the sources of variation in these costs by state? 3. What are the costs associated with each major activity across select states? 4. How much do state cost estimates vary for each activity? 5. What are the conceptual and technical issues involved in develop- ing empirical estimates of these costs? Margaret Goertz began the session with a proposed framework for considering the costs, providing an analysis of what states actually do in implementing a standards-based K-12 education system (Goertz, 2008). Douglas Harris and Lori Taylor followed with a detailed investigation of the challenges of estimating the costs of this kind of enterprise, as well as empirical estimates of the costs incurred in three states, Florida, North Dakota, and Texas (Harris and Taylor, 2008b).  

10 ASSESSING THE ROLE OF K-12 ACADEMIC STANDARDS IN STATES BOX 2-1 Major Standards-Based Reform Activities 1. Standards-setting • Developing and revising academic content standards • Setting performance standards • Disseminating standards and training 2. State Assessment • Aligning assessment with standards • Item development • Test construction • Test administration • Test scoring • Score reporting • Technical review, validation of the system 3. State Accountability System • Data system (student, school, district) • Reporting (school, district) • Identifying school status, monitoring progress • Other accountability measures (process, etc.) 4. Rewards and Sanctions • Rewards to successful/improving schools • Sanctions for underperforming/failing districts, schools, or students • Intervention for failing schools, districts • Intervention for failing students SOURCE: Goertz (2008). COST ESTIMATION FRAMEWORK Goertz organized her framework around what she identified as the six primary activities that comprise a standards-based reform system, though she noted that others might define the major activities differently; see Box 2-1. Goertz cautioned that the first four activities, which describe the mechanics of the standards themselves and the accountability system, generally receive most of the attention, but that the last two activities, which describe the ways in which the standards and accountability sys- tem may affect teaching and learning, are equally important. Goertz used these six activities as the basis for a discussion of ways in which variation in implementation may affect the costs to states. Looking at the first three activities, she identified several primary sources of varia- tion with cost implications. The frequency with which a state reviews and revises its standards and updates its assessments (as well as the number 

ESTIMATING COSTS 11 of subjects for which there are standards and the number of assessments) is one factor. A second factor is the process used for setting and review- ing content and performance standards, which varies in complexity, in part because of the number of people and groups involved. States may rely primarily on their department of education staff and volunteers, for example, or hire a contractor, use paid experts, or perhaps do all three. With regard to rewards and sanctions (activity 4), there is a large range of approaches and of potential costs. Responses to classifications of schools or districts as falling short of the No Child Left Behind (NCLB) performance targets might begin with instructional audits or needs assessments. Interventions might include developing school improve- ment plans, measures to build capacity at the district level, or professional development in such areas as curriculum and instruction, data analysis, assessment, and leadership. For failing students, states vary in terms of how they determine eligibility and in how they structure and deliver remediation, as well as in how much funding may be available from the state for this purpose. The characteristics of a state also play a significant role. Readily apparent differences, such as the size of a state and the demographics of its student population affect costs in predictable ways, but other factors are important as well. States vary in their mechanisms for funding public education and in the relative share that is paid by districts. States that have had standards-based reform policies for a decade or more are likely to have more firmly established systems and streamlined processes, which reduce costs. States with newer policies may thus have somewhat higher costs. A state’s fiscal health also plays a role, perhaps because education budgets may remain more stable in the absence of economic downturns. A key component of the variation in costs is the number of person hours required to accomplish the tasks involved. Salaries for regular staff, fees for consultants or contractors, and stipends for teachers who take on extra responsibilities are the prime costs for most of the activities. States also incur meeting and travel costs, as well as the costs of providing grants to districts, schools, or regional consortia. These are all costs that can be estimated, but Goertz noted that it is inherently more difficult to estimate the cost of a policy idea, such as standards-based reform, than a specific program. Goertz returned to a point raised earlier, that standards-based reform is a term that can mean different things to different people. In order to estimate costs, one must determine which expenditures should be classed as standards-based reform costs and which are general K-12 edu- cation costs. One must also ask which costs would be incurred by states even without that conceptual approach and which are extra expenditures. Another complication is the task of distinguishing state costs from local 

12 ASSESSING THE ROLE OF K-12 ACADEMIC STANDARDS IN STATES costs. Since states’ education funding formulas vary, the extent to which costs incurred at the local level are covered by state K-12 education fund- ing can vary significantly—which makes it more difficult to compare across states. APPLYING THE FRAMEWORK IN THREE STATES Harris and Taylor began by echoing the points that standards-based reform is a complex idea and that developing cost estimates for an idea is inherently difficult. Thus, in their analysis of the costs they focused on two questions: 1. What costs are now being incurred by the nation to create, update, and minimally comply with standards, assessments, and account- ability under current state and federal laws and rules? 2. What costs would the nation incur if the typical state system of standards, assessments, and accountability were replaced by a single common system? To answer these questions, they first developed estimates of the costs of standards-based reform activities for three states—Florida, North Dakota, and Texas—that they viewed as reflecting a range in terms of size, education spending, and approaches to assessment and accountability. Their notion was that since no one state could be viewed as representative or typical, they could use the data from these three states to compile a cost profile for a “typical state” and use that to calculate costs for the nation. They noted that their estimates are all based on the costs of simply com- plying with all requirements, rather than the costs of meeting goals, such as bringing all students to proficiency. They also cautioned that their work was preliminary and that they could not offer definitive estimates.  The challenge of comparing the costs of the current system with those of a potential common system raises several economic concepts. One important distinction is that between fixed costs, those that are the same regardless of the scale of the program, and variable costs, those that vary depending on the scale. A significant fixed cost of standards- based reform is developing the content standards and setting the per- formance standards: this cost would be the same whether the standards   The committee worked with Diane Massell and her colleagues and with Harris and Taylor to develop a list of states for analysis according to a variety of criteria, including geographic diversity, nature of standards and accountability program, and availability of cost data.   Harris and Taylor subsequently completed their analysis and expanded on it for the second workshop. 

ESTIMATING COSTS 13 were to apply to 100 students or 1 million. However, Harris explained, a common standard is likely to set a higher bar, which is likely to mean significant additional costs. Moreover, a substantial majority of the costs of standards-based reform are variable, and depend on design decisions (including how high the bar is set). Other variables would be the number of subjects and grade levels to be included, the frequency of assessment, the degree of support provided to schools, and the incentives and sanc- tions to be implemented. Another distinction used by economists is that between opportunity costs and expenditures: reforms may switch resources from one activity to another and it is important to include such resource costs in the cost esti- mate, even if they do not result in new expenditures. So, for example, the time that teachers need to spend on testing-related tasks does not require additional state or district expenditures because the teachers would oth- erwise have been engaged in another task and would be receiving their salaries. However, there is a real resource cost in the loss of whatever they might otherwise have accomplished during that time. Harris and Taylor focused on these costs (they used the term “real resource costs” to avoid technical jargon) in order to capture that value. Harris and Taylor highlighted some of the features of the three states’ systems to illustrate more specifically the kinds of variation that affect costs. For example, Florida and Texas have both gone considerably beyond NCLB requirements (by testing at more grades and in more sub- jects), while North Dakota has not. Florida has a bonus program so that districts can give substantial monetary rewards to teachers for improve- ments made by their students, while Texas has a program to identify and accelerate certain students. Assigning the costs of these kinds of policies is complex, Harris explained, because one could argue that they are part of the cost of meeting the requirement—to push students toward proficiency—or one could argue that they are not required elements of a standards-based reform system. Moreover, a state government might reallocate resources in a way that poses a conundrum for cost estima- tors, for example, by switching instruction time from music to math. If making this change is not treated as a cost, the implication is that music instruction has no value. Assigning a dollar value to what is lost is not straightforward, but treating it as an additional cost may unreasonably inflate the overall estimate. Having described some of the issues and assumptions that guided their work, Harris and Taylor presented their cost data. Their principal sources were national databases and data made publicly available by the three states. Their focus was to provide a rough comparative picture of the costs per pupil and per standard (e.g., 3rd-grade mathematics) of devel- oping standards. The authors included both per-pupil and per-standards 

14 ASSESSING THE ROLE OF K-12 ACADEMIC STANDARDS IN STATES TABLE 2-1  Preliminary Cost Estimates for Developing Standards in Three States Florida North Dakota Texas Standards   State $37,853/standard $15,385/standard $21,853/standard   Local $15.31/student N/A N/A Assessments   State-contracts $15.10/student $33.47/student $20.46/student   State-admin $10.21/student $1.02/student $2.38/student   Local $20.72/student $13.26/student $12.60/student Accountability   State N/A $1.31/student $1.60/student   Local $0 $0 $0 NOTE: Estimates are the lower-bound estimates for real resource costs. SOURCE: Harris and Taylor (2008a). costs in order to capture both one-time (initial development) and annual costs (such as those for administering assessments). Their paper provides further detail about the costs they considered. Their preliminary cost estimates for the three states, which are tentative, they emphasized, are shown in Table 2-1. For context, annual expenditures for pre-K-12 educa- tion are $15.5 billion for Florida, which has 2,539,929 students; $711.0 mil- lion for North Dakota, which has 104,225 students; and $28.2 billion for Texas, which has 4,259,823 students (http://www.edweek.org/rc/states/ [April 2008]). None of these three states could be viewed as typical, but North Dakota came closest to approximating the situation for a state that is meeting, but not going beyond, the legal requirements and rules. Harris and Taylor have developed a plan for breaking the North Dakota costs into the fixed and variable categories in order to adjust those estimates for use as the typical state. They had not completed this complex analy- sis in time for the workshop. Nevertheless, their preliminary response to question 1, regarding the costs of current systems, is that the costs of standards-based assessment and accountability systems is a relatively small percentage of total educational costs. Harris and Taylor’s portrait of the complexity of estimating costs and identifying potential savings that could be achieved with a model based on common standards stimulated a range of reactions. Discus- sant Thomas Toch was struck by how small the investment has been. He argued that standardized tests have become, by default, the central driver of standards-based reform, the mechanism that largely determines what is taught and when and how it is taught. Yet of the average $8,000 

ESTIMATING COSTS 15 spent per pupil annually, only one-half of 1 percent goes toward building the tests that are to measure progress toward the high standards that are widely supported. From his perspective, a move to common standards and tests would, by reducing the significant financial burden to states of developing their own standards and assessment programs, free resources for other targeted investments in improving teaching and learning. Discussant Susan Traiman also voiced concerns about the quality of the existing system. She questioned the usefulness of estimating the cost of producing systems that have received relatively low marks for quality, referring not just to the three states that were analyzed, but to all 50. She was also concerned that the approach Harris and Taylor used did not account for relative quality among states’ standards, pointing out that a state with a smaller number of very focused, thoughtfully developed, standards might have a higher per-standard cost than a state with a mul- titude of standards. She argued that a more critical question was what kinds of resources the nation and the states are willing to invest to make sure that all students meet rigorous standards. Other participants added to this point, noting that what would be of most interest would be a sense of the cost of developing common stan- dards that had the effect of helping more students meet high standards. To calculate the cost in that way would mean including the costs for such interventions as professional development courses and new instructional materials. In their paper, Harris and Taylor (2008b) had addressed two addi- tional categories of costs that could be analyzed. The first is what they termed necessary but nonrequired costs, such as periodic assessments used by districts to monitor students’ progress toward the benchmarks as measured by standardized tests. They developed tentative estimates that districts spend an average of $5 to $10 per student on these supplemen- tary assessments. The second category was what they called “apparently new resources,” the costs of making changes in teacher preparation and certification, curriculum, textbook selection, and other system elements in order to align them with standards. While these two additional categories may not capture the full scope of the investment needed to implement standards-based reform that addresses persistent achievement gaps and raises achievement to desired levels, they do point toward additional analyses that could be helpful. Discussant Dave Driscoll pointed out additional costs that could be considered in light of the earlier discussion about resources and capacity. One example is the cost to Massachusetts of releasing every single test item that is used so that the process is completely transparent to students, parents, and the public: doing so increases the cost substantially (other states bank many items for reuse), but it has important political and prac- 

16 ASSESSING THE ROLE OF K-12 ACADEMIC STANDARDS IN STATES tical benefits. Driscoll also noted that “at the beginning we had plenty of money . . . the legislature provided whatever we needed.” Over time, however, budgets have been squeezed. The Massachusetts legislature had initially provided extra funds to support students who were in danger of failing the high school graduation test, but as first-time pass rates passed 80 percent, those funds were discontinued. Discussant Mark Harris considered the capacity issue from a qualita- tive perspective. He observed that although states prize their autonomy and flexibility in developing systems that will best serve their students, many nevertheless base much of their instruction on commercially avail- able programs that have very little link to state standards. These pro- grams are often designed to provide so-called “teacher-proof” curricula and instructional plans and thus do very little to develop the capacities of teachers who use them or to push the state-specific education goals forward. Participants proposed many potential costs that had not been con- sidered in the analysis, while acknowledging both the complexity of developing the estimates and their value as a starting point for discussion. Laurie Wise summed up the message that many drew from the consider- ation of costs with a reminder of the fundamental question: “Is what we are investing in actually helping students to meet these standards, not just in defining and measuring them?”