3
Decisions Required to Compute the Indicators
Dropout and completion rates are among the most basic indicators of the effectiveness of a school, a school district, or a state education system. Although calculating the actual rates may seem simple, judgments must be made about who to include in the base group of students being tracked (i.e., the cohort), who to count as a graduate or a dropout, how many years to follow these students, and how to construct the formula. Each decision substantially affects the resulting rates. A number of reports have contrasted popular calculations in terms of their accuracy, bias, and ease of computation (e.g., Mishel and Roy, 2006; National Institute of Statistical Sciences and the Education Statistics Services Institute, 2005; Swanson, 2003; Warren, 2004). The technical definitional issues that are embedded in those calculations are at least as deterministic of bias and accuracy as the computations.^{1}
Dropout and completion rates are based on a ratio, and one of the first decisions that needs to be made is how to define the numerator. Depending on the type of rate, the numerator represents dropouts, completers, or graduates. Identifying the students who are graduates is somewhat straightforward—they are the diploma earners. Identifying the students who fall into the other categories and distinguishing between graduates and other completers are more difficult. The second key decision is how to define the denominator, the full cohort of students with the potential of graduating or dropping out. Decisions
about how to include transfer students and students retained in grade in the cohort can have a major impact on its size and on the rate that is calculated.
At the workshop, Elaine Allensworth, with the Consortium on Chicago School Research, made a presentation about the kinds of decisions that must be made and the ways that they affect the calculation of the rates. This chapter is drawn from her paper (Allensworth, 2008).
DECISIONS AFFECTING THE NUMERATOR
Distinguishing Between Graduation and Completion
There is generally little ambiguity about who has obtained a high school diploma—lists of graduates must be maintained. However, there are multiple means of completing school besides earning a regular diploma, and not all students complete high school in the same time span. Accounting for different forms of completion has become increasingly complicated. In addition to granting diplomas, schools may offer programs that award General Educational Development (GED) credentials, or students may receive GEDs outside a K-12 school system.^{2} Some districts offer alternative schools for students who have dropped out or are at risk of dropping out, and these schools may grant diplomas based on easier standards than the regular high school standards.^{3} A number of states offer different levels of diplomas or grant certificates of attendance for students who do not pass state-required examinations or meet other criteria (Laird et al., 2008). Growth in home schooling and distance learning further complicates the ways in which completion is defined (National Institute of Statistical Sciences and the Education Statistics Services Institute, 2005).
There are a number of valid reasons for producing graduation rates that include only students who earn a regular high school diploma. There is little evidence that alternative methods of completion benefit most students who pursue these pathways to finishing high school. As noted in Chapter 1, GED recipients do not usually realize the same economic outcomes as high school graduates and are less likely to pursue higher education. Also, there is substantial variability across states and districts in the extent to which other forms of completion are available and recorded in an accessible manner.^{4} Constraining
completion rates to diploma recipients provides some degree of consistency across states and districts with different policies, programs, and data-recording procedures. It also helps maintain stability over time; that is, more comprehensive completion rates might increase over time simply because of an expansion in the availability of alternative forms of completion.^{5} For purposes of accountability, counting only diploma recipients prevents perverse incentives for schools to push students out of regular programs into alternative or GED programs.
Constraining completion indicators to diploma recipients also has some shortcomings. Diploma requirements are not equivalent across states and districts, and they may change over time, making them not equivalent across different cohorts of students, even within the same school district.^{6} In addition, when used for accountability, rates that include only graduates provide no incentive to districts to offer recovery programs for students not likely to obtain a regular diploma. Diploma-based rates also cannot be calculated for schools that offer alternative credentials, and thus, their effectiveness cannot be compared with schools that offer regular diplomas.
Timing of Completion
It may seem desirable to include all completers in the rate, regardless of how long it took them to complete their education. If the rates are being used as a tool to encourage good practices, it makes sense to allow a long time for completion so that schools do not have an incentive to give up on students who do not complete high school in four years. Time limitations for completion are problematic for students with disabilities: those whose Individualized Educational Plans (IEPs) specify that they remain in school beyond age 18 or who are ungraded and whose time in high school is not clearly defined. Time limitations may exclude summer graduates or students who take just one extra term to complete school. Practically, each extra year that is incorporated into the statistic means an additional year’s wait before the completion rate can be reported. Whether the statistics are used for accountability, program evaluation, or assessment of the current state of educational attainment—it is generally critical to get timely information.
equivalency recipients, and other). The variability in defining completers is notable across states (Mishel and Roy, 2006). |
Time limits bring questions about how to count students still enrolled in school. Counting them as noncompleters maintains the definition as a four-year rate. However, this may not accurately represent the true state of high school graduation in the school or region. For instance, at the workshop, Mel Riddile, with the National Association of Secondary School Principals, noted that in some school districts, a student who does not graduate in June but finishes incomplete coursework over the summer, so that the diploma is awarded in July or August, is not always counted as a graduate in four-year rates. An alternative is to exclude students still enrolled from the indicator—removing them from both the numerator and the denominator. This allows the completion rate to be the opposite of the dropout rate, which is conceptually easier for a wide audience to understand. This also avoids any incentives to push out students who can no longer be counted as graduates. However, excluding large numbers of students from the calculation can be difficult to justify if there is concern that all students be counted. Furthermore, rates have the potential to be manipulated by delaying classification of students as dropouts.^{7}
Distinguishing Between Dropouts and Transfers
Determining which students are dropouts is much more difficult than determining which students are graduates or completers. While students can graduate or complete a program only once, they can drop out of school repeatedly and from different school systems, making it difficult to know how to classify them at any given point in time.
Students who drop out often have high absentee rates. It is not always clear at which date these students should be considered dropouts, since they may have been absent for long periods of time before officially being dropped from the enrollment list. Some students never show up for high school. If a student finished grade 8 but does not officially enroll in high school, should the student be considered a dropout from the high school he or she never attended? The National Center for Education Statistics (NCES) requires schools to define enrollment on October 1. How are students classified if they leave grade 9 during September, after less than a month of school?
It is challenging to track down students who simply stop showing up at school. The student may have transferred to another school system, but verifying the transfer is difficult and requires administrative resources. Schools may miscode dropouts as transfers, even though their status is actually unknown,
a practice that leads to underestimates of the true dropout rate. Of particular concern is the way that schools code students who leave school to pursue a GED credential. Some may miscode them as transfers, particularly if they leave to enroll in a GED-granting institution, and as a result may eliminate them from both the numerator and the denominator when calculating rates, a practice that also leads to underestimates of the true dropout rate. Generally, students who leave school to pursue a GED credential should only be considered as transfers if they enroll in a program that grants diplomas (and in this case removed from the denominator of the rates). This tendency may be more prevalent when the rates are used for accountability purposes. For this reason, the National Forum on Education Statistics (2005) encourages states to require validation of transfers to verify (usually through a transcript request) that the student has enrolled in another school that grants regular diplomas.
Policies that require unverified student leavers to be coded as dropouts provide a substantial incentive for schools to obtain valid records of transfers. However, validation costs money,^{8} and schools that have the greatest demand for validation of transfers—those with large numbers of mobile students—may not have the necessary resources. In general, requiring validation will inflate dropout rates and decrease completion rates, because students will be classified as dropouts until their transfer status can be verified. For instance, at the workshop, Rob Curtin, with the Massachusetts Department of Education, described a policy change in his state that required that all students coded as in-state transfers to be verified or, if unverified, to be coded as dropouts. This policy change produced a half-percent increase in the dropout rate in the first year it was implemented. Statewide data systems with individual student identifiers are useful for in-state validation, but they do not remedy the problem for across-state and out-of-country transfers.
Another issue is how to classify students who are incarcerated or otherwise institutionalized. It may seem logical to classify these students as transfers since the decision to leave their school was not voluntary, and the receiving institutions should provide them schooling. However, there is often no verification that incarcerated students are enrolled in a program leading to an accredited high school diploma or that they plan to re-enroll in a regular high school when released. Therefore, they are sometimes counted as dropouts, unless they reenroll in school within the time frame being studied.
Finally, decisions must be made about how to classify students whose leave status changes over time. There are many potential paths to eventually becoming a dropout or a completer (Pallas, 1989). Most problematic is that students who have already been counted as dropouts may re-enroll and drop
^{8} |
For a discussion of some of the issues around validation and data requirements, see the final report of the National Institute of Statistical Sciences and the Education Statistics Services Institute’s (2005, Chapter 6 and Appendix D) task force on graduation, completion, and dropout indicators. |
out a second or third time. For instance, at the workshop, Robin Taylor, with the Delaware Department of Education, reported that in her state 3 percent of the dropouts left school multiple times.
It is also unclear how to count dropouts who have re-enrolled in a different school or entered a school system through an alternative school. Since they enrolled as dropouts from another place, it may not be fair to the receiving school to count them as regular transfers into the new school or system. Problems of re-enrollment overlap with issues of transfer between schools, and the ways in which they can be handled depend on the way that the group or cohort being measured is defined in the first place.
DECISIONS AFFECTING THE DENOMINATOR
Defining the Base Grade of the Cohort
The cohort represented by dropout and completion indicators is almost always defined by grade, such as students entering grade 9 for completion statistics and students in grades 9-12 for dropout statistics. One complication posed by defining the base group by grade level is that students repeat grades. Grade 9, in particular, is frequently repeated, particularly by students who eventually drop out.^{9} Thus, statistics defined by grade level often double- or triple-count students in rates representing different cohorts or years, and the students who are double-counted are those who are least likely to complete school. If students who are repeating a grade are not removed from the calculations, the statistics will be biased against schools with high levels of grade repetition. Because these schools are also likely to have high dropout rates, this accentuates the real differences in dropout/completion rates across schools and can make issues of equity appear more extreme than they really are (Mishel and Roy, 2006; Warren, 2004).
Even grade retention prior to high school can introduce problems in calculating the rates because a student’s probability of dropping out is highly correlated with age.^{10} If larger numbers of students enter high school at age 16 than in the past, dropout rates will rise simply because a larger percentage of the students are at an age when they are permitted to drop out. These age
effects occur beyond any effects of grade retention, which itself increases the probability of eventually dropping out of school (Alexander, Entwisle, and Dauber, 2003; Allensworth, 2005a).
Changes in grade retention patterns prior to high school can also affect the comparability of grade 9 cohorts over time by delaying the entry of the academically weakest students. If larger numbers of students are retained in elementary school one year (e.g., because of the implementation of promotion standards), the dropout rate for one freshman cohort may improve simply because many of the cohort’s low-achieving students have been moved into the next cohort. These kinds of policies affect the comparability of grade-based cohort rates over time and over cohorts.
Schools that have unusual grade configurations may not always be included in rates that define cohorts by grade. For example, middle schools that contain grade 9 but not grade 12 would not produce a graduation rate, even though they have a full grade 9 class. Incorporating ungraded special education students into grade-based cohorts is also problematic.^{11}
One way to deal with these issues is to define cohorts by age instead of grade level. Age-based cohorts are not affected by patterns of grade retention in years prior to or during high school. This makes the rates more sensitive, accurate, stable, and unbiased than when the cohort is defined by grade level. In addition, students who never make it to high school can be included in the rates with their age peers, allowing the statistics to include middle grade dropouts. Special education students can also be included with their age peers, even if they are in ungraded classrooms. For districts and states, age-based rates can be more useful than grade-based rates to gauge trends over time and to assess the current state of education. However, they are not useful as indicators of school performance, since schools, parents, and the public generally want to know how many of the students who attend a school in a given year graduate or drop out later.
Specifying Enrollment Timelines
Defining the cohort also requires consideration of when the student enrolled in school. Because students could enroll in a school after a set date or leave school before that date, it is more inclusive to define the group as including any student enrolled over the period of a year (e.g., from October 1 of one year to September 30 of the next year) than to choose one date in the year for the base enrollment (e.g., enrolled by the 30th day of school). Most often, schools define school year enrollment from either October to October
or June to June. Choosing one over the other has little effect on the resulting rates, but it does affect their interpretation—as students who dropped out in the school year versus students who were enrolled one year and did not return by the second year.^{12}
Indicators that base their denominator on students enrolled on a specific date are particularly susceptible to undercounting students who transfer between schools. Those that compare the number of dropouts or completers over a set period (e.g., the school year, the next four years) to the enrollment at the set date will include students in the numerator who are not in the denominator. For instance, any student who enrolled in a school after the date that defined the cohort and then dropped out will be included in the numerator but not in the denominator. This leads to the potential for dropout rates that exceed 100 percent. In contrast, students could be included in the denominator who left the school soon after the cohort was defined and have no chance of being in the numerator, deflating the statistic.^{13} If in- and out-mobility rates are similar, and if the students who leave have the same probability of completing/dropping out as the students who enter, then this does not result in bias or inaccuracy. However, these latter conditions do not always hold.
Adjustments can be made to account for in- and out-mobility so that the numerator remains a subset of the denominator. However, some of these adjustments require demographic assumptions that, if untrue, can introduce additional inaccuracy into the rates. Warren (2004) demonstrates how inaccuracies can be introduced when assumptions about in- and out-mobility, cohort size, and grade retention do not hold. In addition, adjustments that account for demographic instability must also assume that students who leave a school are similar to students who enter in terms of their probability of completion/dropout. This is often not the case when schools have different enrollment policies. If the students who leave a school are qualitatively different from the students who enter in a way that is systematically related to dropout/completion, the rates will be biased, even after adjustments are made to the denominator.
An alternative to defining the cohort on a specific date is to include all students enrolled at any time during the school year, or at any time over the four years of high school, for completion rates (with students entering at higher
grades in each year after grade 9).^{14} However, the data requirements are higher for this more inclusive definition; that is, schools must maintain individual-level records of students’ dates of enrollment and departure. Furthermore, this definition results in students being counted at multiple institutions; if statistics are aggregated to higher level units (such as the district or state level), mobile students will be double-counted in the aggregate rates.
Decisions also must be made about whether to include students who are enrolled for a fraction of a year. Should schools be held responsible for students who enter and drop out within a week? If no minimum time period is set, including all students enrolled at a school in the denominator will generally increase dropout statistics because mobile students are more likely to drop out than other students. However, if a sizeable minimum time period is used to define who is a student (e.g., all students who enter after October 1 and remain through the end of the year), then dropout rates will be depressed, since, by definition, new students are included in the indicator only if they stay in school. In the end, issues of transfer and mobility are made no easier by including students enrolled over a period of time versus a snapshot; while the indicators are more inclusive, issues of aggregation and data requirements are more problematic.
Handling Transfer Students
When a student transfers between schools, it is not clear which unit should take responsibility for that student. In theory, there are four options: attribution to the receiving school, the sending school, both, or neither. In practice, schools tend to mix classification decisions depending on students’ final status. For example, the graduation rate for the state of Illinois counts transfer students who graduate with their receiving school but those who drop out to neither school. Most of the graduation rates based on aggregate counts of students (including the Cumulative Proportion Index and the graduation rate in the Common Core of Data) attribute transfers who drop out to their sending school, but those who graduate to both schools—as a graduate at their receiving school and a nongraduate at their sending school. Mixing attribution inherently produces bias in the statistics.
Attribution to Both Schools
Counting students at both their sending and receiving schools is the most inclusive method for dealing with transfers. It ensures that all students are
counted. However, it also ensures that transfer students will be counted at least twice—and more than twice for highly mobile students. This is problematic when school-level statistics are aggregated to produce district- or state-level rates because the mobile students (who are most at risk of dropping out) will be overcounted. Counting students at both schools also requires that schools share information or that information on graduation/dropout is maintained in a way that students’ final status can be attributed to their first school.
An alternative might be to weight students’ contribution to their school’s rates by their time enrolled. This alternative is even more data intense, since it would require information about students’ dates of entry and departure at each school to be available and shared between schools. It also would require strict guidelines for defining a dropout date for students with many absences to define the percentage of time enrolled at the final school. It is doubtful that such a method could be practically implemented.
Attribution to Neither School
Counting transfer students at neither the sending nor the receiving school has the obvious disadvantage of excluding all mobile students, which would produce an overly optimistic picture of the state of education at all levels of aggregation. This option eases calculation and data requirements, because no knowledge of final status is necessary. There are also political advantages, since schools do not feel they are being held accountable for problems created in other schools. In practice, only nongraduates are at risk of being excluded from the statistics of all schools—schools tend to want to count all of their graduates. This biases graduation rates upward, leading to more bias than if all transfer students were simply excluded.
Attribution to the Receiving School
Counting students with their final/receiving school makes practical sense because the final status is known to the school to which the student is attributed. It also makes intuitive sense to group students with the school from which they graduate or drop out. Schools generally feel that they should get credit for all of the students they graduate.
Although there are many practical advantages to counting transfers with their receiving school, doing so produces rates that are biased in favor of schools that can control their enrollment, and it can provide perverse incentives to transfer students at risk of not completing school. Attributing dropouts to their receiving school, instead of their sending school, assumes the problems that led students to drop out occurred at the final school. Yet, as discussed in Chapter 5, students’ performance in their first year of high school is extremely predictive of eventual graduation (Allensworth and Easton, 2007). If a student
leaves a school because of problems at that sending school, it may not be fair to credit that student’s eventual withdrawal to the receiving school, especially if the student spent little time at the receiving school. This results in bias in the rates when schools have different enrollment policies. Schools that can choose whether to enroll a transfer student can boost their graduation rate by accepting students who have already shown some success in high school (e.g., on-time eleventh or twelfth graders).^{15} This can happen, for example, if a school uses a lottery to pre-enroll eighth graders into grade 9 and then uses lotteries for grades 10-12; such a system precludes grade 9 repeaters. Likewise, schools with open enrollment can seem to be doing worse than they should since they accept students who have been unsuccessful in other places. This is a flaw inherent with the National Governors Association (NGA) graduation rate; it is biased in favor of schools that have some control over their enrollment.
Attribution to the Sending School
The final option is to count students with their first (sending) school. This is the method used for graduation rates at the college level in the NCES Graduation Rate Survey. Given the strong relationship between first-year course performance and eventual graduation, it makes sense from a conceptual standpoint to count students with their first school. It also prevents schools from benefiting by encouraging poorly performing students to transfer, particularly to schools that may not best serve their interests. It may seem unfair to count a student who transferred and then dropped out against the graduation rate of their original school. However, unless students are systematically leaving a school for a particular reason (e.g., because of safety concerns at the school that are forcing students out), the dropout and graduation rates of the school’s former students should balance each other and not substantially affect the dropout rate. Using the first school also allows schools without a grade 12 (e.g., a middle school with grade 9) to be compared with other schools.
The problem with this approach is that students’ final outcomes may not be known to the sending school. It can be done if student-level data are available at a district or state level, but there will still be out-of-district or out-of-state transfers that cannot be included in the calculations. The NCES college graduation rate circumvents this problem by counting all transfer students as nongraduates from their original college. This results in statistics that are unbiased and accurately represent the graduation rate of students who have matriculated at a given college, thus circumventing the problem of overcounting transfer students if statistics are aggregated. The disadvantage is that it underrepresents
the percentage of students who graduate overall; that is, it indicates only the percentage of students who graduate from their initial institution.
For purposes of accountability, this method does not produce perverse incentives to push students out of a school, but neither does it provide an incentive to ensure that students who transfer into a school are well served by the receiving school. Politically, it is difficult to convince schools that their graduates should not be counted in their completion indicators.
Choosing Among Methods
Each method of defining the numerator and denominator in a dropout/completion statistic has advantages and disadvantages. No method will produce a statistic that is ideal for all purposes, and decisions about who to include or exclude will have an impact on the rate that is produced. Likewise, rules about who is included and excluded can produce incentives for school officials to act in ways that are not always in the best interest of students. For instance, a school can lower its dropout rate if it is allowed to classify students as transfers rather than dropouts when they leave school and enroll in a GED-granting program. A sole focus on the on-time (four-year) graduation rate can produce incentives for schools to push out students who are likely to take longer than four years to finish high school, particularly if they are allowed to remove these students from both the numerator and the denominator of the rate.
There are no optimal ways for handling these decisions that overcome all the potential disadvantages. However, steps can be taken to ensure that the impact of the decisions is understood. For instance, requiring school officials to document the status of transfer students and the decisions made in calculating the rates (i.e., who is included in the numerator and the denominator) is critical both for interpreting the rates and for revealing unintended incentives in the rules. Reporting a variety of rates—such as both on-time and eventual graduation rates, completion rates that include other methods of finishing high school, rates that do not remove transfer students or incorporate new students, and age-based rates—can provide a more comprehensive picture of schools’ effectiveness at graduating students. Transparency with regard to the methodology underlying the statistics that are created and acknowledgment of their strengths and weaknesses is essential for understanding.
COMPUTING DISAGGREGATED RATES
Reporting disaggregated dropout and completion rates requires decisions about who should be included in each group. The No Child Left Behind (NCLB) Act requires schools to provide separate rates for students grouped by race/ethnicity, disability status, and English language proficiency. Yet it is not always clear who should be included in each group. For example, students
can now self-identify their ethnicity and race using multiple categories (U.S. Department of Education, 2008), and students may change their race/ethnicity designation(s) from year to year. This can make classification difficult and inconsistent across schools. Changes in the ways students are classified result in rates that are not comparable over time, which can lead to false conclusions about the rates for particular groups.
English Language Learners
Calculating dropout and completion rates for English language learners (ELLs) is particularly difficult for a number of reasons. First, definitions of English proficiency are not always consistent across districts and states (Abedi and Dietel, 2004; National Research Council, 2004, in press). States have different criteria for initially classifying students as ELL and for reclassifying them (i.e., removing them from the ELL classification because they have acquired sufficient proficiency in English). This means that cross-state and/or cross-district comparisons of dropout and graduation rates for this subgroup may not be valid. More importantly, students’ English proficiency status changes over time as they acquire sufficient English language skills to be removed from the ELL classification. Many students who are initially classified as ELL in the elementary grades are no longer classified as such in high school.
Students who are still classified as ELL in high school—the years when dropout and completion statistics are calculated—are generally a select group. They may be students who have had difficulty learning English and have remained in the classification longer than is typical. Or they may have been placed in the classification because they recently moved to the United States. Both of these factors are related to completing school. That is, students’ rate of acquisition of English is correlated with their academic achievement and their socioeconomic status,^{16} factors that are correlated with dropping out and completion. Immigration at older ages is also related to the likelihood of graduation, as children who move to the United States at older ages tend to obtain fewer years of education than children who immigrate when very young (Allensworth, 1997). Highly mobile students are also more likely to remain classified as ELL longer than students who are less mobile because they tend to have long periods of “service breaks” due to absence or transfer; mobility is another characteristic correlated with high school completion. For these reasons, students still in the bilingual or English as a second language program
in high school would be expected to have higher dropout rates than students not in these programs.
In order to obtain the most accurate estimates of schools’ success rates at graduating ELL students, it is essential to know which students have been classified as ELL at any time during their schooling, particularly during the primary grades. This requires data systems to include records on ELL status from the primary grades onward that remain constant as students progress to higher grades—a requirement not easily met, since many states do not have access to primary grade records for these students. These primary grade records would need to accompany students who transfer, so that receiving schools can correctly classify students who were formerly classified as ELL. Otherwise, statistics that are aggregated from school records will fail to include former ELL students who transferred out of their original school.
Students with Disabilities
Unlike ELL status, students’ disability status does not typically change over time. That is, having a hearing, visual, emotional, or cognitive impairment or a learning disability is usually not a temporary condition. However, classification criteria for these students are not always consistent across schools, districts, or time (National Research Council, 2004), and thus their status as a recipient of special education services may change over time. The classification criteria depend, in part, on the resources available in the school and the district for diagnosis, as well as district policies about designating students as disabled.^{17} Because the criteria for placing students into the category differ across schools and districts, the dropout and completion rates for this group of students may not be comparable across units.
Furthermore, students who have achievement well below grade level are sometimes classified as learning disabled simply on the basis of their low achievement, even though they may not meet the other criteria for the learning disability classification. These students may be low-achieving because they actually have a learning disability, or their low achievement may be due to other factors that are related to the likelihood of completion, such as years of low engagement in school. Classifying students into the group based on a
characteristic (low achievement) that itself is highly correlated with dropping out can produce misleading results when the indicator is calculated. That is, the dropout/completion indicator may look worse for this group because it includes students who may not belong in the category and are at a higher risk of dropping out due to the characteristic that caused them to be placed in the category.
In addition, the time element in cohort/group definition is also particularly problematic for students with disabilities, as some of them are given more than four years to complete high school in their Individualized Education Plan or are to remain in the school system past age 18.
AGGREGATING RATES
An additional issue to consider when designing indicators is whether they will be aggregated to represent completion or dropout rates at higher levels of analysis. Rates that are accurately defined at the school level are subject to a number of errors when aggregated to the district, state, or federal level.^{18} For instance, the ways that transfer students are handled can result in their being omitted or double-counted in aggregated rates. Having individual-level student data can alleviate these problems. However, subgroup definitions (e.g., defining who is an ELL student or a special education student) become more problematic at the state level, where there may be less detailed longitudinal information on individual students.
Students still enrolled after the date they should have graduated are often counted multiple times, particularly if the rates are calculated from aggregate numbers of students instead of longitudinal student records. They are initially counted as noncompleters in their own cohort; they are counted again in the cohort that follows, either as completers or noncompleters. This introduces inaccuracy in the rates, but whether they are inflated or deflated depends on the specific way in which students are double-counted and whether adjustments are made to the denominator along with the numerator. Incorporating students who remain in school an additional year also makes the indicator less sensitive to changes in dropout and completion that may be occurring in the school, which in turn makes them less useful for evaluating the effects of new policies and practices.
CONCLUSIONS AND RECOMMENDATIONS
Graduation and dropout rates are critical outcome measures of secondary schooling and should be incorporated into accountability systems. However, there are inherent weaknesses in any method of calculating those rates. Even the NGA graduation rate, which we encourage states to adopt, is biased in favor of schools that can control their enrollments. However, if the limitations of the calculations are made explicit, alternative rates could be calculated to verify any conclusions that are made with the statistics. For example, if charter schools were judged to be more effective at retaining students based on the NGA rate, it would be prudent to calculate alternative statistics not biased by school’s enrollment policies to verify this conclusion. This would require data to be available to calculate the complementary statistics. Thus, it is important to produce other indicators of the outcomes of secondary schooling and the development of a robust data system that can produce different variations of the completion rate estimates.
To the greatest extent possible, the methods for computing the rates should meet the following criteria: (1) provide the most accurate assessment possible of how many students actually complete or drop out of school, (2) not be biased in favor of certain types of schools, (3) be inclusive of all students but not double-count them, (4) be stable enough to validly track trends over time, and (5) be sensitive to real changes in student outcomes. All of this must be done within the constraints of the data systems available to schools and given the purpose for which the statistics are to be used. No indicator is perfect, and trade-offs among the criteria are usually required. Ultimately, decisions about the indicators and the ways to calculate them should reflect the intended purpose and uses. The trade-offs, strengths, and shortcomings should be made explicit. We therefore recommend:
RECOMMENDATION 3-1: The strengths and weaknesses of dropout and completion rates should be made explicit when the rates are reported.
RECOMMENDATION 3-2: Rates should be accompanied with documentation about the underlying decisions that were made regarding students who transfer from one school to another, are retained in grade, receive a GED credential or an alternative diploma, and take longer than four years to graduate.
Because decisions about how to handle various groups of students can affect the rates, we think it is important to supplement dropout and completion indicators with information to help users accurately interpret them. As described in this chapter, schools and states have different policies for handling transfer students. Documentation of how transfers were handled is critical for interpreting school-level rates. Also useful is an estimate of the transfer/leave
rate and supplementary graduation/dropout rates that do not remove transfers or incorporate new students. This additional information would allow examination of the ways in which schools’ policies for handling transfer students affect the reported rates.
Policies for grade retention also vary across schools, districts, states, and time. These policies, and particularly year-to-year changes in these policies, can cause trends in the rates to fluctuate over time. Age-based cohort rates can provide information to help users understand and evaluate trends in grade-based rates. Age-based rates have the advantage that they are unaffected by patterns in grade retention that may have affected one cohort differentially from another. They are also more inclusive, in that they can include students who never make it to high school and can include special education students with their peers. If the limitations associated with a reported rate are made explicit, supplemental rates can be calculated to verify any conclusions that are based on the statistics, although this would require data to be available to calculate the supplemental statistics. We therefore recommend:
RECOMMENDATION 3-3: To the extent possible, data should be made available to allow supplementary rates to be calculated that compensate for the limitations in reported rates and help users to further understand the rates. Types of supplementary information include transfer rates, rates that do not remove transfer students or incorporate new students, age-based rates, and the percentage of students with unknown graduation status.
The federal government requires states and districts to produce 4-year graduation rates that include diploma recipients only. As discussed above, there are compelling reasons for using this statistic as the primary indicator of high school completion. However, there are also legitimate reasons to produce more inclusive completion indicators that allow students more time to complete high school and include other forms of completion, such as GEDs and alternative diplomas. Unless there is a common definition, such statistics will not be comparable across districts and states and over time. We therefore recommend:
RECOMMENDATION 3-4: In addition to the standard graduation rate that is limited to 4-year recipients of regular diplomas, states and districts should produce a comprehensive completion rate that includes all forms of completion and allows students up to six years for completion. This should be used as a supplemental indicator to the 4-year graduation rate, which should continue to be used as the primary indicator for gauging school, district, and state performance.
As part of the NCLB regulations, states and districts are expected to report disaggregated graduation rates, such as for students grouped by low-income
eligibility, disability status, and English language learner status, and to track their progress over time. These subgroup statistics are often not comparable across schools, districts, or states due to differing methods and rates of identification and reclassification into and out of the subgroup. Furthermore, the methods by which students are placed into subgroups can lead to inaccurate judgments about educational efficacy in a school system for members of the subgroup. For English language learners, inaccuracies are introduced because classification into the subgroup changes over time, and the rate of reclassification is correlated with dropping out. For students with disabilities, underidentification of disabilities and different methods of classifying disabilities result in lack of comparability. Furthermore, because some students with disabilities are expected to remain in school for more than four years, the subgroup statistics for students with disabilities will disproportionately be affected by decisions about the number of years allowed for graduation in the indicators (e.g., four-year versus five-year rates).
The main purpose of subgroup statistics is to gauge the degree to which schools, districts, and states are serving particular groups of students. To make these judgments fairly, alternative statistics should provide supplemental information for subgroups. With regard to graduation rates for these subgroups, we recommend:
RECOMMENDATION 3-5: To improve knowledge about graduation rates among subgroups, alternate statistics should complement conventional indicators. Alternative graduation rates for English language learners (ELLs) should include former ELL students as well as students currently classified in this category. Thus, records on ELL status should accompany students as they progress through grades, change ELL status, and transfer across districts. Alternative graduation rates for special education students and English language learners should allow additional years toward graduation.