National Academies Press: OpenBook

Educating One and All: Students with Disabilities and Standards-Based Reform (1997)

Chapter:C USING THE PROSPECTS DATA TO REPORT ON THE ACHIEVEMENT OF STUDENTS WITH DISABILITIES

« Previous: B WORKSHOP SUMMARY: STUDENTS WITH DISABILITIES AND STANDARDS-BASED REFORM
Suggested Citation:"C USING THE PROSPECTS DATA TO REPORT ON THE ACHIEVEMENT OF STUDENTS WITH DISABILITIES." National Research Council. 1997. Educating One and All: Students with Disabilities and Standards-Based Reform. Washington, DC: The National Academies Press. doi: 10.17226/5788.
×

APPENDIX
C

Using the Prospects Data to Report on the Achievement of Students with Disabilities

INTRODUCTION

In 1988, Congress mandated a "national longitudinal study of eligible children" to assess the effect of Chapter 1 (now renamed Title I) on students' academic achievement and other measures of school success. This study, title Prospects: The Congressionally Mandated Study of Educational Opportunity and Growth, was designed to evaluate the short-and long-term consequences of Chapter 1 program participation by following large national samples of public school-children in three grade cohorts, as well as their parents, teachers, and principals. Baseline data were collected in spring 1991 for third and seventh grade students and in fall 1991 for first grade students.

There were three stages of sampling for Prospects: (1) selection of a sample of school districts, (2) selection of a sample of schools within sampled districts, and (3) subsampling of students, but only in very large schools. Within most sampled schools, all students enrolled in all classrooms containing the target sample grades were included in the sample. Thus, the Prospects study includes all enrolled students within designated grades with no exclusions on the basis of disability, lack of English proficiency, or any other reasons. Thus, Prospects was designed to include approximately 7 to 10 percent more students compared with other national studies. If a student with a disability was excused from participating in some activities on which data were gathered, (e.g., achievement testing, self-administered questionnaire), every attempt was made to complete the remainder of the data collection protocol for that student.

A rich collection of information was gathered, including responses to a district Chapter 1 coordinator questionnaire, a school and program questionnaire (completed by principal or other staff member), a classroom teacher question-

Suggested Citation:"C USING THE PROSPECTS DATA TO REPORT ON THE ACHIEVEMENT OF STUDENTS WITH DISABILITIES." National Research Council. 1997. Educating One and All: Students with Disabilities and Standards-Based Reform. Washington, DC: The National Academies Press. doi: 10.17226/5788.
×

naire, a student questionnaire, the Comprehensive Test of Basic Skills, a parent questionnaire, as well as student record information and a student profile (ratings completed by the teacher).

The study was designed as a six-year longitudinal study for evaluating Chapter 1. However, funding for the study was terminated before it was completed. In these analyses, we therefore use only the first two years of the study, 1991 and 1992, and use only the data for the third grade cohort. The design was national in scope and focused on cohorts in grades 1, 3, and 7, with oversampling of low-income districts and schools. The sample include 337 schools, with 10,333 students in the third grade cohort. For a detailed description of the study, see U.S. Department of Education (1993). In this appendix we refer to the program as Chapter 1.

SIMPLE POINT ESTIMATES OF ACHIEVEMENT

By far the most common method of assessing and reporting achievement based on standardized tests is to report single, point estimates or cohort scores, perhaps broken down by group categories. The most common statistics are either to report median or mean scores, by selected grades. Because the reported scores are usually based on a national probability distribution, individual student scores are measured relative to the national population of students in a given grade. Institutional scores (by district or school) are aggregates of individual scores and allow for the same comparisons—ignoring within-institution variation.

Examples of third and fourth grade Comprehensive Test of Basic Skills reading and math scores, from the Prospects Study, appear in Table C-1.1 Normal curve equivalents (NCEs) are used as the basic metric. 2 The table provides means and standard deviations for the total population of students tested and relevant subpopulations. The third set of columns—change scores—is an individual change score based only on students taking both the third and fourth grade tests.

The information conveyed is certainly relevant. The total population, which is a sample of students in schools with high concentrations of Chapter 1 students, is below the national mean of 50 on each test, as expected. And between the third and fourth grades, students decline relative to the national norms—more in math

1  

The scores reported in this appendix are unweighted total reading and total math for the third grade sample, although we focus on the fourth grade follow-up for most of the analysis. The fourth grade tests, administered in 1992, included 90 reading test items (40 vocabulary, 50 comprehension) and 94 math items (44 computation, 50 concepts and applications). The subtest reliabilites are between .89 and .94 for the total population.

2  

Normal curve equivalents are used because national percentile rankings are not interval-level data. One of the problems with this transformation is that the very lowest and highest ends of the distribution are compressed. This tends to inflate very low-end scores and deflate very high-end scores. The lower end inflation may affect this population, which has quite a few test scores below the 10th national percentile.

Suggested Citation:"C USING THE PROSPECTS DATA TO REPORT ON THE ACHIEVEMENT OF STUDENTS WITH DISABILITIES." National Research Council. 1997. Educating One and All: Students with Disabilities and Standards-Based Reform. Washington, DC: The National Academies Press. doi: 10.17226/5788.
×

TABLE C-1 Third and Fourth Grade Prospects Achievement Test Data, 1991-1992

 

 

Third Grade

Fourth Grade

Change Scores

 

 

Reading

Math

Reading

Math

Reading

Math

Total Population

Mean

46.8

47.7

45.4

45.4

-1.1

-2.7

Standard Deviation

20.6

20.2

20.5

22.0

12.8

14.6

N

13,431

13,167

10,584

10,584

7,906

7,692

Free Lunch

Mean

41.0

43.0

39.0

39.9

-1.6

-3.1

Standard Deviation

19.0

18.9

18.1

20.1

12.5

14.3

N

4,752

4,696

4,304

4,282

3,109

3,064

Non-Free Lunch

Mean

55.1

54.9

53.8

53.1

-0.9

-2.1

Standard Deviation

19.2

19.2

19.7

21.4

12.8

14.3

N

5,890

5,744

4,817

4,674

3,891

3,757

Females

Mean

48.6

48.4

47.5

46.4

-0.8

-2.3

Standard Deviation

19.6

19.4

20.0

21.1

12.2

13.6

N

6,683

6,562

5,223

5,125

4,001

3,897

Males

Mean

45.2

47.1

43.6

44.7

-1.5

-3.0

Standard Deviation

21.5

20.9

20.8

22.8

13.3

15.0

N

6,625

6,489

5,204

5,101

3,903

3,793

African American

Mean

37.3

38.3

36.0

35.6

-1.7

-3.0

Standard Deviation

18.3

18.4

17.3

19.1

12.9

14.4

N

2,824

2,801

1,984

1,976

1,524

1,507

Asian American

Mean

47.4

55.5

48.8

59.2

1.6

1.2

Standard Deviation

19.6

20.0

19.1

21.7

11.4

13.8

N

604

596

469

461

376

369

Hispanic American

Mean

37.4

41.2

36.2

38.3

1.2

-3.9

Standard Deviation

19.0

19.1

18.0

19.5

12.7

14.0

N

2,125

2,078

1,920

1,889

1,398

1,366

Other American

Mean

46.2

47.0

43.9

45.3

-2.2

-2.1

Standard Deviation

19.1

19.1

19.0

20.8

11.4

13.2

N

283

278

195

191

150

142

White American

Mean

53.4

52.8

52.1

50.8

-1.2

-2.7

Standard Deviation

19.5

19.3

19.8

21.5

12.9

14.5

N

6,605

6,423

5,132

4,992

4,027

3,880

Disabled

Mean

41.7

42.7

40.2

39.9

-1.2

-2.9

Standard Deviation

22.6

21.4

20.4

22.4

12.4

14.7

N

1,152

1,124

821

796

582

562

Suggested Citation:"C USING THE PROSPECTS DATA TO REPORT ON THE ACHIEVEMENT OF STUDENTS WITH DISABILITIES." National Research Council. 1997. Educating One and All: Students with Disabilities and Standards-Based Reform. Washington, DC: The National Academies Press. doi: 10.17226/5788.
×

 

 

Third Grade

Fourth Grade

Change Scores

 

 

Reading

Math

Reading

Math

Reading

Math

Emotional Disability

Mean

36.3

35.2

33.5

31.5

-1.6

-3.4

Standard Deviation

21.0

21.7

18.5

22.9

12.6

17.0

N

95

98

85

81

56

52

Learning Disability

Mean

27.1

30.3

29.6

29.3

-0.5

-2.3

Standard Deviation

18.6

17.5

15.8

17.1

12.1

12.2

N

286

278

217

209

133

127

Physical Disability

Mean

44.7

45.7

43.3

43.5

-1.2

-2.4

Standard Deviation

21.4

21.1

19.1

20.5

12.7

15.2

N

203

189

130

126

100

98

Speech Disability

Mean

41.1

43.5

41.5

42.7

0.1

-1.2

Standard Deviation

21.2

20.3

20.6

22.9

12.2

15.6

N

307

303

234

225

168

164

Other Health Disability

Mean

48.6

48.5

45.5

44.4

-1.6

-4.3

Standard Deviation

22.2

20.9

20.6

22.8

12.8

14.6

N

399

388

266

258

195

189

(-2.7 NCEs) than reading (-1.1 NCEs). The group differences are also relevant and often quite stark. For example at this age, girls do better than boys on all tests, and drop behind the national population over the year less than boys do. Asian American students score lower than whites on reading but somewhat higher on math; however, Asian students improve more than the national population, or any racial group, on both reading and math. The differences between African and Hispanic Americans and whites and Asians is considerable in both grades on both tests—at times approaching a full standard deviation.

The variance within groups also provides useful information. First, as is typical of large sample test data, the variances around the mean are not very different between groups. For example, the largest differences in variances by race for the four tests over the two grades are 17.3 (African American, fourth grade, reading) to 21.5 (white, fourth grade, math) and all but 3 of the 20 variances are between 18 and 20. However—and critical—these variances within groups may be very misleading for assessing both achievement levels and educational progress. And because the variances are misrepresented by such simple reporting, so are the differences in the means between groups. This can be simply illustrated by using relatively simple and then more complex multivariate estimates of group differences.

Suggested Citation:"C USING THE PROSPECTS DATA TO REPORT ON THE ACHIEVEMENT OF STUDENTS WITH DISABILITIES." National Research Council. 1997. Educating One and All: Students with Disabilities and Standards-Based Reform. Washington, DC: The National Academies Press. doi: 10.17226/5788.
×

MULTIVARIATE ESTIMATES OF ACHIEVEMENT

A range of more complex estimation models can be used to provide a more accurate and richer picture of educational achievement than is obtained by reporting simple, mean point estimates of achievement. The problem is that these estimates require increasingly complex statistical procedures and more elaborate and costly data. In Tables C-2a, C-2b, C-3a, and C-3b data complexity increases in the columns marked Model I to Model III (for Table C-2a and C-3a) and Model IV to VI (for Table C-2b and C-3b). The first level of complexity (Models I and IV) requires multivariate estimates. These variables include: (1) a student income measure—qualifying for free lunch or not, (2) student gender, (3) student race, and (4) student disability status.

Models II and V add variables on family status. These variables—family income, parent education, parent employment, and marital status—were acquired in the Prospects study through parent surveys. Models III and VI add behavioral and attitude data for individual families—data obtained from parent surveys. For purposes of these analyses, the variables include a measure of parent academic educational expectations for their child, an index of satisfaction with the school their child attends, number of school contacts, and three questions on parental involvement (at home, through participation in school organizations, and through attendance at school events).

Finally, Tables C-2a and C-2b are distinguished from C-3a and C-3b by modeling fourth grade student achievement with (C-3a and C-3b) and without controlling for prior achievement (third grade achievement test scores). Tables C-3a and C-3b include prior test scores as independent variables. These models allow change-score, achievement progress assessments.

Cohort, Point-Estimate Models

Increasingly more complex and more accurate estimations of point or cohort scores (when reporting by grade), are depicted in Tables C-2a and C-2b. Table C-4 provides descriptive statistics for variables used in Tables C-2 and C-3. The differences between Tables C-2a and C-2b (and later Tables C-3a and C-3b) are in the modeling of students with disabilities. In Table C-2a, a general indicator variable for being disabled or not is included; in Table C-2b, indicator variables are included for each of 5 types of disability.3

In Model I, for both reading and math, all the variables are indicator variables and the coefficients can be interpreted as differences in means between the relevant categories. Thus, the coefficient for free lunch eligibility for reading means that students whose family income qualifies for government-provided free lunch (1.35 times the poverty line), on average, and controlling for other gender

3  

The reference group, when individual disability groups are represented by indicator variables, are all students without the relevant disability, including students with other types of disabilities.

Suggested Citation:"C USING THE PROSPECTS DATA TO REPORT ON THE ACHIEVEMENT OF STUDENTS WITH DISABILITIES." National Research Council. 1997. Educating One and All: Students with Disabilities and Standards-Based Reform. Washington, DC: The National Academies Press. doi: 10.17226/5788.
×

and racial differences, scored -9.88 normal curve equivalent points less than students who did not qualify for the subsidy. Similarly, girls scored 3.85 points more than boys, African Americans -11.26 less than whites, Asian American 1.78 points less than whites, etc. The size of these differences can be compared with the standard deviations for the fourth grade tests for these groups reported in Table C-4.

There are several important differences between the results derived from these models and the simple descriptive group differences as reported in Table C-1. First, if one computes the crude differences in means in Table C-1 for any category (free-lunch vs. non-free lunch; African American vs. white), in each case the indicator variables in Table C-2a represent smaller differences. The reason is that several of the independent variables are correlated, and thus failure to control for that correlation produces artificially higher estimates of group differences. Specifically, by simply reporting racial group means, we fail to account for the considerable diversity in group populations—in this simple model, the differences in income and gender of students within racial groups.4

Models II and III add precision and explanatory information, but also reduce sample sizes. In this national sample, the reduction in sample sizes results from the failure of sample families to complete surveys. In addition, reduction in sample sizes may affect the accuracy of the estimates of subpopulations, such as students with a given disability. Despite these problems, the added information provides insights into factors affecting achievement, and potentially useful data for specifying realistic expectations for schools and districts. For example, the effect of parent education is obvious and, as we shall see, impervious to the inclusion of almost every variable we can include. Regardless of race, income, employment, or marital status and despite attitudes and direct parent support of education, having a parent who has more education is a significant predictor of higher student achievement.

The same is true of educational expectations held by parents for their children. As measured by a question querying how many years of education they expect their child to complete, ''expectations" are a very significant and strong predictor of higher test scores. This result also carries over into more complex models.

These results tell us something not only about the puzzle of education, but also about how to assess educational systems and specify institutional expectations. They also illustrate the variances within groups of students and the policy implications of excluding such control variables from assessments.

Finally, Models I through III provide useful insights into how students with disabilities could be included in systemic reform assessment systems. Model I

4  

A fully specified model, such as used in analysis of variance, would also include interactive terms between gender, race, and income. In addition, school-level variables could be included. This would require more complex statistical methods and assumptions.

Suggested Citation:"C USING THE PROSPECTS DATA TO REPORT ON THE ACHIEVEMENT OF STUDENTS WITH DISABILITIES." National Research Council. 1997. Educating One and All: Students with Disabilities and Standards-Based Reform. Washington, DC: The National Academies Press. doi: 10.17226/5788.
×

TABLE C-2a Fourth Grade Cohort Regression Models, 1992: Disability Indicator Variable

 

 

Reading Models

Math Models

 

 

Model I

Model II

Model III

Model I

Model II

Model III

Prior Tests

Prior Reading

Prior Math

District SES

Free Lunch (1 = Yes)

-9.88***

-4.78***

-2.81***

-9.00***

-3.72***

-1.27

Gender (1 = Female)

3.85***

4.69***

3.44***

1.48***

2.00***

0.35

African American

-11.26***

-9.52***

-8.34***

-10.36***

-8.65***

-7.28***

Asian American

-1.78

-3.69***

-6.57***

9.31***

6.88***

4.36***

Hispanic American

-11.87***

-9.80***

-9.74***

-8.69***

-7.49***

-7.09***

Other American

-5.21***

-5.76***

-4.07*

2.34

-3.11

-2.02

Disabled (1 = Yes)

-6.90***

-6.44***

3.19***

-6.97***

-6.55***

-3.62***

Family SES

Income

0.87***

0.61***

0.91***

0.63***

Respondent Education

2.24***

0.90***

2.15***

0.86***

Respondent Employment

-0.54*

0.04

-0.17

0.61

Respondent Marital Status

0.80

0.02

1.12*

0.13

Family Attitudes/Behavior

Expectations

3.39***

3.54***

School Dissatisfaction

-0.14***

-0.19***

Parental Involvement—Home

0.49***

0.62***

Parental Involvement—Attendance

0.20

0.09

Parental Involvement—Organizations

0.66***

1.04***

School Contacts

-0.60***

-0.58***

Constants

54.48***

39.02***

19.66***

54.12***

37.63***

14.65***

R2

.20

.23

.28

.15

.18

.25

SE

18.12

17.89

16.84

20.07

19.90

18.65

F

305.54

187.00

90.28

205.78

133.05

73.43

(df)

(8,393, 7)

(6,681, 11)

(3,807, 17)

(8,234, 7)

(6,545, 11)

(3,741, 17)

*** probability that B = 0 < .001

** probability that B = 0 < .01

* probability that B = 0 < .05

Suggested Citation:"C USING THE PROSPECTS DATA TO REPORT ON THE ACHIEVEMENT OF STUDENTS WITH DISABILITIES." National Research Council. 1997. Educating One and All: Students with Disabilities and Standards-Based Reform. Washington, DC: The National Academies Press. doi: 10.17226/5788.
×

TABLE C-2b Fourth Grade Cohort Regression Models, 1992: Categories of Disability

 

 

Reading Models

Math Models

 

 

Model IV

Model V

Model VI

Model IV

Model V

Model VI

Prior Tests

Prior Reading

Prior Math

District SES

Free Lunch (1 = Yes)

-9.93***

-4.76***

-2.95***

-9.05***

-3.69***

-1.34

Gender (1 = Female)

3.72***

4.58***

3.41***

1.38**

1.92***

0.40

African American

-11.25***

-9.51***

-8.49***

-10.34***

-8.62***

-7.36***

Asian American

-1.81

-3.65***

-6.38***

9.12***

6.86***

4.42**

Hispanic American

-11.78***

-9.72***

-9.57***

-8.63***

-7.42***

-6.92***

Other American

-4.80**

-5.40***

-4.12*

-2.18

-2.84

-2.28

Disabled

Emotional

-9.82***

-8.93***

-5.00

-11.34***

-10.05***

-6.98

Learning

-16.81***

-17.44***

-17.49***

-16.44***

-16.78***

-16.12***

Physical

-2.56

-1.80

-2.44

-1.00

-0.53

-4.80

Speech

-2.99*

-2.36

1.30

-2.36

-1.32

2.45

Other

-0.54

-0.54

1.49

-1.92

-2.34

-0.54

Family SES

Income

0.89***

0.64***

0.94***

0.66***

Respondent Education

2.22***

0.90***

2.15***

0.88***

Respondent Employment

-0.52*

0

-0.18

0.54

Respondent Marital Status

0.68

-0.21

1.24

0.03

Family Attitudes/Behavior

Expectations

3.33***

3.49***

School Dissatisfaction

-0.15***

-0.20***

Parental Involvement—Home

0.48***

0.60***

Parental Involvement—Attendance

0.26

0.16

Parental Involvement—Organization

0.59***

0.97***

School Contacts

-0.57***

-0.57***

Constants

54.53***

38.97***

20.42***

54.17***

37.55***

15.18***

R2

.21

.24

.29

.16

.19

.26

SE

18.02

17.75

16.70

19.99

19.81

18.56

F

202.84

143.58

76.13

136.55

101.61

61.27

(df)

(8,264, 11)

(6,583, 15)

(3,741, 21)

(8,106, 11)

(6,448, 15)

(3,676, 21)

*** probability that B = 0 < .001

** probability that B = 0 < .01

* probability that B = 0 < .05

Suggested Citation:"C USING THE PROSPECTS DATA TO REPORT ON THE ACHIEVEMENT OF STUDENTS WITH DISABILITIES." National Research Council. 1997. Educating One and All: Students with Disabilities and Standards-Based Reform. Washington, DC: The National Academies Press. doi: 10.17226/5788.
×

TABLE C-3a Fourth Grade Value-Added Regression Models, 1991—1992: Disability Indicator Variable

 

 

Reading Models

Math Models

 

 

Model I

Model II

Model III

Model I

Model II

Model III

Prior Tests

Prior Reading

0.62***

0.61***

0.61***

0.21***

0.20***

0.17***

Prior Math

0.21***

0.21***

0.19***

0.65***

0.66***

0.66***

District SES

Free Lunch (1 = Yes)

-2.49***

-1.66***

-0.24

-1.52***

-0.50

1.06

Gender (1 = Female)

1.81***

2.06***

1.75***

0.36

0.41

-0.42

African American

-2.25***

-1.84***

-1.54*

-1.16*

-0.79

-0.62

Asian American

0.55

-0.14

-1.04

6.61***

5.55***

4.76***

Hispanic American

-2.53***

-2.39***

-2.91***

-1.46**

-1.42*

-1.59*

Other American

-1.53

-2.29

-1.77

1.26

0.51

0.43

Disabled (1 = Yes)

-0.86

-0.88

0.57

-1.17

-1.35

-0.42

Family SES

Income

0

0.17

0

0.21

Respondent Education

0.81***

0.42*

0.91***

0.54***

Respondent Employment

-0.30

-0.24

-0.25

-0.19

Respondent Marital Status

0.25

-0.27

0.62

-0.12

Family Attitudes/Behavior

Expectations

0.73***

0.76***

School Dissatisfaction

-0.06*

-0.07**

Parental Involvement—Home

0.08

0.17**

Parental Involvement—Attendance

0.11

-0.06

Parental Involvement—Organizations

0.19

0.44***

School Contacts

-0.44***

-0.47***

Constants

8.24***

5.58***

5.32*

5.47***

2.35*

0.82

R2

.67

.68

.67

.61

.62

.63

SE

11.55

11.58

11.29

13.45

13.39

12.87

F

1,438.60

814.63

325.23

1,109.74

643.35

268.69

(df)

(6,323, 9)

(5,067, 13)

(2,967, 19)

(6,320, 9)

(5,062, 13)

(2,967, 19)

*** probability that B = 0 < .001

** probability that B = 0 < .01

* probability that B = 0 < .05

Suggested Citation:"C USING THE PROSPECTS DATA TO REPORT ON THE ACHIEVEMENT OF STUDENTS WITH DISABILITIES." National Research Council. 1997. Educating One and All: Students with Disabilities and Standards-Based Reform. Washington, DC: The National Academies Press. doi: 10.17226/5788.
×

TABLE C-3b Fourth Grade Value-Added Regression Models, 1991–1992: Categories of Disability

 

 

Reading Models

Math Models

 

 

Model IV

Model V

Model VI

Model IV

Model V

Model VI

Prior Tests

Prior Reading

0.62***

0.61***

0.61***

0.21***

0.20***

0.16***

Prior Math

0.21***

0.21***

0.19***

0.65***

0.66***

0.66***

District SES

Free Lunch (1 = Yes)

-2.54***

-1.73***

-0.37

-1.54***

-0.54

0.98

Gender (1 = Female)

1.71***

1.99***

1.68***

0.30

0.38

-0.36

African American

-2.26***

-1.84***

-1.68*

-1.16*

-0.77

-0.70

Asian American

0.68

0

-0.91

6.55***

5.54***

4.67***

Hispanic American

-2.43***

-2.30***

-2.79***

-1.43**

-1.38*

-1.54*

Other American

-1.26

-2.02

-1.61

1.30

0.60

0.32

Disabled

Emotional

-2.57

-3.56

-1.98

-2.58

-4.23

-4.51

Learning

-3.20**

-3.27*

-4.09*

-1.59

-1.97

-3.54

Physical

-0.02

0

0.46

0.41

0.86

-0.68

Speech

0.53

1.00

3.01*

0.57

0.86

2.77

Other

0.34

0.12

2.13

-2.19*

-2.48*

-0.95

Family SES

Income

0

0.15

0.01

0.22

Respondent Education

0.82***

0.43**

0.94***

0.56***

Respondent Employment

-0.28

-0.26

-0.26

-0.22

Respondent Marital Status

0.21

-0.37

0.59

-0.19

Family Attitudes/Behavior

Expectations

0.77***

0.78***

School Dissatisfaction

-0.06*

-0.07**

Parental Involvement—Home

.07

0.16**

Parental Involvement—Attendance

0.12

-0.02

Parental Involvement—Organization

0.17

0.44**

School Contacts

-0.43***

-0.48***

Constants

8.34***

5.67***

5.83*

5.50***

2.38*

1.10

R2

.67

.68

.68

.61

.62

.63

SE

11.53

11.56

11.25

13.46

13.41

12.88

F

985.78

616.76

266.77

755.12

483.25

218.50

(df)

(6,217, 13)

(4,988, 17)

(2,913, 23)

(6,214, 13)

(4,983, 17)

(2,913, 23)

*** probability that B = 0 < .001

** probability that B = 0 < .01

* probability that B = 0 < .05

Suggested Citation:"C USING THE PROSPECTS DATA TO REPORT ON THE ACHIEVEMENT OF STUDENTS WITH DISABILITIES." National Research Council. 1997. Educating One and All: Students with Disabilities and Standards-Based Reform. Washington, DC: The National Academies Press. doi: 10.17226/5788.
×

TABLE C-4 Fourth Grade Cohort and Value-Added Regressions: Variable Definitions and Statistics

 

Mean

Standard Deviation

Range

(N)

Reading NCE (1992)

45.42

20.48

98.00

10,584

Math NCE (1992)

45.41

21.96

98.00

10,388

Free Lunch (1 = Yes)

0.47

0.50

1.00

9,221

Gender (1 = Female)

0.50

0.50

1.00

10,542

African American

0.20

0.40

1.00

9,810

Asian American

0.05

0.21

1.00

9,810

Hispanic American

0.20

0.40

1.00

9,810

Other American

0.02

0.14

1.00

9,810

Disabled (1 = Yes)

0.08

0.27

1.00

10,543

Emotional

0.01

0.09

1.00

9,791

Learning

0.02

0.15

1.00

9,925

Physical

0.01

0.12

1.00

9,837

Speech

0.02

0.15

1.00

9,945

Other

0.03

0.16

1.00

9,977

Income

6.71

2.72

9.00

8,696

Respondent Education

3.31

1.77

7.00

8,364

Respondent Employment

2.15

0.91

2.00

9,178

Respondent Marital Status

0.68

0.47

1.00

9,387

Expectations

5.12

1.70

6.00

7,697

School Dissatisfaction

39.77

9.35

62.00

6,501

Parental Involvement—Home

24.04

4.78

36.00

7,015

Parental Involvement—Attendance

12.26

2.72

16.00

6,998

Parental Involvement—Organization

8.65

2.19

14.00

7,520

School Contacts

11.64

2.46

18.00

8,763

contains a simple indicator variable for disability. The result, controlling for income, gender, and race, is a very significant -6.90 normal curve equivalents in reading and -6.97 in math. As expected, students with disabilities do less well. However, when we control for more variables, the differential scores are partly dissipated. Controlling for family socioeconomic status has only a modest result, but controlling for expectations, satisfaction with the school, and parental involvement has a major effect in predicting reduced differential scores for students with disabilities. Although not suggesting a causal explanation, it is clear that, as before, within-group variance is considerable and must be taken into account in assessment systems.

The differences in results between Tables C-2a and C-2b highlight this fact. In Table C-2b a series of indicator variables are used to represent different types of disabilities. The effects are quite startling. Essentially, when suitable controls are employed, only emotionally disturbed and learning disabled students have

Suggested Citation:"C USING THE PROSPECTS DATA TO REPORT ON THE ACHIEVEMENT OF STUDENTS WITH DISABILITIES." National Research Council. 1997. Educating One and All: Students with Disabilities and Standards-Based Reform. Washington, DC: The National Academies Press. doi: 10.17226/5788.
×

cohort test scores significantly below the rest of the population.5 Learning disabled student scores are close to a standard deviation below the rest of the population; emotionally disturbed students are less far behind.

The effects of including control variables seem to differ for students with emotional disturbances and learning disabilities. The differences in estimated test scores for students with emotional disturbances are smaller than the expected differences computed for fourth graders in Table C-1. And, as more variables are added to explain the variance, the effect of an emotional disability declines to the point that it may not be significant when we include family attitude and behavior effects. In contrast, for both reading and math, the effects of a learning disability are not reduced very much by inclusion of any control variables. The differences in means for fourth graders computed from Table C-1 are very close to the sizes of the effects for students with learning disabilities in Table C-2b, and the size and significance of the coefficient does not change much as more variables are added.

Value-Added Models

Value-added achievement models are based on the assumption that to adequately measure educational progress and the varying contribution of educational institutions, one must control for prior student achievement. Various measures of change can then be constructed and, controlling for relevant student, family, and institutional differences, reasonable expectations based on student progress can be established.

Tables C-3a and C-3b present results of such models for the Prospects study. The tables duplicate those presented in the cohort models depicted in Tables C-2a and C-2b with the addition of third grade reading and math test scores as measures of prior achievement. As expected, the prior tests are very good predictors of fourth grade tests. And the coefficients are quite stable across models. All are significant at the .001 level; the primary tests have coefficients between .61 and .66; and the secondary tests are approximately.2.

What is more interesting are the changes in the coefficients for the remaining independent variables. The most obvious differences are that the coefficients for all independent variables are much smaller. This is to be expected because we are now essentially estimating the variance in changes in achievement, not simply the variance between scores. And changes are smaller numbers.6 What is more relevant is the statistical significance of the coefficients.

5  

"Physical disabilities" include categories for physical, hearing, speech, orthopedic, and deafness disabilities. The other categories were coded as they appear in the tables. Mental retardation was excluded from the analysis because only 5 students with mental retardation were given standardized tests.

6  

The reader can verify the differences by taking the means of fourth grade scores in Table C-4 and subtracting from them the appropriate Bs times the third grade scores for both reading and math. That is the equivalent of what we are estimating once prior achievement has been controlled.

Suggested Citation:"C USING THE PROSPECTS DATA TO REPORT ON THE ACHIEVEMENT OF STUDENTS WITH DISABILITIES." National Research Council. 1997. Educating One and All: Students with Disabilities and Standards-Based Reform. Washington, DC: The National Academies Press. doi: 10.17226/5788.
×

We leave it to the reader to explore the totality of differences emerging from a comparison of Tables C-2 and C-3. We note several interesting observations. Gender differences in math scores are significant in Tables C-2a and C-2b for Models I and II, but none of the value-added differences in Tables C-3a and C-3b are significant, and the Model VI coefficients have a negative sign. This may indicate that the absolute advantage of girls in math in the early years is not matched by greater progress. Similarly, Asian American reading scores are below white scores as indicated in Tables C-2a and C-2b, but those differences disappear once prior achievement is controlled. This means that, in terms of progress on reading, Asian Americans and whites do approximately the same.

Variables that remain significant with value-added measures include: (1) some racial effects—extraordinarily positive for Asian Americans on math, negative for African and Hispanic Americans on reading and at times on math; (2) parent's education, which remains positively related to increase learning; and (3) parental expectations—there is a positive effect of higher parental educational expectations. The only parental involvement measure that seems to matter is the scale measuring the frequency of school contracts—the effect is to depress fourth grade test scores.

The implications of value-added models for students with disabilities are quite striking. If all students with disabilities are considered together, Table C-3a indicates that there is no reason to expect statistically significant differences between the populations with and without disabilities. In fact, controlling for the full range of variables (Table C-3a, Model VI), the coefficients vary closely around zero.

Different disabilities suggest different value-added results. The only consistent significant effect is for reading scores for students with learning disabilities. As reported above, cohort score estimates for students with learning disabilities were consistently close to a standard deviation behind students without learning disabilities—regardless of which model was estimated. It also appears that, between the third and fourth grades, students with learning disabilities fall further behind, and surprisingly, the effect seems to increase in size as more control variables are included in the equation. The only other effects that approach conventional levels of significance are a positive effect for students with speech impairments on the full reading model (Model VI), and small negative effects on math for those with "other" disabilities or health problems.

CONCLUSIONS

If further analysis confirms these results, they suggest several conclusions. First, different types of disabilities need to be treated separately. Second, the conclusions differ when one models cohort and value-added achievement measures. Third, students with learning disabilities need to be studied further and perhaps treated quite differently in assessment systems. Results of these analyses

Suggested Citation:"C USING THE PROSPECTS DATA TO REPORT ON THE ACHIEVEMENT OF STUDENTS WITH DISABILITIES." National Research Council. 1997. Educating One and All: Students with Disabilities and Standards-Based Reform. Washington, DC: The National Academies Press. doi: 10.17226/5788.
×

suggest that students with learning disabilities may show persistently poor test scores and poor progress despite variation in a host of exogenous factors, which in other populations are related to achievement success.

Despite differences in opinion about what students should know and what is a valid form for testing that knowledge, policy makers undertaking standards-based reforms still need to compare student achievement over time, across populations, and between organizations. This requires internal and test-retest reliability for the instruments as well as the conversion of scores into a known probability distribution so that unbiased trend and intergroup comparisons can be made.7 Value-added models, which control for prior achievement, offer promise as a valid method for reporting achievement scores and should be considered by policy makers.

REFERENCE

U.S. Department of Education 1993. Prospects: The Congressionally Mandated Study of Educational Growth and Opportunity: The Interim Report. July. Washington, DC: U.S. Department of Education.

7  

Although many probability distributions could produce desired comparisons, normal or Student's T distributions are by far the most commonly assumed ones. They allow the use of numerous, common statistical techniques for analyzing results. This explains the requirement under Chapter 1 funding that tests be administered that can be converted into normal curve equivalents.

Suggested Citation:"C USING THE PROSPECTS DATA TO REPORT ON THE ACHIEVEMENT OF STUDENTS WITH DISABILITIES." National Research Council. 1997. Educating One and All: Students with Disabilities and Standards-Based Reform. Washington, DC: The National Academies Press. doi: 10.17226/5788.
×
Page262
Suggested Citation:"C USING THE PROSPECTS DATA TO REPORT ON THE ACHIEVEMENT OF STUDENTS WITH DISABILITIES." National Research Council. 1997. Educating One and All: Students with Disabilities and Standards-Based Reform. Washington, DC: The National Academies Press. doi: 10.17226/5788.
×
Page263
Suggested Citation:"C USING THE PROSPECTS DATA TO REPORT ON THE ACHIEVEMENT OF STUDENTS WITH DISABILITIES." National Research Council. 1997. Educating One and All: Students with Disabilities and Standards-Based Reform. Washington, DC: The National Academies Press. doi: 10.17226/5788.
×
Page264
Suggested Citation:"C USING THE PROSPECTS DATA TO REPORT ON THE ACHIEVEMENT OF STUDENTS WITH DISABILITIES." National Research Council. 1997. Educating One and All: Students with Disabilities and Standards-Based Reform. Washington, DC: The National Academies Press. doi: 10.17226/5788.
×
Page265
Suggested Citation:"C USING THE PROSPECTS DATA TO REPORT ON THE ACHIEVEMENT OF STUDENTS WITH DISABILITIES." National Research Council. 1997. Educating One and All: Students with Disabilities and Standards-Based Reform. Washington, DC: The National Academies Press. doi: 10.17226/5788.
×
Page266
Suggested Citation:"C USING THE PROSPECTS DATA TO REPORT ON THE ACHIEVEMENT OF STUDENTS WITH DISABILITIES." National Research Council. 1997. Educating One and All: Students with Disabilities and Standards-Based Reform. Washington, DC: The National Academies Press. doi: 10.17226/5788.
×
Page267
Suggested Citation:"C USING THE PROSPECTS DATA TO REPORT ON THE ACHIEVEMENT OF STUDENTS WITH DISABILITIES." National Research Council. 1997. Educating One and All: Students with Disabilities and Standards-Based Reform. Washington, DC: The National Academies Press. doi: 10.17226/5788.
×
Page268
Suggested Citation:"C USING THE PROSPECTS DATA TO REPORT ON THE ACHIEVEMENT OF STUDENTS WITH DISABILITIES." National Research Council. 1997. Educating One and All: Students with Disabilities and Standards-Based Reform. Washington, DC: The National Academies Press. doi: 10.17226/5788.
×
Page269
Suggested Citation:"C USING THE PROSPECTS DATA TO REPORT ON THE ACHIEVEMENT OF STUDENTS WITH DISABILITIES." National Research Council. 1997. Educating One and All: Students with Disabilities and Standards-Based Reform. Washington, DC: The National Academies Press. doi: 10.17226/5788.
×
Page270
Suggested Citation:"C USING THE PROSPECTS DATA TO REPORT ON THE ACHIEVEMENT OF STUDENTS WITH DISABILITIES." National Research Council. 1997. Educating One and All: Students with Disabilities and Standards-Based Reform. Washington, DC: The National Academies Press. doi: 10.17226/5788.
×
Page271
Suggested Citation:"C USING THE PROSPECTS DATA TO REPORT ON THE ACHIEVEMENT OF STUDENTS WITH DISABILITIES." National Research Council. 1997. Educating One and All: Students with Disabilities and Standards-Based Reform. Washington, DC: The National Academies Press. doi: 10.17226/5788.
×
Page272
Suggested Citation:"C USING THE PROSPECTS DATA TO REPORT ON THE ACHIEVEMENT OF STUDENTS WITH DISABILITIES." National Research Council. 1997. Educating One and All: Students with Disabilities and Standards-Based Reform. Washington, DC: The National Academies Press. doi: 10.17226/5788.
×
Page273
Suggested Citation:"C USING THE PROSPECTS DATA TO REPORT ON THE ACHIEVEMENT OF STUDENTS WITH DISABILITIES." National Research Council. 1997. Educating One and All: Students with Disabilities and Standards-Based Reform. Washington, DC: The National Academies Press. doi: 10.17226/5788.
×
Page274
Suggested Citation:"C USING THE PROSPECTS DATA TO REPORT ON THE ACHIEVEMENT OF STUDENTS WITH DISABILITIES." National Research Council. 1997. Educating One and All: Students with Disabilities and Standards-Based Reform. Washington, DC: The National Academies Press. doi: 10.17226/5788.
×
Page275
Next: D BIOGRAPHICAL SKETCHES »
Educating One and All: Students with Disabilities and Standards-Based Reform Get This Book
×
Buy Paperback | $80.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

In the movement toward standards-based education, an important question stands out: How will this reform affect the 10% of school-aged children who have disabilities and thus qualify for special education?

In Educating One and All, an expert committee addresses how to reconcile common learning for all students with individualized education for "one"—the unique student. The book makes recommendations to states and communities that have adopted standards-based reform and that seek policies and practices to make reform consistent with the requirements of special education.

The committee explores the ideas, implementation issues, and legislative initiatives behind the tradition of special education for people with disabilities. It investigates the policy and practice implications of the current reform movement toward high educational standards for all students.

Educating One and All examines the curricula and expected outcomes of standards-based education and the educational experience of students with disabilities—and identifies points of alignment between the two areas. The volume documents the diverse population of students with disabilities and their school experiences. Because approaches to assessment and accountability are key to standards-based reforms, the committee analyzes how assessment systems currently address students with disabilities, including testing accommodations. The book addresses legal and resource implications, as well as parental participation in children's education.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!