Assessing Research-Doctorate Programs A Methodology Study (2003) / Chapter Skim
Currently Skimming:
Appendix G: Technical and Statistical Techniques
Appendix G: Technical and Statistical Techniques
Pages 137-154
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 137... ...
CorreIa1es of Reputation Analysis ~7
|
From page 138... ...
The confidence interval analysis performed in the last two assessments illustrated this point. However, users of the assessments chose to ignore this and focused instead on specific scores obtained by averaging questionnaire responses.
|
From page 139... ...
The advantage in the Bootstrap method, on the other hand, is an established method with a developed theory for the statistical accuracy of survey measurements. A Comparison of the Random Halves and Bootstrap Methods The differences between the methods can be demonstrated by the following simple example.
|
From page 140... ...
The table below summarizes what occurs for three possible half samples for RHO. Note that in the cases, where the average ratings are the same random tie splitting is used and the rank order is clenoted by l.5.
|
From page 141... ...
. In summary, the RH method calls for repeatecIly taking "half-samples" of the rating matnx, averaging the resulting ratings for A en cl B
|
From page 142... ...
Analysis of the Expected Variance for the Two Methods A natural question to ask is: What do the Ranclom Halves and Boot methods produce for probability distributions of average ratings for programs? Drawing on some results from probability theory it can be shown that these methods give similar results.
|
From page 143... ...
RH(21: in this case, RA takes on these three possible values with the corresponding probabilities. Possible average ratings I/2 ~ 3/2 Probabi liti e s I /3 I /3 I /3 The mean of this distribution is (~/21(~/3)
|
From page 144... ...
This is a larger set of possible average ratings for A than either one of the RH methods gives. This is due to the richer set of samples available under the Boot method.
|
From page 145... ...
1985. Introduction to Variance Estimation.
|
From page 146... ...
This variability can be shown in the following manner: Associated with each coefficient al is a 95%-conficlence interval [L~, Uil, en cl by ranclomly selecting values for the coefficients within their confidence intervals, a predicted average rating rn can be generated for program n. A measure of how close the set of rn ratings is to the rn ratings can be calculated by r - r ~~ < p s F
|
From page 147... ...
is applied to the 1995 ratings of programs in Mathematics. Mathematics Using the STATA statistical package and applying a forward stepwise, least-squares linear regression on a large number of quantitative variables which characterized publications, citations, faculty size and rank, research grant support, number of doctorates by gender en cl race/ethnicity, graduate students by gentler, graduate student support, and time to degree, the following seven variables were identifier!
|
From page 148... ...
The following is scatter plot of the actual 1995 ratings and the predicted ratings. Plot of the Predicted Faculty Quality Score Against the Actual 1995 Score for Programs in Mathematics 6 .
|
From page 149... ...
Quality Score Predicted Ranks Bootstrap Ranks Maximum Minimum 1 st 3rd 1 st 3rd Institution Quartile Quartile Quartile Quartile Dartmouth College 2.73 2.51 73 76 53 62 Boston University 2.70 2.42 77 80 48 52 Brandeis University 3.17 2.88 49 51 32 36 Harvard University 4.41 4.09 8 9 2 4 Massachusetts Inst of Technology 5.27 4.93 2 2 3 4 U of Massachusetts at Amherst 3.40 3.11 38 40 54 60 Northeastern University 2.41 2.13 99 103 70 80 Brown University 4.60 4.31 5 6 26 29 Brown University-Applied Math 4.59 4.26 6 6 14 17 Universityof Rhode Island 1.69 1.40 128 129 122 125 University of Connecticut 2.66 2.39 79 83 98 102 Wesleyan University 2.31 2.09 104 107 101 110 Yale University 3.38 3.13 38 40 7 8 Adelphi University 1.07 0.82 138 138 130 133 CUNY - Grad Sch & Univ Center 3.38 3.10 40 41 30 32 Clarkson University 2.49 2.21 90 94 109 118 Columbia University 4.32 3.99 11 11 10 12 Cornell University 4.81 4.46 3 4 14 16 New York University 4.83 4.50 3 4 7 8 Polytechnic University 2.15 1.88 112 114 98 105 Rensselaer Polytechnic Inst 3.64 3.36 27 30 48 52 University of Rochester 3.10 2.83 52 54 56 62 149
|
From page 150... ...
50 State Univof New York-Albany 2.55 2.33 85 88 82 90 State Univof New York-Binghamton 2.55 2.33 85 87 65 75 State Univ of New York-Buffalo 3.00 2.76 57 59 61 70 State Univ of New York-Stony Brook 3.60 3.31 30 32 19 22 Syracuse University 2.42 2.18 95 100 76 84 Princeton University 4.52 4.21 7 7 2 3 Rutgers State Univ-New Brunswick 4.06 3.77 16 18 17 20 Stevens Inst of Technology 1.73 1.48 127 127 121 128 Carnegie Mellon University 3.63 3.33 28 31 34 40 English Language and Literature Applying the same method to the 1995 programs in English Language and Literature, a slightly different result is obtained, since programs in this field do not have the same productivity characteristics as those in Mathematics. Again, forward stepwise least squares linear regression was applied to a large number of quantitative variables, and the following were iclentifiec} as being the most significant: (nopubs2)
|
From page 151... ...
from the regression is 0.42429, and the variation in scores from the 1995 confidence interval calculation has an RMSE of 0.2544. r - - or- ~ -- I -- ~ -A -- - 0 - - - ~ Plot of the Predicted Faculty Quality Score Against the Actual 1995 Score for Programs in English Language and Literature 6 5 4 o U)
|
From page 152... ...
This was again done 100 times and the results are given as the Predicted Ranks in the table with the Ranclom Halves rankings. Quality Score Predicted Ranking Random Halves Ranks Maximum Maximum 1 st 3rd 1 st 3rd Institution Quartile Quartile Quartile Quartile Universityof New Hampshire 2.74 2.56 91 93 70 77 Boston College 2.57 2.42 96 98 59 64 Boston University 3.80 3.59 20 21 38 42 Brandeis University 3.63 3.40 19 21 44 55 Harvard University 5.55 5.05 1 1 2 3 U of Massachusetts at Amherst 3.84 3.51 30 34 38 43 Tufts University 2.35 2.22 108 110 67 74 Brown University 4.21 3.78 15 16 13 15 University of Rhode Island 2.39 2.22 113 115 94 113 University of Connecticut 3.26 3.05 53 57 79 87 Yale University 5.07 4.52 5 6 2 3 CUNY- Grad Sch & Univ Center 3.50 3.21 42 48 18 19 Columbia University 4.90 4.24 9 10 ~ 7 9 Cornell University 4.71 4.16 13 13 6 8 St John's University 1.93 1.86 127 127 119 122 Fordham University 2.38 2.23 103 106 104 112 New York University 3.59 3.25 26 28 18 20 Drew University 2.30 2.15 116 119 123 126 Universityof Rochester 3.30 3.02 30 33 44 48 State Univ of New York Binghamton 3.01 2.72 62 64 65 69 APPENDIX G
|
From page 153... ...
APPENDIX G 153 State Univ of New York-Buffalo 3.65 3.16 30 37 25 27 State U of New York-Stony Brook 3.17 2.77 48 55 46 52 Syracuse University 2.53 2.38 95 98 71 76 Indiana Univ of Pennsylvania 2.19 1.93 124 126 122 124 Princeton University 4.82 4.39 5 6 12 14 Rutgers State Univ-New Brunswick 3.96 3.62 22 23 16 18 Carnegie Mellon University 3.17 3.01 33 35 52 54
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.