A Data-Based Assessment of Research-Doctorate Programs in the United States (with CD) (2011) / Chapter Skim
Currently Skimming:
Appendix J: A Technical Discussion of the Process of Rating and Ranking Programs in a Field
Appendix J: A Technical Discussion of the Process of Rating and Ranking Programs in a Field
Pages 285-304
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 285... ...
-based weights, the regression (R) -based weights, and the details of the calculations of the endpoints of the 90 percent ranges • the simulation of the uncertainty in the weights by random-halves sampling, • the simulation of the uncertainty in the values of the program variables, • the combination of the simulated weights for the significant program variables with the simulated standardized values of the program variables to obtain simulated rankings, and • the resulting 90 percent ranges of rankings that are the primary rating and ranking quantities that we report.
|
From page 286... ...
A complete array with a faculty raters. An incomplete array, with ratings importance value for every program variable by value for every program (that satisfies the only for the sampled programs and rated only every responding faculty member.
|
From page 287... ...
At least 8 program characteristics will have a score of 0 for each faculty respondent, more than 8 would be zero if the respondent selected less than 4 as the "important" or 2 as the "most important" characteristics. A final question asked faculty respondents to indicate the relative importance of each of the three categories by assigning them values that summed to 100 over the three 1 The importance of program attributes to program quality is surveyed in Section G of the faculty questionnaire.
|
From page 288... ...
Faculty characteristics Tablei.J-1. The 21 Program Characteristics Listed in the Faculty Questionnaire Number of publications per faculty member ii.
|
From page 289... ...
The averages, { xk }, are the direct or surveybased weights of the faculty respondents because they directly give the average relative importance of each program variable, as indicated by the faculty questionnaire responses in the field of study. Thus, the final 20 importance measures of the program characteristics for each faculty respondent are non-negative and sum to 1.0.
|
From page 290... ...
Because each program's average rating is determined by a different random sample of graduate faculty raters, it is highly unlikely that any two programs will be evaluated by exactly the same set of raters. Denote the vector of the average ratings for the sampled programs in a field by r .
|
From page 291... ...
The resulting unstable regression coefficients would have been unusable for our purposes. For example, as expected, when we fit a linear model that included all 20 of the program variables, we found that for a number of the variables, the coefficients and their signs did not make intuitive sense.
|
From page 292... ...
ˆ (5) In the same way, the matrix of estimated variances and covariances of γˆ , obtained from the least-squares output, may be transformed to the corresponding matrix for m .4ˆ th ˆ The regression coefficient for the k program variable, denoted by m , is the regressionbased weight for program characteristic k as a predictor of the average ratings of the programs by ˆ ˆ ˆ ˆ the faculty raters, and m = (m1 , m2 ,..., m20 )
|
From page 293... ...
The data for mathematics, reported in Assessing Research Doctorate Programs: A Methodology Study, indicate that using 49 programs did a reasonably good job of reproducing the predictions based on the whole field of 147 mathematics programs.5 Thus, we decided that in developing the regression-based ratings, we would use a sample of 50 programs from a field if it had more than 50 programs and use almost all of the programs in fields with 50 or fewer programs. When there were fewer than 30 programs in a field, it was combined with a larger discipline with similar direct weights for the purposes of estimating the regression-based weights.6 In two cases, computer engineering and engineering science and materials, there were fewer than 25 programs, and these fields were not ranked, although data are reported for all 20 characteristics.7 ˆ There is one final alteration in the values of m that needs to be mentioned.
|
From page 294... ...
X is a complete array whose rows denote the N faculty respondents, while R is an incomplete array whose rows denote the n sampled faculty raters for a field. In the case of X, the RH procedure requires a random sample of size N/2 of the faculty respondents.
|
From page 295... ...
There are other reasons to expect the RH method to produce a useful simulation of the uncertainty of averages.10 The same reasoning applies to the RH sampling of the faculty raters in R to simulate the uncertainty in the average ratings, r , used to obtain the regression-based weights. The procedure was to sample a random half of all raters for programs in a field and compute the average rating for each program from that half sample.
|
From page 296... ...
Each program had its own relative error factor for each program variable, ejk. Just as we had simulated values from the sampling distributions of x and r via RH sampling, we also wanted to reflect the uncertainty in the values of the program variables themselves rather than using the fixed values, {pkj}, in computing program ratings.
|
From page 297... ...
and regression-based weights and in the program data values five percent of a program's rankings in our process are less than this interval and five percent are higher. The interval itself represents what we would expect the typical rankings for that program to be, given the uncertainty in the process and the ratings of the other programs in the field.11 These ninety percent ranges are reported for the R and S measures, as well as for the three dimensional measures.
|
From page 298... ...
In fact, the dimensional measures described in the body of this Guide, are an example12. The technical description of further steps that the committee carried out to obtain ranges of rankings using the combined measures are described in this section -- beginning with an alternative conceptual diagram.
|
From page 299... ...
Perform backwards stepwise regression to obtain a stable fitted equation predicting average ratings from using the optimal fraction to form the remaining PCs. the combined weights, f0.
|
From page 300... ...
Remembering that the standardized values of the program variables for program j are denoted by pjk* , the direct rating for program j, using the average direct weight vector, x , is Xj, is given by 20 Xj = ∑x k =1 k p jk *
|
From page 301... ...
ˆ However, because both mk and xk are subject to uncertainty, we made one additional adjustment to Equation 10 that is described below, following the discussion of how we simulated the uncertainty in both the direct weights and in the average ratings used to form the regressionbased weights.
|
From page 302... ...
denotes the variance of the individual direct weights given to the kth program variable by these faculty respondents. The value of σ2( mk )
|
From page 303... ...
We did examine a method that did, but it simply produced a matrix version of Equation 12 that reduced to the procedure we used when the program variables were uncorrelated, but was otherwise difficult to implement with the resources available to us. Box 8: Eliminating Non-Significant Program Variables After we have obtained the 500 simulated values of the combined weights by applying Equations 17 and 20 to the 500 simulated values for the direct and regression-based weights, we were in a position to examine the distributions of these 500 values of the combined weights for each program variable.
|
From page 304... ...
of Figure J-1. The values for the combined weights that correspond to the eliminated variables are set to 0.0 in each of the final 500 simulated values of f0.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.