The National Academies Press

Currently Skimming:

10 Summary and Discussion
Pages 175-212

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.

From page 175... ... of the reputational survey ratings and an examination of some factors that might possibly have influenced the survey results. This chapter concludes with suggestions for improving studies of this kind -- with particular attention given to the types of measures one would like to have available for an assessment of research-doctorate programs. Read the entire page →
From page 176... ... Survey Results (Measures 08-117. Differences in the mean ratings derived from the reputational survey are not large. Read the entire page →
From page 177... ... 177 TABLE 10.1 Mean Values for Each Program Measure, by Discipline Anthro- Political Psych- Soci pology Economics Geography History Science ology ology Program Size 01 17 23 13 28 23 29 21 02 28 41 16 38 35 71 33 03 51 68 22 51 50 102 49 Program Graduates 04 .48 .21 .26 .26 .28 .39 .38 05 8.3 7.3 8.7 9.2 8.3 6.2 8.2 06 .60 .78 .72 .56 .68 .69 .75 07 .28 .26 .28 .16 .26 .24 .33 Survey Results 08 2.8 2.3 2.8 2.6 2.6 2.5 2.5 09 1.6 1.3 1.6 1.6 1.5 1.6 1.5 10 1.0 1.1 1.0 1.1 1.1 1.1 1.0 11 1.1 .9 1.2 .9 1.0 .7 1,0 University Library 12 .4 .1 .4 .2 .2 .1 .2 Research Support 13 .22 .11 .14 NA . 06 .21 .12 14 NA 832 NA NA 520 1003 790 Publication Records 17 30 52 17 43 43 81 52 18 .61 .63 .51 .58 .63 .66 .66 Total Programs 70 93 49 102 83 150 92 Read the entire page →
From page 178... ... Research Support {Measures 13-14~. Measure 13, the proportion of program faculty who had received ADAMHA, NIH, OR NSF4 research grant awards during the FY1978-80 period, has mean values ranging from .22 and .21 in anthropology and psychology' respectively, to .06 in political science. Read the entire page →
From page 179... ... , survey rating of the scholarly quality of program faculty (measure 08) , university R&D expenditures in a particular discipline (measure 14) Read the entire page →
From page 180... ... As might be expected, correlations of this measure with the other two measures of program size -- number of faculty and doctoral student enrollment -- are reasonably high in all seven disciplines. Of greater interest are the strong positive correlations between measure 02 and measures derived from either reputational survey ratings or publication records. Read the entire page →
From page 181... ... 18 06 .43 . 32 .13 .09 .06 -.06 .19 07 .35 .33 .30 .43 .20 -.06 .32 Survey Results 08 .71 .75 .60 .74 .60 .31 .72 09 .68 .74 .68 .72 .56 .23 .73 10 -.15 .00 .00 .02 -.04 -.04 -.03 11 .67 .71 .57 .77 .58 .39 .68 On iver s ity Library 12 .68 .57 .30 .73 .66 .36 .54 Research Support 13 .39 .54 .42 N/A .24 - . Read the entire page →
From page 182... ... .47 Survey Results 09 .96 .98 .98 .98 .98 .97 .98 10 .21 .35 .19 .24 .13 .05 .33 11 .95 .97 .94 .97 .98 .97 .97 University Library 12 .64 .67 .52 .71 .74 .73 .75 Research Suppor t 13 .46 .76 .52 N/A .40 .75 .63 14 N/A .44 N/A N/A .43 � 49 � 30 Publication Records 17 .75 .78 .78 .79 .71 .74 .80 18 .26 .47 .59 .53 .44 .57 .49 Read the entire page →
From page 183... ... are considerably smaller. The correlation of rated faculty quality with measure 05, the shortness of time from matriculation in graduate school to award of the doctorate, is positive but small in each of the social and behavioral science disciplines. Read the entire page →
From page 184... ... The pattern of relations is quite similar for programs in economics, political science, psychology, and sociology: moderately high correlations with both measures of program size and reputational survey results (except measure 10) and somewhat higher correlations with measure 17, the total number of faculty articles. Read the entire page →
From page 185... ... with Other Measures, by Discipline Anthropology .scipl ine Political Psych- Soci Economics Geography History Science ology ology Program Size 01 N/A . Read the entire page →
From page 186... ... with Other Measures, by Discipline Anthro- Political Psych- Soci pology Economics Geography History Science ology ology Program Size 01 .82 .79 .56 .86 .78 .76 .69 02 .70 .76 .46 .82 .50 .49 .63 03 .53 .47 .33 .75 .42 .34 .47 Program Graduates 04 .14 .32 . Read the entire page →
From page 187... ... For example, in judging the scholarly quality of faculty (measure 08) , survey respondents in each discipline rated between 6 and 8 percent of the programs as being "distinguished" and between 2 and 7 percent as "not sufficient for doctoral education" (see Table 10.6) Read the entire page →
From page 188... ... 188 TABLE 10.6 Distribution of Responses to Each Survey Item, by Discipline Anthro- Eco- Geog- Political Psych- Soci Survey Measure Total pology nomics raphy History Science ology ology 08 SCHOLARLY QUALITY OF PROGRAM FACULTY Distinguished 7.1 6.5 8.0 7.4 7.9 7.1 6.5 6.7 Strong 15.8 18.3 12.0 19.8 16.6 16.2 14.5 16.8 Good 21.3 27.6 17.9 27.4 20.5 25.7 16.6 21.1 Adequate 18.1 21.4 19.5 20.5 16.7 18.9 14.3 19.7 Marginal 9.6 8.5 13.4 8.6 8.3 10.5 7.3 11.2 Not Sufficient for Doctoral Education 4.1 1.9 6.6 4.1 4.6 4.4 2.3 5.1 Don't Know Well Enough to Evaluate 23.9 15.7 22.6 12.2 25.3 17.2 38.6 19.3 TOTAL 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 09 EFFECTIVENESS OF PROGRAM IN EDUCATING SCIENTISTS Extremely Effective 7.0 6.4 7.2 9.9 7.1 7.4 5.8 6.7 Reasonably Effective 27.1 33.0 22.2 40.2 25.8 28.5 22.9 27.0 Minimally Effective 16.0 17.5 17.9 21.9 15.6 18.5 10.8 15.9 Not Effective 4.7 3.3 6.8 6.1 3.5 6.5 2.7 5.7 Don't Know Well Enough to Evaluate 45.3 39.8 45.8 22.0 48.0 39.1 57.9 44.7 TOTAL 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 10 CHANGE IN PROGRAM QUALITY IN LAST FIVE YEARS Better 11.8 12.6 13.8 16.5 10.1 12.9 8.4 12.5 Little or No Change 27.7 32.0 27.7 41.3 27.1 32.3 19.1 27.2 Poorer 8.5 11.3 6.1 17.2 7.4 9.3 5.5 9.3 Don't Know Well Enough to Evaluate 52.0 44.2 52.4 25.1 55.5 45.6 67.0 51.0 TOTAL 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 11 FAMILIARITY WITH WORK OF PROGRAM FACULTY Considerable 23.8 27.3 23.2 33.7 23.7 25.2 16.7 26.1 Some 42.8 49.6 41.0. 48.2 43.8 47.3 35.6 43.5 Little or None 32.0 22.5 34.5 17.0 30.5 26.3 46.1 28.8 No Response 1.5 .7 1.3 1.2 2.0 1.2 1.7 1.7 TOTAL 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 NOTE: For survey measures 08, 09, 10 the "don't knows category includes a small number of cases for which the respondents provided no response to the survey item. Read the entire page →
From page 189... ... For example, for social and behavTABLE 10.7 Survey Item Response Rates, by Discipline and Mean Rating on Measure 08 Survey Measure Total Anthro- Eco- Geog- Political Psych- Socipology nomics raphy History Science ology ology 08 SCHOLARLY QUALITY OF PROGRAM FACULTY Mean Rating on Measure 08 4.0 or Higher 96.9 97.3 98.1 98.6 97.4 98.8 92.5 97.7 3.0 - 3.9 90.2 93.2 94.9 94.0 89.2 94.0 82.3 94.2 2.0 - 2.9 77.4 81.2 84.9 88.1 72.6 84.5 61.0 81.8 Less than 2.0 57.9 68.4 63.5 72.9 58.7 65.6 38.9 65.6 09 EFFECTIVENESS OF PROGRAM IN EDUCATING SCIENTISTS Mean Rating on Measure 08 4.0 or Higher 86.3 91.9 89.6 96.1 84.2 90.1 75.5 86.1 3.0 - 3.9 70.0 71.9 74.9 88.4 66.1 74.7 60.6 70.4 2.0 - 2.9 53.2 54.2 57.7 76.3 47.3 57.6 39.8 52.5 Less than 2.0 35.7 39.2 37.7 59.0 36.2 44.0 22.6 38.7 10 CHANGE IN PROGRAM QUALITY IN LAST FIVE YEARS Mean Rating on Measure 08 4.0 or Higher 76.7 83.4 82.0 91.1 73.6 82.1 59.1 78.8 2 0 - 2 9 68 5 68 4 50 0 84 3 59 3 67 3 47 7 64 8 Less than 2.0 28.0 34.3 29.5 55.2 27.0 34.3 16.6 29.9 Read the entire page →
From page 190... ... For programs with higher mean ratings the estimated errors associated with these means are generally smaller -- a finding consistent with the fact that survey respondents were more likely to furnish evaluations for programs with high reputational standing. The "split-half" correlations presented in Table 10.8 give an indication of the overall reliability of the survey results in each discipline and for each measure. Read the entire page →
From page 191... ... Deviation Correlation Group A Group B Group A Group B N r Anthropology 1.62 1.61 .42 .41 70 .94 Economics 1.33 1.33 .63 .64 93 .96 Geography 1.62 1.62 .49 .47 49 .95 History 1.54 1.56 .55 .51 102 .93 Political Science 1.48 1.46 .55 .56 83 .95 Psychology 1.56 1.52 .54 .54 150 .92 Sociology 1.47 1.48 .57 .53 92 .95 MEASURE 10: IMPROVEMENT IN PROGRAM IN LAST FIVE YEARS Discipline Mean Rating Std. Deviation Correlation Group A Group B Group A Group B N r Anthropology 1.03 1.01 .26 .27 70 .78 Economics 1.13 1.11 .29 .27 93 .84 Geography .97 1.01 .30 .32 49 .94 History 1.06 1.04 .21 .24 102 .63 Political Science 1.06 1.06 .22 .24 83 .75 Psychology 1.07 1.09 .27 .26 150 .64 Sociology 1.03 1.04 .31 .32 92 .85 MEASURE 11: FAMILIARITY WITH WORK OF PROGRAM FACULTY Discipline Mean Rating Std. Read the entire page →
From page 192... ... It is evident from the data reported in Table 10.10 that mean ratings of the scholarly quality of program faculty tend to be higher if the evaluator has considerable familiarity with the program. There is nothing surprising or, for that matter, disconcerting about such an association. Read the entire page →
From page 193... ... 193 TABLE 10.9 Comparison of Mean Ratings for 11 Mathematics Programs Included in Two Separate Survey Administrations Evaluators Rating the Same Survey All Evaluators Program in Both Surveys Measure First Second First Second N X N X N X N X Program A 08 100 4.9 114 4.9 50 4.9 50 4.9 09 90 2.7 100 2.8 42 2.7 43 2.7 10 74 1.2 83 1.2 38 1.1 34 1.2 11 100 1.6 115 1.6 50 1.5 50 1.6 Program B 08 94 4.6 115 4.6 48 4.6 flu 4. Read the entire page →
From page 194... ... (This similarity arises, in part, because only a small fraction of evaluations are given on the basis of no more than ~little" familiarity with the program.) The third column in Table 10.10 presents the correlation in each d iscipline between the array of mean ratings supplied by respondents claiming "considerable" familiarity and the mean ratings of those indicating "some" or "little or no" familiarity with particular programs. Read the entire page →
From page 195... ... In all disciplines except anthropology, the mean ratings provided by the group furnished faculty names are lower than the mean ratings supplied by other respondents (see Table 10.12~. Although the differences are small, they attract attention because they are consistent with findings in the mathematical and physical sciences, humanities, engineering, and biological sciences and because the direction of the differences was not anticipated. Read the entire page →
From page 196... ... This hypothesis may apply especially to the smaller and less visible programs and is confirmed TABLE 10.12 Mean Ratings of Scholarly Quality of Program Faculty, by Type of Survey Form Provided to Evaluator MEAN RATINGS Names No Name_ CORRELATION Anthropology 2.79 2.56 .89 70 Economics 2.29 2.30 .95 93 Geography 2.74 2.78 .96 49 History 2.61 2.62 .90 102 Political Science 2.57 2.82 .90 83 Psychology 2.55 2.70 .82 146 Sociology 2.46 2.90 .91 92 NOTE: N reported in last column represents the number of programs with a rating from at least one evaluator in each of the two groups. Read the entire page →
From page 197... ... Furthermore, the high correlations found between the mean ratings of the two groups indicate that the relative standings of programs are not dramatically influenced by the geographic proximity of those evaluating them. Another consideration that troubles some critics is that large programs may be unfairly favored in a faculty survey because they are 2 See Appendix I for a list of the states included in each region. Read the entire page →
From page 198... ... To be certain that this was the case, mean ratings of the scholarly quality of faculty were recalculated for every social and behavioral science program -- with the evaluations provided by alumni excluded. The results were compared with the mean scores based on a full set of evaluations. Read the entire page →
From page 199... ... Similar findings, presented in the mathematical and physical sciences volume of the committee's report, were obtained from an analysis of survey evaluators in differing specialties within physics and within statistics/biostatistics. TABLE 10.15 Mean Ratings of Scholarly Quality of Program Faculty, by Evaluator's Field of Specialty within Psychology MEAN RATINGS CORRELATION Clinical Other r N Psychology 2.63 2.52 .91 150 NOTE: N reported in last column represents the number of programs with a rating from at least one evaluator in each of the two groups. Read the entire page →
From page 200... ... On the other hand, it is wise to keep in mind that these reputational ratings are measures of Perceived program quality rather than of ~quality" in some ideal or absolute sense. What this means is that, just as for all of the more objective measures, the reputational ratings represent only a partial view of what most of us would consider quality to be; hence, they must be kept in careful perspective. Read the entire page →
From page 201... ... Although there are obvious similarities in the two surveys, there are also some important differences that should be kept in mind in examining individual program ratings of the scholarly quality of faculty. Already mentioned in this chapter is the inclusion, on the form sent to 90 percent of the sample members in the committee's survey, of the names and academic ranks of faculty and the numbers of doctoral graduates in the previous five years. Read the entire page →
From page 202... ... Figures 10.1-10.7 plot the mean ratings of scholarly quality of faculty in programs included in both surveys; sets of ratings are graphed for 38 programs in anthropology, 71 in economics, 31 in geography, 79 in history, 61 in political science, 103 in psychology, and 65 in sociology. Since in the Roose-Andersen study programs were identified by institution and discipline (but not by department) Read the entire page →
From page 203... ... 4.0 FIGURE 10.1 Mean rating of scholarly quality of faculty (measure 08) versus mean rating of faculty in the Roose-Andersen study -- 38 programs in anthropology. Read the entire page →
From page 204... ... FIGURE 10.2 Mean rating of scholarly quality of faculty (measure 08) versus mean rating of faculty in the Roose-Andersen study -- 71 programs in economics. Read the entire page →
From page 205... ... FIGURE 10.3 Mean rating of scholarly quality of faculty (measure 08) versus mean rating of faculty in the Roose-Andersen study -- 31 programs in geography. Read the entire page →
From page 206... ... FIGURE 10.4 Mean rating of scholarly quality of faculty (measure 08) versus mean rating of faculty in the Roose-Andersen study -- 79 programs in history. Read the entire page →
From page 207... ... 4.0 FIGURE 10.5 Mean rating of scholarly quality of faculty (measure 08) versus mean rating of faculty in the Roose-Andersen study -- 61 programs in political science. Read the entire page →
From page 208... ... FIGURE 10.6 Mean rating of scholarly quality of faculty (measure 08) versus mean rating of faculty in the Roose-Andersen study -- 103 programs in psychology. Read the entire page →
From page 209... ... FIGURE 10.7 Mean rating of scholarly quality of faculty (measure 08) versus mean rating of faculty in the Roose-Andersen study -- 65 programs in sociology. Read the entire page →
From page 210... ... How do program graduates compare with regard to their publication records? Also desired might be measures of the quality of the students applying for admittance to a graduate program (e.g., Graduate Record Examination scores, undergraduate grade point averages) Read the entire page →
From page 211... ... To add evaluators from nonacademic sectors would require a major effort in identifying the survey population from which a sample could be selected. Although such an effort is likely to involve considerable costs in both time and financial resources, the come mittee believes that the addition of evaluators from nonacademic settings would be of value in providing a different perspective to the reputational assessment and that comparisons between the ratings supplied by academic and nonacademic evaluators would be of particular interest. Read the entire page →

From page 175...

... of the reputational survey ratings and an examination of some factors that might possibly have influenced the survey results. This chapter concludes with suggestions for improving studies of this kind -- with particular attention given to the types of measures one would like to have available for an assessment of research-doctorate programs.

10 Summary and Discussion Pages 175-212

10 Summary and Discussion
Pages 175-212