The National Academies Press

Currently Skimming:

12 Summary and Discussion
Pages 181-216

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.

From page 181... ... Also presented here are an analysis of the reliability (consistency) of the reputational survey ratings and an examination of some factors that might possibly have influenced the survey results. Read the entire page →
From page 182... ... In all nine disciplines the mean rating of scholarly quality of program faculty See Table 2.1 for a description of each of the measures and the units in which values of a measure are reported. 2 The second table in each of the nine earlier chapters presents the standard deviation and decile values for each measure. Read the entire page →
From page 183... ... 183 TABLE 12.1 Mean Values for Each Program Measure, by Discipline Art Linguis- Philos History Classics English French German tics Music ophy Spanish Program Size 01 13 11 31 11 9 14 20 14 10 02 18 10 44 15 13 19 26 18 13 03 33 17 62 20 15 34 42 29 24 Program Graduates 04 .32 .28 .20 .26 .28 .36 .12 .27 .24 05 9.3 7.7 9.1 9.2 8.9 7.9 10.0 7.9 9.0 06 .67 .58 .57 .48 .51 .62 .64 .57 .60 07 .35 .32 .20 .19 .25 .28 .24 .25 .27 S urvey Results 08 2.7 2.9 2.5 2.6 2.9 2.8 2.8 2.6 2.7 09 1.5 1.6 1.5 1.6 1.7 1.6 1.6 1.5 1.6 10 1.1 .9 1.0 1.0 .9 1.1 1.1 1.1 1.0 11 1.1 1.2 .9 1.0 1.2 1.2 1.0 1.1 1.1 University Librar, 12 .7 1.0 .2 .5 .5 .8 .6 .3 .3 Total Programs 41 35 106 58 48 35 53 77 69 Read the entire page →
From page 184... ... Special attention is given to the correlations involving the number of FY197S-79 program graduates (measure 02) and the survey rating of the scholarly quality of program faculty (measure 08~. Read the entire page →
From page 185... ... As might be expected, correlations of this measure with the other two measures of program size -- number of faculty and doctoral student enrollment -- are reasonably high in all nine disciplines. Of greater interest are the strong positive correlations in many disciplines between measure 02 and measures derived from either reputational survey ratings or university library size. Read the entire page →
From page 186... ... 03 .13 -. 27 06 .33 .21 .02 -.11 .08 .03 .12 .19 .08 07 .13 .25 .21 .07 -.05 .05 -.17 .28 .10 Survey Results 08 .76 .66 .68 .64 .58 .50 .12 .42 .42 09 .74 .72 .66 .67 .66 .53 .13 .45 .48 10 - .06 .07 . Read the entire page →
From page 187... ... 57 . 28 .61 .15 Survey Results 96 .98 97 98 .98 .99 .97 97 10 , 31 . Read the entire page →
From page 188... ... Nonetheless, certain similarities across disciplines may be seen in the correlations among the measures. High correlations consistently appear among measures 08, 09, and 11 from the reputational survey, and these measures also are prominently related to program size (measures 01, 02, and 03~-except in music -- and to library size (measure 12 -- except in linguistics. Read the entire page →
From page 189... ... and to employment prospects of program graduates (especially measure 07~. ANALYS I S OF THE SURVEY RESPONSE Measures 08-11, derived from the reputational survey, may be of particular interest to many readers since measures of this type have been the most widely used (and frequently criticized) Read the entire page →
From page 190... ... 24.8 36.3 Minimally Effeetive 17.1 19.5 19.9 13.9 16.7 18.9 16.3 18.0 14.6 20.8 Not Effeetive 4.8 6.5 5.2 4.2 4.0 4.6 4.5 5.9 5.9 3.6 Don't Know Well Enough to Evaluate 38.8 32.2 28.7 48.8 39.4 25.1 36.4 37.7 48.0 32.4 TOTAL 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 10 CHANGE IN PROGRAM QUALITY IN LAST FIVE YEARS Better 11.2 14.9 10.9 8.3 11.3 9.5 16.6 9.0 11.8 11.9 Little or No Change 31.0 31.2 34.0 21.4 31.5 41.9 31.7 28.9 32.7 34.5 Poorer 9.9 8.9 15.2 7.3 10.0 14.6 10.4 6.5 8.0 12.0 Don't Know Well Enough to Evaluate 47.9 45.0 39.9 63.0 47.2 33.9 41.3 55.6 47.6 41.5 TOTAL 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 11 FAMILIARITY WITH WORK OF PROGRAM FACULTY Considerable 30.2 33.2 39.7 20.4 28.1 36.8 37.9 26.5 30.7 33.0 Some 43.8 40.4 43.7 44.2 44.6 45.8 41.3 41.6 42.6 46.9 Little or None 25.1 24.5 16.3 34.5 26.9 16.7 20.4 29.7 26.2 18.6 No Response 9 1 9 4 9 3 7 4 2 3 5 1 4 TOTAL 100 0 100 0 100 0 100 0 100 0 100 0 100 0 100 0 100 0 100 0 NOTE: For survey measures 08, 09, 10 the Don't know" category includes a small number of eases for which the respondents provided no response to the survey item. Read the entire page →
From page 191... ... The corresponding percentages of "don't know" responses for measures 09 and 10 are considerably larger -- 39 and 48 percent, respectively -- suggesting that survey respondents found it more difficult (or were less willing) to judge program effectiveness and change than to judge the scholarly quality of program faculty. Read the entire page →
From page 192... ... 192 S urvey Measur e TABLE 12.5 Survey Item Response Rates, by Discipline and Mean Rating on Measure 08 Art Total History Classics English French German tics Music ophy Spanish Linguis- Philos 0 8 SCHOLARLY QUALITY OF PROGRAM FACULTY Mean Rating on Measure 08 4.0 or Higher 97.1 99.3 98.1 98.1 99.0 97.5 99.7 87.6 96.3 99.2 3 0 - 3.9 93.3 93.0 95.4 93.2 94.0 94.2 98.9 82.6 92.5 94.7 2 0 - 2.9 81.3 81.3 88.6 73.9 80.1 86.2 90.0 73.1 81.2 84.5 Less than 2.0 65.7 64.7 72.5 57.8 63.4 79.6 60.0 58.1 67.1 78.3 O 9 EFFECTIVENESS OF PROGRAM IN EDUCATING SCIENTISTS Mean Rating on Measure 08 4.0 or Higher 89.0 95.7 94.8 86.2 91.9 91.4 96.0 79.8 82.2 92.9 3.0 - 3.9 75.4 81.8 80.1 67.7 77.5 80.9 84.6 70.6 63.2 82.0 2.0 - 2.9 57.4 63.8 67.1 48.1 57.1 68.3 60.0 60.9 48.5 62.2 Less than 2.0 42.8 48.6 50.6 34.8 41.6 62.2 31.9 46.0 37.2 53.8 10 CHANGE IN PROGRAM QUALITY IN LAST FIVE YEARS Mean Rating on Measure 08 4.0 or Higher 79.1 86.5 83.4 68.7 84.5 84.2 86.5 65.1 78.9 85.0 3.0 - 3.9 66.7 72.5 69.3 50.7 69.3 73.5 78.7 55.9 66.7 73.1 2.0 - 2.9 49.2 51.8 56.4 35.1 49.9 59.8 58.5 40.5 51.6 52.8 Less than 2.0 32.1 30.5 38.4 22,0 32.9 50.2 22.0 26.3 34.2 44.7 Read the entire page →
From page 193... ... Deviation Correlation Group A Group B Group A Group B N r Art History 1.51 1.55 .63 .66 41 .97 Classics 1.62 1.60 .50 .55 35 .95 English 1.51 1.53 .55 .52 106 .94 French 1.59 1.59 .48 .49 58 .95 German 1.72 1.70 .S0 .52 48 .97 Linguistics 1.61 1.59 .57 .55 3S .98 Music 1.S4 1.S9 .S6 .S6 S3 .90 Philosophy 1.48 1.47 .S3 .S4 77 .9S Spanish 1.61 1.63 .40 .40 69 .94 MEASURE 10: IMPROVEMENT IN PROGRAM IN LAST FIVE YEARS Discipline Mean Rating Std. Deviation Correlation Group A Group B Group A Group B N r Art History 1.09 1.11 .22 .22 41 .72 Classics .94 .90 .22 .24 3S .68 English .99 1.00 .26 .2S 106 .66 French 1.01 1.01 .2S .27 S8 .78 German .91 .90 .22 .24 48 .8S Linguistics 1.08 1.0S .38 .40 3S .90 Music 1.0S 1.03 .20 .22 S3 .S6 Philosophy 1.06 1.0S .32 .29 77 .88 Spanish .97 1.02 .24 .26 69 .75 MEASURE 11: FAMILIARITY WITH WORK OF PROGRAM FACULTY Discipline Mean Rating Std. Read the entire page →
From page 194... ... As mentioned in Chapter VI of the mathematical and physical science volume of the committee's report, 11 mathematics programs, 7 selected at random, were included on a second form sent to 178 survey respondents in this discipline, and 116 individuals (65 percent) furnished responses to the second survey. Read the entire page →
From page 195... ... 195 TABLE 12.7 Comparison of Mean Ratings for 11 Mathematics Programs Included in Two Separate Survey Administrations Survey All Evaluators Measure First Second First N X N X N X Evaluators Rating the Same Program in Both Surveys Second N X 08 100 4.9 114 4.9 50 4.9 50 4.9 09 90 2.7 100 2.8 42 2.7 43 2.7 10 74 1.2 83 1.2 38 1.1 34 1.2 11 100 1.6 115 1.6 50 1.5 50 1.6 08 94 4.6 115 4.6 48 4.6 50 4.5 09 81 2.6 91 2.5 40 2.6 39 2.5 10 69 1.0 82 1.0 37 1.0 36 0.9 11 98 1.4 116 1.4 50 1.5 50 1.5 Program C 08 86 3.4 103 3.6 42 3.4 44 3.5 09 56 2.0 66 2.1 28 2.1 29 2.0 10 55 1.1 62 1.3 30 1.2 27 1.4 11 99 1.0 116 1.1 50 1.1 50 1.0 Program D 08 74 3.0 93 3.0 37 2.8 38 2.9 09 50 1.8 48 1.6 27 1.7 16 1.6 10 46 1.4 52 1.5 24 1.4 23 1.5 11 90 1.0 113 0.9 46 1.0 46 0.9 Program E 08 69 3.0 95 3.1 39 3.0 46 3.1 09 40 1.8 60 1.9 25 1.8 30 1.8 10 36 0.8 58 0.9 24 0.8 29 0.9 11 96 0.8 115 0.9 52 0.9 52 1.0 Program F 08 63 2.9 90 3.0 26 3.0 32 3.1 09 35 1.8 46 1.7 10 1.6 13 1.8 10 32 1.1 43 1.1 11 1.3 12 1.2 11 95 0.7 115 0.8 43 0.7 44 0.7 Program G 08 69 2.7 92 2.8 39 2.7 39 3.0 09 35 1.7 45 1.6 17 1.7 19 1.7 10 36 1.1 43 1.2 17 1.1 19 1.2 11 85 0.9 116 0.8 46 0.9 46 0.9 Program H 08 58 2.2 73 2.5 36 2.2 37 2.4 09 32 1.3 43 1.3 22 1.2 19 1.3 10 30 1.5 39 1.5 20 1.7 17 1.4 11 90 0.7 116 0.6 51 0.7 52 0.6 08 55 2.0 74 1.9 30 1.9 30 2.0 09 33 1.0 41 0.9 19 1.0 18 0.8 10 27 1.2 31 1.1 15 1.1 13 1.2 11 99 0.5 115 0.5 50 0.5 50 0.5 Program J 08 51 1.5 67 1.5 26 1.4 28 1.4 09 31 0.8 36 0.7 14 0.6 14 0.7 10 26 1.2 23 1.1 14 1.2 12 1.3 11 96 0.5 113 0.3 49 0.4 48 0.4 Program K 08 33 1.2 48 1.2 17 1.1 21 1.4 09 19 0.8 21 0.5 11 0.6 8 0.4 10 12 0.8 15 0.9 5 1.0 5 0.8 11 99 0.2 114 0.2 48 0.2 47 0.2 Read the entire page →
From page 196... ... While visibility and quality are scarcely the same, the world of research in higher education is structured to encourage high quality to achieve high visibility, so that any association of the two is far from spurious. TABLE 12.8 Mean Ratings of Scholarly Quality of Program Faculty, by Evaluator's Familiarity with Work of Faculty MEAN RATINGS CORRELATION Consid- Some/ erable Little r N Art History 2.75 2.59 .97 41 Classics 3.05 2.78 .96 35 English 2.94 2.37 .87 106 French 2.84 2.50 .95 58 German 3.09 2.76 .97 48 Linguistics 3.09 2.60 .95 34 Music 2.98 2.67 .91 53 Philosophy 2.96 2.38 .92 77 Spanish 2.90 2.48 .94 69 NOTE: N reported in last column represents the number of programs with a rating from at least one evaluator in each of the two groups. Read the entire page →
From page 197... ... in the previous five years. Since earlier reputational surveys had not provided such information, 10 percent of the sample members, randomly selected, were given forms without faculty names or doctoral data, as a "control group." Although one might expect that those given faculty names would have been more likely than other survey respondents to provide evaluations of the scholarly quality of program faculty, consistently large differences were not found (see Table 12.9) Read the entire page →
From page 198... ... However interesting these effects may be, one should not lose sight of the fact that they are small at best and that their existence does not necessarily imply that a program's relative standing on TABLE 12.10 Mean Ratings of Scholarly Quality of Program Faculty, by Type of Survey Form Provided to Evaluator MEAN RATINGS CORRELATION No Names r N Art History 2.64 2.93 .92 41 Classics 2.89 3.18 .91 35 English 2.49 2.58 .91 105 French 2.61 2.70 .84 58 German 2.87 3.14 .91 48 Linguistics 2.73 3.18 .91 35 Music 2.78 2.85 .92 53 Philosophy 2.55 2.60 .93 77 Spanish 2.66 2.56 .86 69 NOTE: N reported in last column represents the number of programs with a rating from at least one evaluator in each of the two groups. eAs shown in Table 2.3, the survey response rate for those furnished faculty names is approximately 6 percentage points higher than that for those not given this information. Read the entire page →
From page 199... ... However, the correlations between the mean ratings supplied by the two groups are reasonably high -- ranging from .84 to .93 in the nine disciplines (see Table 12.10~. Were these coefficients adjusted for the fact that the group furnished forms without names constituted only about 10 percent of the survey respondents they would be substantially larger. Read the entire page →
From page 200... ... To be certain that this was the case, mean ratings of the scholarly quality of faculty were recalculated for every humanities program -- with the evaluations provided by alumni excluded. The results were compared with the mean scores based on a full set of evaluations. Read the entire page →
From page 201... ... . The results of the analysis, which are presented in the mathematical and physical science volume of the committee's report, indicate that there is a high degree of correlation in the mean ratings furnished by those in differing specialty fields within these two disciplines. Read the entire page →
From page 202... ... This generality has self-evident appeal On the other hand, it is wise to keep in mind that these reputational ratings are measures of perceived program quality rather than of "quality" in some ideal or absolute sense. What this means is that, just as for all of the more objective measures, the reputational ratings represent only a partial view of what most of us would consider quality to be; hence, they must be kept in careful perspective. Read the entire page →
From page 203... ... Although there are obvious similarities in the two surveys, there are also some important differences that should be kept in mind in examining individual program ratings of the scholarly quality of faculty. Already mentioned in this chapter is the inclusion, on the form sent to 90 percent of the sample members in the committee's survey, of the names and academic ranks of faculty and the numbers of doctoral graduates in the previous five years. Read the entire page →
From page 204... ... Figures 12.1-12.9 plot the mean ratings of scholarly quality of faculty in programs included in both surveys; sets of ratings are graphed for 26 programs in art history, 30 in classics, 82 in English, 49 in French, 36 in German, 26 in linguistics, 34 in music, 58 in philosophy, and 52 in Spanish. Since in the Roose-Andersen study programs were identified by institution and discipline (but not by department) Read the entire page →
From page 205... ... FIGURE 12.1 Mean rating of scholarly quality of faculty (measure 08) versus mean rating of faculty in the Roose-Andersen study -- 26 programs in art history. Read the entire page →
From page 206... ... FIGURE 12.2 Mean rating of scholarly quality of faculty (measure 08) versus mean rating of faculty in the Roose-Andersen study -- 30 programs in classics. Read the entire page →
From page 207... ... r = .91 FIGURE 12.3 Mean rating of scholarly quality of faculty (measure 08) versus mean rating of faculty in the Roose-Andersen study -- 82 programs in English language & literature. Read the entire page →
From page 208... ... FIGURE 12.4 Mean rating of scholarly quality of faculty (measure 08) versus mean rating of faculty in the Roose-Andersen study -- 49 programs in French language & 1 iterature. Read the entire page →
From page 209... ... FIGURE 12.5 Mean rating of scholarly quality of faculty (measure 08) versus mean rating of faculty in the Roose-Andersen study -- 36 programs in German language ~ literature. Read the entire page →
From page 210... ... FIGURE 12.6 Mean rating of scholarly quality of faculty (measure 08) versus mean rating of faculty in the Roose-Andersen study -- 26 programs in linguistics. Read the entire page →
From page 211... ... r = .94 FIGURE 12.7 Mean rating of scholarly quality of faculty (measure 08) versus mean rating of faculty in the Roose-Andersen study -- 34 programs in music. Read the entire page →
From page 212... ... FIGURE 12.8 Mean rating of scholarly quality of faculty (measure 08) versus mean rating of faculty in the Roose-Andersen study -- 58 programs in philosophy. Read the entire page →
From page 213... ... FIGURE 12.9 Mean rating of scholarly quality of faculty (measure 08) versus mean rating of faculty in the Roose-Andersen study -- 52 programs in Spanish language & literature. Read the entire page →
From page 214... ... Also, one must keep in mind that the correlations are based on the reputational ratings of only three-fourths of the programs evaluated in this assessment in these disciplines and do not take into account the emergence of many new programs that did not exist or were too small to be rated in the Roose-Andersen study. FUTURE STUDIES One of the most important objectives in undertaking this assessment was to test new measures not used extensively in past evaluations of graduate programs. Read the entire page →
From page 215... ... For example, what fraction of the program faculty were invited to present papers at national meetings? What fraction had been elected to prestigious organizations/groups in their field? Read the entire page →

From page 181...

... Also presented here are an analysis of the reliability (consistency) of the reputational survey ratings and an examination of some factors that might possibly have influenced the survey results.

12 Summary and Discussion Pages 181-216

12 Summary and Discussion
Pages 181-216