A Data-Based Assessment of Research-Doctorate Programs in the United States (with CD) (2011)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

1 Introduction This assessment of research doctorate programs conducted by the National Research Council (NRC) presents data that provide, for the first time in one place, basic information about many aspects of doctoral education in the United States. The data were compiled on a uniform basis1 and collected from the universities themselves. Data from the assessment will allow comparisons of similar doctoral programs, with the goal of informing efforts to improve current practices in doctoral education, and will help matriculating students pick the graduate programs best suited to their individual needs.2 The assessment, which covers doctoral programs in 62 fields3 at 221 institutions,4 offers accessible data about program characteristics that will be of interest to policy makers, researchers, university administrators, and faculty, as well as to students who are considering doctoral study. Furthermore, in an illustrative manner, the assessment analyzes and combines these data to create two ranges of rankings based on overall measures of program quality that were derived from faculty perceptions of program quality approached in two different ways. The National Research Council has a tradition of conducting careful assessments of doctoral education in the United States. The first NRC assessment, published in 1982,5 was a rich source of data for educational planners and policy makers, as well as a source of reputational ratings of perceived program quality obtained from raters who were acquainted with the programs in the discipline.6 The next NRC study, published in 1995, expanded the coverage of 1 The data were collected on a uniform basis in the sense that the universities were given careful definitions of the data elements and, if these definitions were adhered to, the data would be uniform. Data were checked and validated for internal consistency, but differences may still exist in university definitions of what a doctoral program is. 2 The study covers Ph.D. programs, some of which are offered in professional schools. It does not cover doctoral programs in professional fields. 3 In addition to the 59 fields with program rankings, a full set of data was collected for three fields: (1) languages, societies, and cultures, for which rankings could not be calculated because of the heterogeneity of subfields, which made the calculation of rankings for the field as a whole impractical; (2) computer engineering, which was initially identified as a field separate from electrical engineering and computer engineering, but had only 20 programs, fewer than the 25 required for the application of the ranking methodology; and (3) “Engineering Science and Materials (not elsewhere classified),” which also was not ranked when it turned out to have only 16 eligible programs. Data on size of faculty and enrollment were also collected for 14 emerging fields that did not have enough programs to qualify for inclusion in the ranking study. 4 The institutions include 212 universities and nine combinations of universities that offer joint programs. 5 The 1982 study and the 1995 study were both conducted under the auspices of the Conference Board of Associated Research Councils of which the NRC is a member. The current study members initially consulted closely with the Conference Board, but this study was conducted primarily by the NRC. 6 National Research Council, An Assessment of Research-Doctorate Programs in the United States. 5 vols.(Washington, D.C.: National Academy Press, 1982). 9 

10 A DATA-BASED ASSESSMENT OF RESEARCH-DOCTORATE PROGRAMS IN THE U.S. fields and types of data.7 The current study continues that tradition but uses a methodology that directly relates a measure of perceived reputation to quantified variables. Earlier studies relied on a program questionnaire and a faculty “rating” questionnaire. In addition to these, the current study fielded an institutional questionnaire, a student questionnaire in five fields, and an extensive faculty questionnaire. These numbers refer only to the program measures. The questionnaires appear in Appendix D. The expansion of field coverage over time is shown in Table 1-1. TABLE 1-1 Coverage of NRC Studies, 1982–2006 Study Year Number of Fields Covered Measures Collected 1982 16 (except for the humanities for which publication measures were 32 not collected) 1993 41 14 2006 59 ranked, 3 unranked but 20 (for rankings including the reputational measures) plus expanded with complete data, and 14 coverage of completion, student service in teaching and research, unranked emerging fields support services provided, student costs and financial support, with partial data interdisciplinarity, and postdoctoral study. Distinct from the earlier studies, the primary purpose of the current study, as outlined in the study’s statement of task, was the following: “(1) the collection of quantitative data through questionnaires administered to institutions, programs, faculty, and ’admitted to candidacy’ students (in selected fields); (2) collection of program data on publications, citations, and dissertation keywords;[8] and (3) the design and construction of program ratings using the collected data including quantitatively based estimates of program quality.” WHO WILL FIND THESE DATA USEFUL? These data will be useful to administrators, faculty, students considering doctoral study, and to those concerned with governance and policy related to doctoral education, as well as to the employers of Ph.D.’s outside of academia. In addition to comparisons of specific characteristics of interest, users will be able to understand the calculation of ranges of rankings of doctoral programs in each field through a spreadsheet downloadable from the National Academies Press Web site, http://www.nap.edu/rdp. Details of the illustrative rankings can be obtained by clicking on links provided in this spreadsheet. This study uses a methodology that permits users interested in rankings to understand the sources of those rankings. It also enables programs and individuals to benchmark themselves against peer or nearby programs using criteria that seem to them most appropriate. Examples are discussed in Chapter 5. DESIGN OF THE STUDY 7 National Research Council, Research Doctorate Programs in the United States: Continuity and Change (Washington, D.C.: National Academy Press, 1995). This report refers to data from this study as the 1993 data, but it refers to the study itself as the 1995 study. 8 As the study proceeded, the collection of dissertation keywords proved too complex, and this effort was dropped. 

INTRODUCTION 11 This study was designed with two objectives in mind: first, to collect comparable data across doctoral programs that would permit benchmarking for faculty and administrators, and second, to relate these data to measures of overall program quality and measures of particular aspects of doctoral programs. How the study was designed to achieve these objectives is described briefly in Chapter 2. Briefly, to characterize doctoral programs, data were collected from universities, their programs, and faculty in 62 fields, as well as from students in five fields.9 The data reported by the programs reflected the size, scope, and practices of each program, as well as financial aid and training practices that affect students. In addition, data were collected about time to degree and completion rates and whether the program tracked its students after completion. Because interdisciplinarity is an issue of increasing importance for doctoral programs, the program questionnaire gathered data to address this issue by counting faculty from outside the program who were engaged in supervising dissertations and by asking directly whether the program was considered to be interdisciplinary. The faculty questionnaire collected data on funding, work history, and publications, as well as on demographic characteristics. One section of the questionnaire asked the respondent to rate the relative importance of program, faculty, and demographic characteristics to program quality. It also asked whether the faculty member would be willing to respond to a questionnaire asking for ratings of programs. Nonrespondents were replaced in the rating study until approximately 50 raters were obtained for each sampled program in each discipline. See Appendix H for details by discipline. The student questionnaire, administered to advanced10 students in physics, chemical engineering, neuroscience, economics, and English, asked about student educational background and demographic characteristics, as well as research experiences in the program, scholarly productivity, career objectives, and satisfaction with a variety of aspects of the program. The size of the sample for each questionnaire and response rates are shown in Table 1-2. 9 The five fields were chemical engineering, physics, neuroscience, economics, and English. These fields were chosen because they are large, represented all but one of the broad fields, and were viewed by the committee as appropriate for a pilot study to understand whether such a questionnaire could provide useful information. 10 “Advanced” means students who have been admitted to candidacy. 

12 A DATA-BASED ASSESSMENT OF RESEARCH-DOCTORATE PROGRAMS IN THE U.S. TABLE 1-2 Study Questionnaires and Response Rates Questionnaire Total Responses Response Rate (%) Institution and program 221 institutions and combinations of institutions; 100 4,838 rated programs Faculty 87,515 88 Student (five fields) 11,888 73 a Rating 7,932 a Nonrespondents were replaced in the rating study until approximately 50 raters were obtained or each sampled program in each discipline. See Appendix H for details by discipline. The importance of the measured variables to perceived quality was ascertained in two ways: (1) from the relative importance of weights calculated from answers to the faculty questionnaire, and (2) from taking a sample of programs and faculty in each field and statistically deriving weights for each variable from the faculty’s response to a rating questionnaire.11 Both of these approaches reflect faculty values, which are discussed in Chapter 5. The method of obtaining rankings through two separate ways of calculating ranges of overall rankings is discussed briefly in Chapter 4 and in far more detail in the methodology guide. Chapter 3 also compares the current methodology to that of the 1995 study and explains some sources of noncomparability. Chapter 5 discusses the ways in which the study ascertains faculty values, which are key to understanding the rankings in the study. In Chapter 6 users learn how different groups may wish to approach and use these data. And Chapter 7 discusses some general patterns of the data and presents the principal characteristics of the programs in the study. It contrasts the methodology and results from the 1995 study with the current study and then presents a description of important findings about doctoral education in 2006–2007.12 It also presents selected findings from the faculty and student questionnaires. The concluding Chapter 8 provides the committee’s views of how the data from the study might be the subject of future work. WHAT THIS STUDY HAS REVEALED Doctoral education in the United States is a far-flung and varied enterprise. Every field has its highly ranked and renowned programs, which are typically characterized by a large size and the high research productivity of faculty. To be sure, there are also many smaller programs with high rates of completion and times to degree similar to highly ranked programs. However, doctoral education is in fact concentrated in relatively few institutions whose programs have many students and faculty. Of the 221 institutions and combinations of institutions that participated in the study, half of the Ph.D.’s were granted by 37 universities, or 17 percent of the total participating in the study. Because most of these programs are in public institutions, the health of 11 Each faculty member was asked to rate 15 programs, and these faculty ratings were then related to the variables for each sampled program. Data on numbers of program, raters, and raters per program for each field are shown in Appendix H. 12 Much more data are available than will be reported in the spreadsheet for this report. This report focuses on 20 program characteristics, but many more questions were asked. The committee plans to make the full database for all questionnaires except the rating questionnaire available to interested parties, unless particular items would violate individual confidentiality restrictions. Items whose answers would violate individual confidentiality restrictions will be masked for this dataset. Researchers who wish to use the full dataset with unmasked values must apply to the NRC and agree to comply with confidentiality restrictions in their published data. 

INTRODUCTION 13 these institutions and the nation’s ability to produce highly trained researchers and the next generation of professors are inextricably linked. As an illustration of the kinds of data-based rankings that can be produced, the committee explains and reports rankings based on two measures. One measure, the S ranking, is based on a survey of the importance to faculty in a given field of the general characteristics of doctoral programs. The other, the R ranking, is based on values reflected in ratings given to a sample of programs by a sample of faculty in a field. These latter measures are then related, through a regression, to the same measures used in the S ranking for the sampled programs, and the coefficients13 from that regression are used as weights to calculate these rankings for all programs in the field.14 The uncertainty in all rankings is measured in part by calculating the ranking 500 times, with a different half sample of raters taken each time, so that all rankings are presented as ranges of rankings. In addition to these overall rankings, the study provides ranges of program rankings, based on the weights obtained for subsets of the S measure in each field. These rankings address three specific dimensions of doctoral education: (1) research activity, (2) student support and outcomes, and (3) diversity of the educational environment. For all measures, attention is given to the presentation of statistical uncertainties in the reported results. The ranking methodology is based on faculty values, expressed either explicitly through the questionnaire results that are used to calculate S rankings or implicitly through the ratings of a sample of programs that are used to calculate the R rankings. The measures viewed as most important to the quality of a doctoral program are related primarily to faculty research productivity. According to faculty, publications, citations, grants, and awards matter more than other metrics. In some cases the ranges of R rankings and S rankings do not overlap. One interesting and important difference between the weights that result in the R and S rankings is that the one measure of program size—the average number of Ph.D.’s granted over the previous five years— often receives the largest weight in the R rankings but relatively small in the S rankings. Faculty appear to not assign as much importance to program size when assigning weights directly as when assigning them indirectly based on their rating of programs. Program size, while not likely to be a direct cause of higher program quality, may serve as a surrogate for other program features that do exert positive influences on quality. Another possible cause of these differences between the R and S measures is heterogeneity in the modes of scholarship in the field so that the statistical model does not fit very well.15 A table showing the correlation of the medians of the two measures for programs in each discipline appears in Appendix G. Meanwhile, measures other than the range of R rankings and S rankings may be important to others engaged in doctoral education, such as granting agencies and the students themselves, and as such should not be ignored. The committee 13 The coefficient expresses the relation between the rating and a particular characteristic when all the other characteristics are taken into account—that is, through a multivariate regression. The committee interprets them as weights that express the contribution of the particular characteristic to the variation in the rating. 14 The sample was designed to reflect the national population of faculty in each field with respect to faculty rank, program size, and geographic distribution. 15 Heterogeneity would create problems if two subfields in the same discipline had different modes of scholarship, so that the relationship between number of publications per faculty member and rating was different for each subfield. For example, if the rate of publication was much lower for programs in one subfield, highly rated programs dominated by this subfield would appear to be anomalous when combined with the subfield with a higher rate of publication. This problem could be solved by dividing the field and estimating the coefficients separately for the R ranking. 

14 A DATA-BASED ASSESSMENT OF RESEARCH-DOCTORATE PROGRAMS IN THE U.S. approached comparisons in three distinct areas through the dimensional measures. These measures summarize the program characteristics of research activity, student treatment and outcomes, and diversity of the academic environment. Student treatment and outcomes is related to research activity, because programs with a high level of research activity have the resources to treat students better. Programs with a high level of research activity have more faculty with research funding, and they typically exist in research universities with higher levels of available support. Many such programs have high rates of student funding in the first year and relatively high completion rates. They often do not, however, have shorter median times to degree. Based on data from the National Science Foundation Survey of Doctorate Recipients, the committee found that less than 50 percent of Ph.D.’s in each broad field has definite plans to seek an academic position or postdoctoral study in academia. Thus the findings of this study are important to employers of Ph.D.’s in the nonacademic sectors, as well as to academia. Furthermore, many Ph.D.’s are now employers in research-intensive businesses, and the characteristics of the programs from which they hire Ph.D.’s may be useful to them. Diversity among the faculty has improved impressively since the 1995 NRC study. Gender diversity has increased substantially in all fields, and the percentage of new Ph.D.’s who are female has risen from 38 percent to 45 percent overall, although the percentages are still low in the physical and mathematical sciences (30 percent) and engineering (20 percent). The racial diversity of Ph.D.’s has also grown markedly, at an average annual rate of 4.6 percent, whereas the number of nonminority Ph.D.’s has declined by 1.7 percent. Underrepresented minorities were 7.4 percent of Ph.D.’s overall in 1993 and were 13.5 percent in 2006, but their proportion remains low, especially in the more highly ranked programs in science and engineering.16 Overall, the number of Ph.D.’s granted annually to white males declined from 12,867 in 1993 to 7,297 in 2006.17 The ratio of faculty to students has changed since the 1995 NRC study. The ratio of faculty to Ph.D.’s graduated increased in most broad fields from 1993 to 2006, the years in which the data were collected. This finding may reflect a deeper faculty involvement in doctoral education, or it may be partially a result of definitional changes between the two studies.18 Finally and most importantly, this study is a tool that can be useful to administrators, faculty, students, and others with an interest in the characteristics of doctoral programs. Users can pick programs of interest and measures of interest and make customized comparisons. For students, these comparisons may be along the lines of funding and completion rates, or characteristics of programs near their homes. 16 “Underrepresented minorities” refers to African Americans, Hispanics, and American Indians. 17 Source: National Science Foundation. 18 In the 1995 study, programs were asked for the “names and ranks of all faculty members who participate significantly in education toward the research doctorate.” In the 2006 questionnaire, programs were given a far more specific definition of faculty, who were divided into three categories: core, new, and associated. The definitions are as follows: Core Faculty. Faculty who (1) have served as a chair or member of a program dissertation committee in the past five academic years (2001-2002 through 2005-2006), or (2) are serving as a member of the graduate admissions or curriculum committee. The faculty member must be currently (2006-2007) and formally designated as faculty in the program, and not be an outside reader who reads the dissertation but does not contribute substantially to its development. Include emeritus faculty only if the faculty member has, within the past three years, either chaired a dissertation committee or been the primary instructor for a regular Ph.D. course. New Faculty. Faculty who are not core and (1) do not meet the criteria for core faculty, but who have been hired in tenured or tenure-track positions within the past three academic years (2003-2004 through 2005-2006) and (2) are currently employed at your university and are expected to become involved in doctoral education in your program. Associated Faculty. Faculty who are neither core nor new, but (1) have chaired or served on program dissertation committees in the past five years (2001-2002 through 2005-2006), and (2) have a current (2006-2007) appointment at your institution, but who are not designated faculty in the program. They should not be outside readers, or faculty currently employed at other universities, unless they are on leave from the faculty at your institution. Include emeritus faculty only if the faculty member has, within the past three years, either chaired a dissertation committee or been the primary instructor for a regular Ph.D. course. 

INTRODUCTION 15 Administrators may find comparisons with peer programs nationwide or regionally. With that in mind, six months after the release of this report and the accompanying data, the National Research Council will hold a workshop at which researchers and others who have used the data will report on the uses they have made of them. The proceedings of this workshop will be published as a workshop report and will expand on the descriptive summary discussion provided in this report. Whatever their interest, all users will find that they have access to information about doctoral programs that was not available in the past.