Origins of the Study
As part of the Minority Health Initiative launched by the National Institutes of Health (NIH) in 1992, the Assessment of NIH Minority Research Training Programs was initiated by the Office of Research on Minority Health (ORMH) in the Office of the Director at NIH.1 The goal of this study was to answer a fundamental question: Do the NIH minority research training programs work? The study was implemented in three phases with ORMH conducting phases 1 and 2.
Phase 1 was completed in 1993. It presented an overview of NIH extramural research training programs and summarized available information and trend data for each of the major NIH minority research training programs.2 Phase 1 findings also documented an overall pattern of minority underrepresentation3 in the biological, behavioral, and clinical sciences (hereafter referred to as “biomedical” sciences). Phase 2 was completed in 1997. It assessed the feasibility of a trans-NIH assessment of minority research training programs and provided a potential scope for that endeavor. Research questions and potential data sources useful to that assessment, which would be phase 3 of the study were identified.4
In 2001, the National Center on Minority Health and Health Disparities (NCMHD, formerly ORMH) contracted with the National Academies to undertake the phase 3 assessment as an independent study that would draw on the findings of ORMH’s earlier work. NCMHD chose the National Academies based on its independence, its ability to collect and integrate quantitative and qualitative data from NIH institutes and
centers, and its ability to convene national experts who could analyze and assess these data in an objective manner.
In order to assess and analyze NIH minority trainee educational and career outcomes, the study committee was charged with addressing the following questions to the extent that they could be addressed using available data from NIH:
Do the NIH minority research training programs work?
Which minority programs and which features of minority programs have been most successful in helping individual students and faculty members move a step forward toward productive careers as research scientists?
Which minority programs have been least successful and why?
What additional factors contribute to minority trainee success, including characteristics of individual participants and the academic institutions at which they received NIH research training support and/or obtained their terminal degree?
How can a system be set up that would better address assessment questions in the future?
In addition, the study committee was charged with developing policy recommendations for an improved coordinated tracking information system that would do the following:
Provide NIH administrators a means for obtaining improved annual feedback on minority research training programs;
Assist the development of future goals;
Assist the development of performance measures; and
Assist the improvement of program effectiveness.
Assessing Program Outcomes
To answer the question, “Do the NIH minority research training programs work?” the committee developed three metrics for assessing program success which it applied in the course of its work. First, the committee undertook a thorough analysis of historic NIH program announcements for these programs in order to identify their stated goals. This analysis established that the foremost goal of NIH minority research training programs is, and always has been, to increase the number of Ph.D.-level minority biomedical researchers. However, success in reaching this goal was not quantified among any of the program announcements. Second, the committee considered the work of phases 1 and 2 of this study which recommended examining whether or not trainees had advanced to the “next step” in the science educational and/or career trajectory. Third, the committee also considered the value that participation in the program provides
to the trainee. All but one member of the committee also believes that, regardless of whether each trainee advances to the next step in his or her education or becomes a Ph.D.-level researcher, the programs provide important and valuable training experiences for all participants that should be considered in assessing whether a program works.
Given disparities in educational opportunities available to trainees prior to enrollment in any of the NIH programs, it would be inappropriate to expect or demand that minority trainees, as a whole, attain the same average rates of professional success as nonminority trainees. Indeed, the training programs exist because of the need to overcome this gap. An additional and appropriate standard for evaluating minority programs, therefore, is the “value added” by the program to all its participants. This introduces a set of measurement problems as outlined below, but it is a critical foundation of the committee’s analysis and recommendations. Thus, the following principles bear upon any discussion of minority research training program success:
More than one family generation is needed to establish a research training pipeline that is both attractive to minorities and successful at producing large numbers of Ph.D.-level scientists.
Building capacity and sustaining minority interest in science require the visible promotion of role models. Such persons may include science teachers, professors, medical doctors, entrepreneurs, and others, who open a window to science careers and opportunities to which young minds might not otherwise have been exposed.
The research training pipeline is necessarily leaky. Those who exit the pipeline early to become part of the scientific workforce are not program failures.
The research pipeline is not always a straight line. Some will exit the pipeline only to return some years later.
Programs designed for those who are in early career stages should endorse a broad definition of success. Programs for trainees at later career stages may adopt a more highly prescribed definition of success.
The study committee was charged with addressing its study questions to the extent that they could be using available data from NIH supplemented by interviews with minority trainees and program administrators. Simply put, the committee was not able to obtain all of the data it wished. While the committee met its charge to the extent feasible, it could not answer all of the research questions in as direct and complete a manner as it would have liked, and it now advocates for a future study. If the advice provided in this report leads to corresponding action, it will improve the programs in the short run and facilitate a more comprehensive study in the future. Indeed, extensive data collection efforts, ongoing deliberations, and analyses allowed the committee to identify critical data elements that should be collected by NIH on a systematic basis, in order to make future assessments of all NIH research training programs feasible.
The committee conducted a census of extramural NIH minority research training programs that were sponsored by the institutes and centers (ICs) at the time the study began in 2001. There were 79 such programs—too many for a feasible assessment, given the project budget and time constraints. Thus, the committee developed rational inclusion and exclusion criteria in order to distill the training programs it could most effectively assess. The committee decided that the time frame for the study would extend from 1970 to 1999 and the study would include the following career stages: undergraduate, graduate, postdoctoral, and junior faculty. Two trainee comparison groups were also identified—minority and nonminority trainees participating in programs that are not targeted specifically for minorities.
The study evaluates 47 of the original 79 minority programs, these 47 being reclassified into 13 program categories that take into account career stages served by the programs and the letter-number designation associated with each program (e.g., F31, T32, K01, etc.). For a summary of the 13 program categories, see Table 2-2.
Another difficulty faced by the committee at the outset of the study was the prohibition against accessing or viewing individual trainee race and gender data in NIH’s data sets. Given the need to distinguish minority from nonminority trainees for purposes of carrying out this study, the committee was required to rely on an intermediary NIH-approved contractor that was allowed access to individual trainee race or ethnicity and gender data. Since the National Academies had no direct contractual relationship with the NIH-approved contractor, it had little leverage in terms of the deliverables produced.
The committee is cognizant of the sensitivity of race and gender data and the degree to which the NIH Office of the Director strives to protect the privacy of its trainees and grantees, but in this case it made very difficult the very task the committee was contracted by NIH to conduct. Thus, NIH may wish to reconsider its interpretation of how the Privacy Act applies to the degree of access an outside evaluator has to individual trainee race or ethnicity and gender data when that evaluator has been contracted by NIH to conduct an assessment of minority research training programs. It may also want to revisit the value of having more than one contractor approved for access to individual trainee data.
The committee designed a study approach that called for extensive mining of existing NIH electronic trainee data sets, followed by structured interviews with former NIH trainees and semistructured interviews with program administrators who administer these programs both at NIH and at awardee colleges and universities. The NIH data contractor conducted 732 computer-assisted telephone interviews (CATIs) using a
random sample of trainees who were participants in one of these programs prior to 2000. The trainee interviews focused on the following issues:
Trainee educational and career expectations;
Best or worst program features;
Relationship with head of the laboratory or research group (i.e., principal investigator, or PI);
Relationship with trainee’s mentor; and
Relationships with other laboratory or research group members.
A few open-ended response items were also included in the survey. These provided trainees with an opportunity to share what they believed to be the strengths and/or weaknesses of the programs and to suggest ways in which NIH could improve its programs.
In the absence of NIH-wide electronic trainee tracking data, the NIH data contractor achieved a very low response rate from its efforts to locate and interview trainees. This was the case despite its use of two commercial and proprietary databases that together maintain credit card-related contact information for millions of Americans and the query of the U.S. Postal Service address-forwarding database. The committee was disappointed but not entirely surprised by the low response rate. As a result of low location and response values, there is a high likelihood of bias among the survey results. Some evidence suggests that the trainees interviewed for our survey were more likely to be among the more “successful” program participants. For example, among those who participated in the Bridges to the Baccalaureate program, survey respondents were more likely to have transferred to a four-year institution and completed a bachelor’s degree than program participants in general.5
The committee was similarly skeptical about the large numbers of respondents who had at least one family member with a bachelor’s or graduate degree. Thus, the committee believes that data from these interviews may not reflect the responses that would have been obtained had the respondents been more representative of the larger universe of program participants. Nevertheless, the data are instructive in a general way and are described qualitatively in the report and summarized briefly below. Respondent data are reported using a variety of nonspecific phrases such as: “nearly all reported,” “a majority of respondents said,” “a minority of respondents said,” “more likely,” and “less likely.” Such phrases should not be equated with statistical significance.
Minority Training Programs: What Is Working?
The committee concludes that underrepresented minorities are entering the biomedical workforce as a direct result of the NIH minority research training programs.
The administrators of these programs mentioned that there are many more applicants to the undergraduate programs than there are available positions. Thus, recruitment appears to be highly effective at this level. At the undergraduate trainee level, attrition from the programs is minimal, due in part to an effective system of oversight and monitoring of trainees’ progress.
Among trainee respondents at all career stages, there is profound appreciation for what these programs offer and recognition of the prestige associated with being an NIH research trainee. The “best feature” most often cited by trainee respondents across all career stages is the research experience itself. For undergraduate trainees, the acquisition of laboratory skills was key. For graduate trainees, laboratory experience was important but so were graduate-level coursework, research seminars and workshops, learning how to think critically, learning to make cogent research presentations, and learning to teach science to undergraduates. Among postdoctoral and junior faculty trainees, the opportunity to choose a subspecialty and develop research independence was the most valuable aspect of the training programs.
Among undergraduate trainees, mentoring support was cited as the second most valuable feature of the training programs. Mentoring was most often provided in four key areas:
Improving the trainee’s research skills,
Providing motivation and personal growth,
Providing career guidance, and
Promoting the trainee for scholarships and other development opportunities.
Mentoring was also very important to graduate, postdoctoral, and junior faculty trainees who reported many positive interactions and support from their mentors.
Funding support from the training programs was greatly appreciated by undergraduate trainees. Such support came in the form of stipends, summer research positions, and conference travel support. Funding was, for graduate trainees, frequently cited as a best feature. At the graduate level, a funding arrangement exists whereby NIH covers the cost of research training, including stipend and tuition support, research supplies, and benefits. In return, NIH requires that trainees refrain from taking outside jobs in order to devote 100 percent effort to the training experience. For postdoctoral and junior faculty trainees, funding was characterized as “critical and necessary.” The “protected time” that funding provided trainees at this level allowed them to achieve research independence, which is the foremost goal of these programs.
Other positive program elements that trainees mentioned include the foundation of scientific knowledge that the program provided to undergraduate trainees and the opportunities to network and collaborate with other scientists, which was mentioned by trainees at all levels, but especially graduate trainees. Undergraduate trainees underscored the ability of the programs to help them decide whether to attend graduate school or medical school. Graduate and postdoctoral trainees cited frequently the tremendous value in learning how to prepare a competitive grant proposal. According to junior faculty trainees, the K01 award allowed them to progress to the next step in their careers, namely to obtain an R01 research grant.
Minority Training Programs: What Is Not Working?
The committee concludes that NIH can do a better job in training a large cadre of doctoral-level minority biomedical researchers.
At the postdoctoral and junior faculty levels, there appears to be a sharp drop-off among minority trainees. An indicator of this is the gender shift from predominantly female at the undergraduate and graduate career stages to predominantly male at the postdoctoral and junior faculty career stages (see Appendix E). Where do the minority female trainees go? This question warrants further study by NIH.
Although trainees across career stages were extremely grateful for training program funding support, they uniformly stated that the levels of funding are not sufficient and need to be increased. Undergraduates who are already greatly challenged by a demanding research program in addition to a full load of coursework must often take on additional outside work in order to make ends meet. Program administrators call this situation a “recipe for disaster,” and it constitutes a barrier against participation in these programs for lower-income minority students.
Graduate trainees complain similarly. They are contractually prohibited from obtaining outside jobs, yet the stipend support is barely above the poverty line. In the context of the uneven health benefits afforded by these programs, this too is a “catch-22” situation that trainees reported with frustration.
Postdoctoral and junior faculty trainees are similarly disheartened by the low stipends afforded by the training programs. This is especially true when trainees have dependents and/or live in major metropolitan areas where the cost of living vastly exceeds what the stipend offers. All trainee respondents were clear and forceful in stating that trainee stipends have to be more in line with market trends; they need to be increased in order to sustain and build student interest in research careers. This sentiment was echoed by numerous program administrators, one of whom stated that the stiffest competition faced in attracting African-American trainees to a research career comes from the salary opportunities provided by advanced health professional programs.
Although highly cited as a positive element of the training programs, mentoring was also criticized as needing significant improvement. Too many trainees reported negative mentoring experiences in the lab. Some minority undergraduate trainees were given mundane administrative tasks to perform in lieu of experiments; others experienced “benign neglect” by their mentors or, at best, a lack of encouragement. One-half of the minority T32 postdoctoral trainees reported having no mentor at all, a trend that was not replicated with nonminority (T32) postdoctoral trainees. This is a red flag to which NIH must pay attention, especially in the context of the scarce numbers of minority trainees at this relatively advanced career stage.
Training in the biomedical sciences historically assumes that if one is trained, one will therefore be a good trainer (mentor), but this is not necessarily so. Mentoring is a skill, one for which academic researchers rarely receive any formal training. Thus, NIH would be wise to assess a variety of research training methods to see which approaches work best in different situations. The old adage, “Do as I did” does not operationalize well in the context of today’s diverse trainee populations. Training in the absence of optimization research produces the kind of the homogeneity seen among this study’s
postgraduate trainee respondents. Those who are just like their mentors are promoted; those who are different from their mentors are not.
Program administrators emphasized that in addition to the lack of mentor training, mentors receive little credit, encouragement, or support for time taken to mentor trainees. Grants do not provide funds that cover mentoring activities and faculty time. Academic departments do not view mentoring as a legitimate activity that counts toward tenure. Yet, mentoring is absolutely essential to the continued growth and sustenance of our biomedical workforce. NIH should examine these issues and consider changing the value it places on this essential activity in some concrete way.
Minority respondents to our survey provided additional clues that may bear upon their low numbers at higher career stages. Based on the survey data, which the committee believes are biased toward the most successful NIH trainees, minorities publish fewer papers than do nonminority trainees. They have greater difficulty in securing employment after receipt of the doctoral degree. They report less social integration in their laboratories, and this was experienced more by minority trainees at institutions using nonminority training mechanisms. Finally, a large fraction of minority trainees believe that their minority status in some way affected their training experience. Given that one-half of the minority postdoctoral survey respondents reported having no mentor at all, one wonders what factors are at play in these training environments that affect minority trainee outcomes so profoundly.
By the end of 2005, the NIH director should articulate a set of clear and measurable training goals and objectives specific to minority training. The director should take into account the mission of NIH and the integral role of research training in attaining both societal goals (e.g., health and well-being, the ability to support oneself and one’s family, community development) and research goals. Such a policy should be responsive to society’s workforce needs in their broadest sense, with an understanding that contributions to society derive from all parts of the career stage pipeline.
NIH should commit to the continued funding of minority-targeted research training programs. Although the committee cannot substantiate this recommendation in quantitative terms for reasons described throughout this report, it does so in qualitative terms, using survey data that were collected from trainees and program administrators who are the programs’ primary informants. The following reasons underlie this recommendation:
These programs have added many minorities to our science workforce.
The elimination of these programs would likely diminish the number of new minority scientists entering the scientific workforce.
The trainees interviewed indicate overwhelmingly that these programs benefited them. These programs provided research experiences, financial support, and mentoring that were critical to their career success.
Mentoring is a critical part of the career development of all scientist, and is particularly important for minority trainees. Trainee survey data suggest that the diversity of mentors is greater in the minority-targeted programs than in the nontargeted-programs. Atkinson et al.6 found that, when rating mentoring relationships, both mentors and mentees rated their relationships more positively when they were matched for race or ethnicity.
The committee recognizes two distinct and valid approaches to the development of minority research trainees. The training policy of the NIH institutes and centers (ICs) in conducting these programs should emphasize the development of trainees who have already demonstrated promise in the sciences, so that they can overcome the barriers to becoming productive investigators. Two examples of minority training programs that emphasize talent “harvesting” include the National Institute of Mental Health Training and Education (NIMH) Career Opportunities in Research (COR) and the Minority Access to Research Careers (MARC) Undergraduate Student Training in Academic Research (U*STAR) programs. Harvesting talent in this context means supporting trainees who probably would have, for a variety of reasons, “made it” regardless of support from the targeted programs. The NIH training policy should also emphasize the development of other trainees—those without demonstrated science promise—in order to add to the pipeline of trainees interested in pursuing science careers. An example of a minority training program that emphasizes “growing” talent is the Bridges to the Baccalaureate program. Growing talent in this context means the promotion of science and science careers for individuals and communities that may not otherwise have entered science.
NIH should more vigorously monitor the use of racial or ethnic eligibility criteria for these programs. Survey data from trainees and program administrators indicate that non-underrepresented minorities are participating in minority-targeted training programs.
NIH should examine gender differences among its trainee participants. For example, the minority trainee population at the undergraduate level is mostly female, but this proportion declines at each successive career stage, showing that there is substantially more attrition among women who could have become investigators than among men. This trend is particularly striking at the graduate-to-postdoctoral transition where men, conversely, outnumber women. This trend among females is independent of race.
Given comments offered by trainees and program administrators, the committee recommends that NIH conduct a review to ensure that the research infrastructure (e.g., laboratory space, laboratory equipment, active faculty research programs) available to minority trainees at the institution level is adequate and, if not, seek ways to further address this programmatically.
The director of each institute should designate a single individual as minority research training programs coordinator for that institute by the third quarter of FY 2005. Some institutes have centralized training coordinators; others do not. This recommendation would provide consistency and make coordinated efforts more feasible.
The NIH training director should convene a meeting of all minority training coordinators on at least a quarterly basis, beginning with the third quarter of 2005. The goal of these meetings would be to coordinate the administration of NIH minority training programs and the collection of relevant program data. Currently, the administration of these programs is fragmented and, as a consequence, external evaluation is difficult. Given the importance of the NIH training programs to the continuation of U.S. leadership in biomedical research, coordinated efforts are needed to develop, manage, and rigorously evaluate research training programs. The collective management of minority training programs, although not intended to supplant IC independence and expertise, requires ongoing communication and cooperation across disciplinary and institutional lines. Agendas for these meetings are expected to change over time as the collaboration improves communication and advances meaningful planning. The meetings should at a minimum address the following issues:
Clarification of NIH training policies regarding trainee recruitment and documentation of program activities and results,
Discussion of the range of IC training program characteristics,
Sharing of trainee recruitment strategies,
Identification of effective elements of IC training programs,
Review of IC evaluation results, and
Development of long-term objectives for addressing workforce needs and for increasing the participation of underrepresented minorities in science.
The committee of minority training program coordinators should establish appropriate guidelines and measures for evaluating NIH minority research training programs. Training program administrators should participate in an ongoing and rigorous evaluation process. By defining program outcomes and monitoring their achievement, the ICs can better manage their programs. NIH should make a commitment to make available all of the data needed for internal or external evaluations of its training programs.
Further study of the relative effectiveness of minority-targeted versus nontargeted programs should be carried out by NIH institutes and centers under coordination from the Office of the Director. The reasons for this recommendation include the following:
The ICs should establish outcome measures for each training mechanism in a coordinated fashion. To do this, the ICs should identify and document the range of trainee outcomes that result from participation in these programs. Then, the range of outcomes should be codified as either contributing or not contributing to the consensus definition of program success. The committee is cognizant that this recommendation reflects an interactive process.
Continued integrated study of these programs can identify the best features of the programs and the best practices among recipient colleges and universities.
The director of NIH training should administer the funds for evaluation, data collection, and marketing by FY 2006. Centralized training activities should include a centralized and robust evaluation and planning activity. This approach will empower the director of NIH training to be able to coordinate accountability mandates (i.e., PART, Government Performance and Results Act) with organizational policies and procedures.
The general issues reviewed in this report should be revisited periodically at the NIH level, with the next report submitted by 2009. The Office of the Director at NIH should take the lead on this. The numerous, weighty, and very public issues regarding affirmative action that are raised by targeted research training programs require continuing attention by a consortium of the National Center for Minority Health and Health Disparities, the Office of Extramural Programs, the institutes and centers that fund such training programs, and the NIH Office of the Director. NIH should conduct an independent public review and accounting that will help ensure that the programs remain focused and effective. Doing so will inform both the affected groups and the general public of the success of the programs and ensure that funding is being used effectively, thus yielding a positive return on the nation’s investment. The committee believes that five years is a good interval for external review of the program(s), although experience may show that more frequent review would be useful. The committee further believes that the breadth and depth of the issues, compounded by the present fragmentation of many components of NIH, require that the Office of the NIH Director take the lead role.
NIH should develop a relational database that collects a minimum data set (MDS) for all persons who receive funding as trainees, fellows, research assistants, or postdoctorates, including those programs targeted to underrepresented minorities.
The database should be maintained by the Office of the Director of Extramural Programs, headed by the deputy director and NIH research and training officer. The Office of the Director of Extramural Programs should have the overall responsibility for coordination of the database and its constituent parts.
The MDS should be a service to all institutes and contain variables that enable rigorous evaluation and assessment of training programs; institutes may add variables at their discretion.
The MDS should collect data for all trainees, including all those funded through the training mechanisms covered here, as well as for research assistants funded through R and K awards.
The Office of the Director of Extramural Training, in coordination with institute representatives, should develop a data entry system accessible from multiple sources, including external data entry points such as grant-specific progress reports. In addition, the Office of the Director of Extramural Training should develop a user-friendly data entry form for the MDS that is web accessible. The database and data coordination in the deputy director’s office will emulate that of a coordinated data center. Considerations of personal privacy and confidentiality must be high on the list of necessary attributes.
The Office of the Deputy Director of Extramural Training should identify data elements that help in tracking persons who received training funds—both directly and indirectly. These tracking data should be obtained at the time of initial NIH funding and should be updated periodically.
Development of the MDS, database, and data entry system should begin immediately and be completed no later that FY 2008.