Perspectives of NIH Program Administrators
The successes of National Institutes of Health (NIH) training programs are influenced, in part, by the attitudes, expectations, and support of the program administrators. This chapter offers a perspective on the views and practices of 22 program administrators across the NIH institutes and centers (PAICs), in response to their training responsibilities, including training of underrepresented minorities. The interviews draw attention to the absence of inter-IC (institutes and centers) coordination of minority research training programs across the NIH campus as a whole and underscore the widely perceived need for an NIH-wide trainee outcomes tracking system. In addition, the interviews elicited numerous strategies for improving the success of minority training programs and reveal a prevalence of mixed attitudes among PAICs, regarding minority-targeted programs, in general.
The committee developed two interview instruments that were designed to clarify the perspectives of PAICs and identify their views on the strengths, barriers, and strategies associated with the management of training programs, in general, and minority training programs, in particular. One instrument addressed programs that are awarded to individuals (e.g., F31, R03, K01), and a second addressed programs awarded to institutions, such as the T-series training grants (a sample instrument is located in Appendix D). Three pilot interviews were conducted in the course of developing the interview protocol. The pilot interviews were not included in the analysis of PAIC input because of varying protocols, the purposeful selection of particularly supportive participants, and the presence of observers.
The interviews were designed to be ethnographic (e.g., open-ended and exploratory), but unfortunately, in many cases, interviewers allowed the discussions to stray considerably from the interview questions in ways that the committee had not anticipated. As a result, the interview contents are highly variable with regard to topic and relevance to the questions originally identified by the committee.
The format of the interviews was further complicated by the lack of consistent structure in the management of training programs across the NIH ICs. Some have a centralized training officer who coordinates all training (minority targeted and nontargeted); others have a separate minority training officer; still others assign training to individual health scientist administrators depending on the specific research area being funded. In some cases, when training program management was ostensibly centralized, the contractor was told to speak with someone else regarding minority programs. Thus, it appears there is no uniform utilization of minority training programs or training program management policies across the ICs. The inclusion of minority-targeted programs, minority status of participants, career stages served, or whether the program is awarded to an individual or an institution varies considerably across the ICs. For this reason, it was not possible to analyze the response data strictly in terms of career stage. This degree of program management decentralization and heterogeneity made it challenging at times to identify the best respondents for each of the programs. Because some PAIC respondents were responsible for several training programs, it was challenging for the contractor to manage the interviews and ensure that all relevant programs were discussed. Unfortunately, in actual operations, some of the interviews focused on one or more programs that were not even included in the scope of the study.
A total of 22 PAIC interviews were conducted and analyzed for this study. The interviews were administered to a group of rationally selected respondents who administer programs assessed by the study. For reasons of confidentiality, respondent identities and institute affiliations were concealed to anyone other than the contractors who conducted the interviews. The contractor began the interview by introducing the study and explaining how respondents’ comments would be integrated into the study.
The qualitative data analytic program Nu*dist version 6.0 was used to classify response data. Comments were grouped into 347 different clusters of related statements and then subjected to additional analyses by the study committee. The data were organized into four “layers” (or node levels) based on career stage served by the program, whether the program was awarded to individuals or institutions, whether the program was targeted or nontargeted, and the specific topics of discussion.
Common themes are identified and an example quote is provided for each. Rather than duplicating quotations, the prevalence of the response is indicated whenever possible. Unique responses are identified as such. Every effort was taken to ensure accuracy among the quotes reflected in this chapter. Extraneous and irrelevant comments were replaced by ellipses. Words were added to a quotation in cases where missing words could be reasonably discerned from the interview context; added words are included in brackets.
The data are grouped into four major content areas: (1) recruitment strategies; (2) funding application process; (3) definition of program goals; and (4) reliable methods of outcome monitoring.
In general, program administrators expressed one of two views about the adequacy of trainee recruitment to the programs: a modest majority was content with current trainee recruitment strategies, whereas a substantial minority was less so. With regard to the former, respondents said that underrepresented minorities who wish to pursue a career in biomedical research are welcome to locate specific program information on IC web sites or from printed matter such as program brochures or trade journal announcements. Respondents in this group emphasized that it is up to the minority trainee to identify the appropriate training program for him- or herself: “I think the programs are [fine], if any one is at all interested in doing research they know to look at NIH [web sites], and when they get to the NIH, the programs are, I think … easy to find.” Another respondent said, “Because the people come to the web site, they’ll see the ‘diversity page’ that they can click on. They’ll see there are opportunities.” Other respondents in this group mentioned that NIH staff regularly attends professional conferences and other meetings where they give presentations about IC training opportunities. Some respondents added that sometimes NIH staff make contact with key individuals in the community just to let them know about a pending announcement. A few respondents in this group noted that there may not always be an adequate number of eligible minorities in certain geographic regions of the country. “we encourage … investigators to … enroll underrepresented minorities, … we don’t have quotas but we have a firm expectation. There are some realities when you get out of [major urban areas] there aren’t very many minorities enrolled in the school in [less populated areas], …” and thus their expectation is that NIH must accept their limited ability to recruit underrepresented minorities. An unexpected comment came at the end of one interview: “We are not in a position here, as a funding institution, to really have a hands-on meaningful effect on young individuals, convincing them to come into science and stay in science, … that is a reality, … we just aren’t close enough to the individuals.”
The second group felt that current trainee recruitment strategies are largely insufficient and that ICs should conduct a more proactive and targeted outreach to prospective minority trainees. Such targeted outreach may include dissemination of minority trainee success stories, training opportunity advertisements, and program announcements infused into popular media that serve minority audiences. This group of respondent also placed great value in outreach targeted to minority students at the primary and secondary school levels. They noted that in low-income areas, where access to computers is limited, the web and other literature may not be available to potential applicants. In addition, the information provided does not always address the issues that
concern underrepresented minorities in the first place. For example, many minorities want to know how to survive economically and support their families while participating in NIH-supported programs.
Several respondents with this viewpoint suggested that NIH policy and procedures become more flexible in announcing and marketing their research programs. One respondent said, “It would be nice to have the flexibility of being able to structure grants a little bit more … in fact, we can’t pay recruitment costs. Recruitment is a non-allowable cost at NIH, so some kind of [support for] recruitment for our local programs would be nice.”
A small number of these respondents suggested that a broader and more creative approach to recruiting minorities may be needed. “We need people that are actually anthropologists, psychologists … not just geneticists to look at the bigger picture. To … question the assumptions. One of the basic assumptions … [in] most of our programs is that if you get kids involved in a lab, they’re going to get excited and good things will happen. It’s going to turn them on. They’re going to become better students. It’s a gratuitous assumption.” They also suggested studying the process: “[study] the career tracks of successful minority students who have turned into postdocs and funded investigators, [those] who have impacted a given area of science…. That could teach us what the critical components are.”
Another positive approach suggested by multiple respondents focused on broader public outreach. “I think we need to continue to not only just present [approaches], but make sure that we write information that can be published in different venues, … Everybody I know reads Ebony magazine. Why can’t we do an article looking at minority investigators, … on what minority investigators are doing and then the opportunity for research …” This approach could be implemented in several public venues such as popular magazines and public service announcements in Native American, Hispanic, and African-American popular media publications.
One of the most aggressive approaches to minority recruitment included a plan to follow promising minority and economically disadvantaged students from the undergraduate career through the professional career. “… We carry some … postdocs all the way through career development. The program is designed the same way so individuals have support all the way through the career development, but the key thing is for them to know about this.” Another respondent indicated, “We want to follow them. We do have all the mechanisms here to … go with them up to they get an R01. The comprehensive follow-through approach is not yet implemented …” In another perspective on outreach, it has been noted that the Bridges programs seem to draw more Native Americans than does any other program “because there aren’t any other programs that reach out to community colleges specifically.”
Recruiting programs intended for minority junior faculty appeared to have some unique issues. The individual faculty applicants seeking targeted support for mentoring
need to have a well-developed research plan. “We do like to see that in the first year that the research plan is fairly detailed and specifically laid out.” In addition to the research plan, the applicant needs to have a career plan laid out as well. However, it was noted by several respondents that the minority faculty are often located in schools that emphasize teaching and that teaching loads sometimes reach four courses per semester. The minority-serving institutions are focused on teaching to the extent that it may be difficult for supported minority faculty to do the required research or even build the necessary research record to apply for career support. One particularly clear expression of this point was raised at the end of an interview. “I’ve talked to a few minority [faculty] applicants who want to do research but there’s so much heavy emphasis on teaching … they can’t get the time to devote to research. And so I don’t know if there are any more creative ways to do that. Not everyone can take advantage of minority K01 awards, which require a certain amount of hours in research time.” In order to obtain funding, the faculty has to have an active research program. This seems to be a “catch-22.” One individual proposed that minority junior faculty could be encouraged to engage in research if given an opportunity to do so, perhaps using a mechanism that would essentially “buy” one year of teaching credit.
Some of the PAIC interview respondents provided particularly insightful thoughts on common barriers to enrolling targeted minorities into graduate school: “… Some of them … don’t have very good undergraduate records, … sometimes they have very bad grades. Application reviewers and the special emphasis [program administrators need] to realize that some of these students have different realities than other students.” In addition they have added responsibilities. “A lot of them have to work a full-time job while they’re students. A lot of them … have two to three children … those are the things that explain why [they] get Cs or … failed this [course] but I passed [it] later with an A.” It might simply be cultural differences between the applicant and the reviewers. “Sometimes you don’t understand where a person’s coming from…. You’re not familiar with people from that culture. And its one of the things we’re trying to bridge in a rewrite of the program announcements.”
One respondent was clear that recruitment had to happen much earlier than in college: “We try to go out to elementary schools, if possible. I’ve not done that in a number of years, but we have participated in science fairs. And, in terms of educating the community, because in elementary school, while they are still very young, it’s really at this level oftentimes that it’s piqued their interest.” A few respondents discussed science fairs and other activities that engaged students at all levels of training.
There may be some confusion among the various training management staff about targeted mechanisms. For example, some program administrators believed that their nontargeted programs are exclusive to nonminority applicants: “In fact, minorities can’t apply for any of our grants.”
As a separate issue, several respondents suggested that minority recruitment requirements be tied more closely to funding of training grants “when someone comes up
on a competing renewal grant and they don’t show any progress. You can hold up the award on their grant until they submit a revised application to make sure that they are approaching the situation more effectively.”
Funding Application Process
A few training officers felt that minority applicants must meet the same requirements that nonminority applicants do and that essentially nothing can be done, from an NIH perspective, to develop more underrepresented minority biomedical researchers. A second perspective offered by a large number of program administrators was that many more of the applicants from underrepresented communities could succeed in obtaining support for their training if they were given some technical assistance in grant proposal preparation. A third perspective is that NIH itself should make the necessary adjustments to reduce barriers to science participation among members of underrepresented groups.
Several respondents suggested ways in which NIH staff could enhance the likelihood that minority applicants and their institutions are awarded training and research support. One individual explained, “[When the applicant] asks about the review, they’ll have their summary statement, … but also any other thing that didn’t quite make the summary statement we can provide … and, in general, just input about the overall sense [of the review committee at] the time of review.” Another respondent has a specific routine for addressing promising minority grantees who didn’t get funded the first time: “I normally … send my letter with the summary statement. I ask the applicant to share the summary statement with their mentor and contact me if they have any questions. Normally they will contact me because sometimes they don’t know what [the summary statement] really means, ….” Another NIH staffer actually goes beyond this simple approach and takes the time to increase the likelihood that a second effort is successful. “I normally … offer them [the mentor and mentee] a conference call, if they want to amend the application and resubmit it…. I read the summary statement … and give my recommendations [to them]. Most of the … recommendations … are constructive criticisms.” This appears to be a successful approach. The same respondent reported that he works with “the students that … have 10 points or 20 points out of the pay line, … I had about six amended applications. Five of them are likely to get funded” after they are resubmitted.
A different way in which NIH staff can improve the number of minority applicants who win grants is to modify the standard granting procedure. “We’ve had training grants that have done exceptionally well. In review of … one [grant cycle] … we gave [grantees] limited funding. And we gave them limited funding with milestones as to how they were going to achieve this. So, instead of them getting the full five years they were given three years, which meant they had to … apply for the grant after two years. And they were given milestones basically of what they could do to encourage people….”
Some program administrators offered insights about the problems associated with research capacity-building awards to minority-serving institutions. These awards are granted to institutions that demonstrate a commitment to research and have a clear research plan. In those circumstances, reviewers are instructed “to look at how well the faculty [are] prepared in terms of being able to do research—the kinds of research that they are proposing—whether the project is well thought out. If there’s a need for the kind of research being proposed—the experimental details … are there….” If all of those things and more are present, then the capacity-building award is more likely.
One exception to this rule may be the Bridges to the Doctorate program. The criteria for the Bridges program may be more moderate because of the relationship between the two involved institutions. “For Bridges, I’m looking for institutions that have a real interest in educational reform. Institutions that have pools of minority students with potential [and] who really care about developing that potential, … who have a real interest in scientific research as opposed to technical capacity…. I’m interested in research training.”
Some respondents commented that the criteria for minority training grants may include too many barriers for even the most qualified minority institutions. It may be that some institutional applicants are schools that do not already have the infrastructure to sustain an effective and ongoing research program. Faculty at these schools often have a full teaching load with as many as four courses a semester. Further, there is reason to believe that it may be particularly difficult to enlist the caliber of researchers needed to serve as mentors. One respondent said, “People that are just really good. And the real good people … [have heavy] workloads … so how do we … collaborate [with these good] mentors? The really good investigators are too busy to give the kind of time required to be good mentors, too.”
In fact, a couple of respondents suggested that NIH consider revising its grant review policy to include a more culturally sensitive approach that does not pit the minority schools against the rest of the research community using mainstream research community standards. ”But in general, minority students, … some of them are being educated in very good schools. They come from Harvard or MIT [Massachusetts Institute of Technology]. They have a different demeanor than a person who comes from a … very small minority school.” This may influence the way minority pa rticipants are viewed and may hinder their progress in working toward successful research careers.
With some NIH institutes, the minority graduate student seeking fellowship funding will have a better chance of success if the application reflects an early interest in the proposed research area or science in general. “Even some of them, as a high school student, they have demonstrated some kind of interest in science.” An undergraduate science major is also a boost to likely funding as a graduate student: “The other thing though I would look for is what they have done during their undergrad education. If some of them … don’t choose a major until sophomore year and beyond the sophomore
year … the ones that are interested in science, they really would like to start pursuing science the first year. They are loaded with academic requirements for science programs.”
Beyond issues that involve improvements in the applicants and programs, several NIH respondents suggested that changes in NIH policies and procedures could be made that would also help improve the effectiveness of grant training programs. For example, staff expressed a number of concerns about their work environment and its effect on their performance. For example, the impact of outsourcing within the government is reflected in morale problems. “Also, all of the federal government is going through something called A-76. This is outsourcing. We’ve worked real hard to build a really good working team that likes to work together … does a good job. We know each other, … [we] may lose all … [staff] support, … it is exquisitely demoralizing for everybody.” A related NIH staff complaint is the workload: “The real problem that we have is … that we would need more people to really have [an effect on training] … the workload [is increasing] because some of the programs have been growing enormously. That means … more work for the same number of people. So, we are really stretched very thin.” The time pressures, as perceived by NIH staff, have made it difficult to be effective at managing their training programs.
A few respondents commented that mentors need to be trained in mentoring. “Well, I don’t know if this is really NIH’s job, but mentors need training on how to be a mentor. There are those who are natural at it, and there are those who really don’t know how to do it, and so … NIH might have a role in providing mentoring to the mentors.”
Several respondents commented on grants management in general: “One is if institutions we work with have better training in how to manage grants. And the other is if our mechanisms were managed and these clusters of grants were streamlined considerably. I think our grant management practices for these grants are very cumbersome.”
When talking about outreach barriers to minority trainees, some respondents raised the issue of communication challenges. “I think a lot of the efficiency in the process … depends on me. I see myself more as a facilitator … for the applicants than anything else. Sometimes we complain that the grants administration branch takes too long to award the grants that have been recommended for funding…. They really sometimes take a long time.” There have also been complaints about difficulties in accessing administrative offices within the Department of Health and Human Services Administration.
Definition of Program Goals
NIH program administrators reflect diverse perceptions about the goals of research training programs, particularly as they pertain to underrepresented minorities. A majority indicated that programs are geared toward the goal of producing Ph.D.-level researchers. Others indicated that interim vocational outcomes, such as entering the science field at different levels of training, are just as important for generating science role models. Most respondents indicated that the primary goal of the training programs is to produce more researchers in the sciences NIH supports. “With the training grants and career awards, our only interest is the training and we want to push them along until they … become researchers.” Or at least the goal is to produce researchers with promise. They will produce “publications, scientific presentations and so forth. Keep in mind that R01 grants will come. We’re just not there yet.” The ideal outcome expressed by several NIH respondents was “We have some people that come from K01 awards, that go right into the R01…. That’s the big grant. … that’s the one we want to see.”
The same point made by both statements above is also expressed in terms of institutional award programs. In this case, the goals are “increasing research capacity at minority and minority-serving institutions, addressing the underrepresentation of … minorities and other groups, disabled individuals, and so forth in the scientific workforce.” Another goal has a broader impact on the country: “And then the other … goal of this office is to … address health disparities.”
There are some variations on the theme of producing minority scientists. For example, one institute “introduced [minority support] at the dissertation stage because we believe that underrepresented minority students had a particular … difficulty in finding funds…. So that program was meant to address that particular transition. And I suppose it was the easiest program to measure success because every single one of the people we looked at had completed their dissertation and got their doctorate.” In the final analysis, “a success for us is anybody who actually completes the Ph.D.”
At the undergraduate level, the expressed goals are clearly focused on continuing along the education pipeline: “This student with some supplement [will] … go on to get the F31 [or] … go on to graduate school.”
A few respondents identified a different criterion for success, in lieu of formal evaluation, in terms of the number of applications and even number of successful applications. “The target is the number of applications that we would like to get funded. For example, we would like to have a portfolio of let’s say 60 F31 applications.”
There seems to be agreement among many respondents that the details of outcomes are not yet well established. It is possible that academic settings are not the only places in which successful scientists can work. “I think we’re just starting to take a look at it now from all of the programs, but I think there’s enough where … outcomes of all of these career awards [might be evaluated]. How many of these [funded students and
faculty] are getting there? And also, … individuals can go into research and industry and still be doing what they were trained to do.”
Although most respondents gave the same goal of recruiting scientists into various fields and producing scientists who win NIH R01 grants, a few respondents agreed that not all participants should be expected to end up in that situation. These few PAIC respondents who differ from the norm, seemed to feel that progress may be multigenerational: “[undergraduates] are a little bit less successful than pre- and postdocs. … it would be unrealistic to expect every trainee to continue to the next stage. [But] we would like to see that happen.” In discussing predoctoral training programs, one respondent recognized the possibility that phasing science into underrepresented minority communities may be a positive outcome. “What’s the benefit to NIH? … as a whole for … training scientists who are going to go into academia … that’s where the next generation of scientists are going to come from. … 40 percent of them, maybe more in the future, will be minorities, so they’ll also be role models.” Short-term goals may be more realistic than expecting minority Ph.D.s at the first effort. One person characterized this perspective as holistic. “First of all, science is a broad domain. There are 27 institute centers and divisions at the NIH, so perhaps if they choose not to continue within our scientific mission area, they may elect another mission area, related to us or unrelated to us. We don’t consider that a failure. We do view this enterprise, you know, a bit more holistically.”
A second divergent point of view among the respondents is the unique and interesting perspective on the criteria for success among targeted programs: “Another marker of success would be once those people [in targeted programs] are off and running, … are they going back and reaching back for somebody else with the kind of help to … bring them along. I can say that I’m seeing that now.”
More often than not, respondents assured the interviewers that the institute is genuinely interested in increasing the number of minority scientists in the field. “And we really would like to see more minority individuals [participating] because we [dominate science] in the [majority] populations.” This quote was from an individual managing only nontargeted mechanisms. Individuals managing a minority project also strongly supported the idea of programs targeting underrepresented minorities to get them into science: “to integrate these folks into the mainstream. And I think … that kind of philosophy and attitude and way of thinking will come back and help the … next generation. But I think that without these kinds of programs and these kinds of support mechanisms … this underrepresentation and isolation … will continue at the detriment … of increasing the diversity that is absolutely critical.”
One NIH staff member suggested that recruiting minorities is a way of increasing the labor pool in fields with extensive shortages of scientists. “There’s a shortage in many areas of research and science in general in the country. And you’ve asked the premier health organization in the country and I think we’re really in a position where we should be reaching out and trying to target … underrepresented minorities and bring
them into the loop.” Others see that the need to pursue more minority researchers is based on rectifying historical deficits: “If you take a look at the people who consistently have been getting grants … from NIH in general … they tend not to be people of color. For historical reasons, obviously … educational opportunities for kids … for persons of color stink…. And so there was a real attempt to get … them involved in … a bachelor’s degree and then on into professional degrees.” However, another individual suggested that the minority population was a special needs group: “It’s been a continuing interest from the top … recognition of the health disparities that exist … and recognition of the underrepresented minority investigators in the field.”
In some cases the institutes use available mechanisms to help build careers of all promising researchers. For example, a person who is awarded a K01 could end up with a subsequent series of awards. “Then the K22 is our transitional award and that award is made to dovetail from the K01, or either K08 or K23, and is tailored to provide protected time for individuals who are pretty much ready to be independent or have been independent for two years or less … they just need a little protected time to get themselves ready to go into the big grant arena. The R01 arena…”. In this case the goal is to sustain support until the individual reaches as far as possible along the training pipeline.
An interesting digression from the usual perspective on clinical research support was raised by one respondent, who suggested the possibility that K01 awards could be used to support minority clinical research faculty. It is a particularly interesting approach because of the health disparity problem in this country. “The clinical arena many times helps to shape where the research questions are coming from. So we need to target that population. I think there are a lot of people there that would benefit from this.”
A few program administrators view underrepresented minorities as just another group and they do not view the issue of increased minority participation in science as important. “The postdocs themselves need not be minority. I would say that the current appointees are probably at least 40 percent minority, which is higher than typical, average…”. This respondent manages a mixed array of targeted and nontargeted programs at both the individual and the institutional levels. “So we discontinued the R03 program for minority students and allowed it to be supplanted by the generic individual fellowship program under the F31 mechanism. And that transition has worked out well. So those applicants who would have previously perhaps come in through the minority R03 mechanism … would be coming in for the F31 on a faster review cycle.”
A number of NIH program administrators mentioned moving away from targeted programs: “We did not see an advantage to the targeted approach in the case of predoctoral fellows as opposed to the generic approach…. And NIH central went along with us. We did try another targeted program, and that is a career award….” A few respondents admittedly took exception to the practice of minority-targeted programs, saying that such programs should target disadvantaged individuals, rather than racial or ethnic minorities. In one case, an administrator of a minority-targeted program admitted
to steering prospective minority applicants away the minority-focused program and into a nontargeted program because of a belief that “minority-targeted programs are unconstitutional and cannot be defended in court.”
In debating the merits of targeted programs, one respondent said that no program is truly targeted; rather, they are open to any group. “There are none that are exclusive. One that comes closest is a Bridges program. And its authorization and its language says that recipients must be underrepresented minorities as defined by the institution.” Thus, any individual representing a minority, in the eyes of the awardee institution, is eligible.
A few program administrators expressed frustration with members of underrepresented minority groups who participated in their programs. One respondent explained, “Just do[ing a program just] for underrepresented minorities. We [have had] countless problems…. You know they’re bound to leave and don’t tell us so we [don’t] know how to handle [it].” The respondent, in this case, manages a targeted program for junior faculty. “If an institution is a small institution—private, public, you know, mostly a teaching institution with very few grants, they don’t have these sophisticated means to know everything about the grant so that they need a lot of help and they call us.” The staff member went on to say that extensive technical assistance is part of the support needed to foster more targeted minorities’ participation in these programs.
Reliable Methods of Outcomes Monitoring
The greatest agreement among the NIH program administrators was the near-universal concern about the lack of consistent and reliable tracking information on the subsequent careers of individuals whose training was supported by NIH.
As noted in Chapter 2 of this report, there are major shortcomings in the NIH trainee data collection system. Although a new automated tracking system is a possibility in future NIH plans, the paucity of good reliable data remains a major barrier to the evaluation of targeted programs, as noted by one respondent: “The biggest issue that we have is how to collect data to demonstrate that these programs do or don’t work. … [A lack of organized ] data is our biggest snafu. We’ve got tons of data sitting [in] files, paper files. There’s data sitting in our program director’s file about how many hundreds of institutions out there that would be useful to have. But collecting that data is one thing and knowing what to compare it to is something else. And one reason we never had … to collect it effectively is we never knew what to compare it to. We never knew, we have never resolved the issue of how you analyze that data. And without some sense of … analysis, how do you use it so it can tell you something? … we’ve never been able to make the case….” Others mentioned the need for a decent participant tracking system for analyses. “Well, I think the tracking system would be a big one …, if I can just get some systematic information … and do some real analysis.”
Perhaps the NIH team needs to differentiate the functions of research and evaluation. “The people in evaluation speak, talk a different language than geneticists and cell biologists and so, consequently, if our leadership is geneticists and cell biologists, … we sometimes have a difficulty appreciating [the evaluation methodology].” Another evaluation activity suggested by PAIC respondents is the development of a long-term follow-up study that will require an effective tracking system.
The majority of responses indicated that little to no program evaluation has occurred yet. Most respondents said that they wanted to have an evaluation and/or that a plan to evaluate is in the works. “A formal evaluation…. We are thinking of doing that as soon as we get some free time because there is evaluation money and this is something that has to be done.” Alternatively, it is not seen as the respondent’s job: “This evaluation stuff is just not my area.”
Monitoring success in targeted programs is greatly hampered by the lack of data on race or ethnicity. In many cases, there are no race or ethnicity data about the individuals applying for support. “We are hoping that there are minority individuals being included in the right programs, but we have no idea.” One respondent noted that “it’s just unfortunate that [racial or ethnic] data isn’t really asked because we don’t have the measure of, really, who….” Without a clear picture of how many minorities enter science, it will be difficult to mobilize institute interest in growing the number of minority researchers.
One respondent described the situation with targeted programs and their effectiveness as a question yet to be answered. “We often struggle with that because we are trying to see where all the programs are going. I think that what the NIH needs to do is … really beginning to look up all the minority programs because I know that there are all kinds of minority programs throughout the NIH. … I think that the big question is, are we really meeting our goals? How are the numbers [of minority scientists] increasing …? But you need to have a good evaluation strategy with people being willing to invest the money that it takes….”
Some groups conduct local evaluations of their programs. “For the research supplement program, I just completed an evaluation for the director. So I do have the information on that….” The nature of the evaluation was not made clear, and the results were expressed in qualitative terms such as patience, technical assistance support, and visibility.
Some of the barriers to evaluation include the cost, clearances, changing priorities, and data infrastructure. “Our hope was that in having this underlying database we would be able to at least go back and get data on the performance of students…. And it was built with the idea of being able to do program evaluation. As it turns out, it hasn’t been successful in doing that and there are a couple of reasons. One is that we cannot get institutional baseline data. It’s just not there. So there’s no way to say, based
on what’s in the database, this institution has improved its transfer rate…. The other problem is that grants plug along, then they may get six years of support and then they don’t get supported…. Once they lose their support they stop tracking students.” Perhaps the burden of evaluating NIH programs might be more effectively applied as an NIH activity with appropriate infrastructure and administrative support.
According to some NIH staff, attrition, defined as those not entering research careers, is hard to document “because they require tremendous amount of data. They require information on the training pool, they require information on the mentors and on the trainees, what they’ve done, and they also record information on where the trainees are going after they’ve left the training program.”
Because of the qualitative approach taken with the NIH program administrator interviews, the results are obliquely tied to the committee charge. An additional consequence is the fact that the interviews often fail to differentiate the specific program, the career stage, or even whether or not the program under discussion was targeted. As a result, the interview data provide an overall context in which NIH training programs are supported, rather than the detailed mechanics of program operations. Viewed from this perspective, the interviews provided useful information relative to the committee charge. In general, NIH program administrators were supportive of the interview process and its purpose.
There seems to be a strong opinion among a large number of NIH program administrators that NIH should do more to recruit and retain underrepresented minorities in science. Although, a slight majority of respondents view current NIH recruitment efforts (postings on the web, announcements in professional journals, and speaking engagements at professional meetings) as adequate, a sizable fraction believes that current efforts are inadequate. The underrepresentation of minorities in science, according to many of the respondents, results in inadequate scientific input from divergent social or cultural perspectives and detracts from our nation’s ability to resolve health disparities.
Several respondents recommended that the NIH training community review its recruitment and retention practices to find new ways to include more minorities in science training programs. Some respondents recommended an aggressive trainee tracking effort. Others recommended that program administrators provide more technical assistance to minority applicants, in order improve their chances of being funded on a second round.
Minority junior faculty may also require specialized support to enhance their ability to conduct scientific research. For example, faculty serving at smaller schools need more support than is currently offered, because of their heavy teaching load and
other academic responsibilities. New mechanisms, such as funding for “teaching release time,” may have to be developed in order to impact the numbers of minority faculty engaged in research. The NIH program administrators indicated a need for more mentor training as a way to improve outcomes.
Another approach to increasing the number of underrepresented minorities in science requires that the NIH community review its own policies and procedures. Issues to be addressed include the following: What is being done to sustain NIH staff investment in the minority trainee development process? What steps have been taken to maintain a high level of staff competence and morale? NIH administrators need to delineate more clearly the goals of minority-focused programs, and they must provide strong evidence of support for those goals.
There is a clear consensus among the NIH program administrators participating in these interviews that NIH needs to establish a systematic and effective trainee tracking system. Such a system should be able to track the career outcomes of any person supported with NIH funds. An ancillary concern expressed by several other respondents is the need for ongoing internal and external evaluations of the NIH training programs. At this juncture, there is no way to evaluate whether the training programs are effective for underrepresented minorities. A successful effort at tracking requires that NIH provide specific financial and logistical support to ongoing data management and evaluation of all of its training programs, including those for underrepresented minorities.