Findings and Recommendations
The National Institutes of Health (NIH) is the world’s foremost medical research center. Its web site states, “The goal of NIH is to acquire new knowledge to help prevent, detect, diagnose, and treat disease and disability, from the rarest genetic disorder to the common cold. The NIH mission is to uncover new knowledge that will lead to better health for everyone. NIH works toward that mission by conducting research in its own laboratories; supporting the research of non-Federal scientists in universities, medical schools, hospitals, and research institutions throughout the country and abroad; helping in the training of research investigators; and fostering communication of medical and health sciences information.”54
The training of research investigators is a critical component in the attainment of NIH research goals. Efforts by NIH and the research community to increase the participation of underrepresented groups in these training activities are important for maximizing the potential of individuals from underrepresented groups and the biomedical research enterprise. Recent data on graduate enrollments in science and engineering have shown a long-term decline in the number of white males enrolling in NIH research training programs over the last decade. Moreover, the number of international trainees participating in U.S. science and engineering graduate and postdoctoral training programs is also in danger of sharp decline given recent world events. Minority groups therefore, are largely untapped populations that can help to remedy a significant and growing problem.
For this study, the committee was charged with (1) assessing how well these NIH minority research training programs work; (2) identifying characteristics of successful programs, trainees and institutions; (3) recommending strategies to render future assessments feasible; and (4) providing recommendations for a coordinated trainee tracking information system. This chapter synthesizes findings from across the career stage levels and concludes with specific policy recommendations for NIH. These recommendations suggest ways to enhance NIH’s minority research training programs and provide guidance to NIH for future data collection efforts that will enhance the ability of evaluators to assess the success of these programs at regular intervals. This report does not address whether or how ineffectual programs should be dealt with. The
committee believes this is a matter of policy that is best addressed by NIH once legitimate program evaluation becomes feasible.
Data Collection and Accountability
In the course of assessing NIH’s minority research training programs, the study committee engaged in a lengthy and detailed information-gathering process. This effort to collect information quickly became a diagnostic test of the NIH trainee data systems. The test found these data systems and the information available to be uncoordinated and inadequate for the task the committee was assigned. The committee identified the following concerns:
Numerous NIH trainee data sets are distributed across NIH institutes and centers (ICs) in both hard-copy (i.e., annual progress reports, grant supplements) and digital formats (i.e., in-house Excel spreadsheets). Furthermore, archival trainee data sets are housed in off-site storage facilities. The committee determined that some essential trainee tracking data are already being collected on a regular basis by the ICs, but since these data are not stored centrally and electronically, and since the data do not use common definitions, are in different formats, and include quite different forms of information, the task of assessing trainee outcomes across all NIH research training programs is not currently feasible.55
Although NIH-wide trainee data sets such as the Trainee Fellow File (TFF) and the Information for Management, Planning, Analysis, and Coordination (IMPAC) system are centralized and electronic, they do not contain adequate trainee tracking data. The data sets are similarly inadequate to identify minority trainees who participated in targeted programs, because no data element contained within these data sets specifically denotes a program as targeted to minorities. NIH may wish to consider incorporating such a data element into these systems in the future. To that end, NIH-wide consensus on trainee data collection practices in general will ultimately enhance the ability of future evaluators to assess these programs.
In part because of the absence of NIH-wide electronic trainee tracking data, the NIH data contractor achieved a very low response rate from its efforts to locate and interview trainees. This was the case despite its use of two commercial and proprietary credit card databases that together maintain credit card-related contact information for millions of Americans and the query of the U.S. Postal Service address-forwarding database. The committee was disappointed, but not entirely surprised, by the low response rate. Establishing an NIH-wide, centralized, electronic data collection system for trainee tracking is necessary to assess program efficacy on an ongoing basis. Current plans for an electronic system—the electronic Streamlined Noncompeting Award Process (eSNAP)—that would capture trainee tracking data supplied to NIH by recipient institutions in the context of annual progress reports include collection of the following data elements for all trainees: trainee’s first, middle, and last name; date of birth; Social Security number; degrees earned; and role on the project. It is important to note that these data are insufficient for tracking trainee outcomes, as evidenced by the trainee survey discussed in this report. NIH may wish to consider asking trainees to provide parents’ contact information and/or contact information for three individuals who will always know how to locate the trainee. Doing so will render the task of locating trainees after they move on from the program much easier.
The committee experienced additional difficulty at the outset of the study, because it was unable to obtain a comprehensive listing of minority research training programs, current or past, supported by NIH. Perhaps due to the distributed nature of the NIH campus, the independence of the ICs, and the dynamic nature with which programming is offered or retired, no one at NIH maintains such a list. The committee believes that having a ready means for maintaining such an inventory of active programs will greatly assist future accountability practices.
Another difficulty faced by the committee at the study’s outset was the prohibition against accessing or viewing individual trainee race and gender data. Given the necessity of distinguishing minority from nonminority trainees for purposes of carrying out this study, the committee had to rely on an intermediary NIH-approved contractor that was allowed access to individual trainee race and gender data. Since the National Academies had no direct contractual relationship with the NIH-approved contractor, it had little leverage in terms of the deliverables produced.
The committee is cognizant of the sensitivity of race and gender data and the degree to which the NIH Office of the Director strives to protect the privacy of its trainees and grantees, but in this case it made the very task the committee was contracted by NIH to conduct, very difficult. Thus, NIH may wish to revisit its policy regarding access to individual trainee race and gender data for circumstances in which an outside evaluator is used to conduct research training programs assessment. It may also want to revisit the value of having more than one contractor approved for access to individual trainee data.
The committee observed a lack of coordination among NIH minority research training IC representatives. Many of these IC representatives agreed that this lack of coordination works to the detriment of the programs’ overall ability to be effective. At the study’s outset in 2001, the committee convened a meeting of minority training IC representatives on the NIH campus to discuss the administration of these minority research training programs. The IC representatives commented that it was the first time they had ever met in the same room with their programmatic colleagues. The committee believes that discussion of these programs among IC representatives is critical to the overall health and evolution of the programs and will improve the larger effort of collecting coordinated trainee tracking information in the future. NIH may wish to consider establishing a vehicle for bringing minority training IC representatives together on a regular basis. A series of “best-practices” discussions among IC representatives should be part of the ongoing dialogue within this group.
The aforementioned observations provided the committee with a rich appreciation of the complexity inherent to supporting, managing, and accounting for NIH research training program expenditures, whether targeted or not. It is the committee’s view that, at this time, NIH is not adequately equipped for full accountability of its research training programs, targeted or untargeted. The ability to document trainee outcomes is so central to the task of training program evaluation that no remedy short of that goal may suffice.
In addition, the committee believes it is important for NIH to engage trainees actively in the tracking process in order to document trainee outcomes. Throughout the trainee interviews, respondents expressed positive regard for both the programs and NIH’s first effort to contact them directly for feedback. The committee believes that trainee tracking participation should be mandatory (i.e., as a condition of support) and continue annually for five years following the trainees’ participation in the program. Five years will allow sufficient time for most trainees to advance to the next educational or career stage. NIH may wish to consider establishing a system by which trainees can logon to an NIH web site annually, in order to update their contact information and educational or vocational activities. Given the financial costs of trying to locate trainees after they leave the training program and the very low probability of actually finding them, investment in a tracking protocol that engages trainees directly will conserve future resources and render future assessments feasible. Another reason to engage trainees in future evaluations is simply because they are the programs’ primary informants. When a trainee chooses to leave science, for example, NIH needs to know why. Such data are critical to informing the optimization of program features such as recruitment activities, trainee selection criteria, and mentoring practices. In this way, evaluation becomes an iterative process and programs may continue to evolve to greater and greater specificity and success.
Policy and Program Observations
In the following paragraphs, the committee offers some ideas and suggestions based on its efforts to assess whether NIH minority research training programs work.
A thorough analysis of historic NIH program announcements for minority research training programs showed that the stated goal of these programs is, and always has been, to increase the number of Ph.D.-level minority biomedical researchers. Earlier phases of this study recommend that evaluation of minority research training programs should employ “advancement to the next step” in the science, technology, engineering and mathematics (STEM) pipeline as a metric for assessing program success.56,57 For reasons described in this report, the study committee could not reliably document trainee educational and career outcomes. Thus, it was unable to apply either metric of program success reliably and, as a result, decided to address the issue on philosophical grounds using qualitative descriptors. For example, respondent data are reported using nonspecific terms such as “a majority of respondents said” or “a minority of respondents said”. Such phrases should not be equated with statistical significance.
Furthermore, it is the committee’s view that NIH minority research training programs have intrinsic value both to individual trainees and society as a whole, even when a Ph.D. is not conferred, by virtue of their “value added.”
Given the stated mission of NIH, increasing the number of minority doctoral-level biomedical researchers is an appropriate benchmark for program success, although it must be acknowledged that, in reality, only a subset of trainees will achieve this, as is true for nonminority trainees. In other words, some measure of trainee loss from the programs (and therefore the STEM pipeline) should be expected and tolerated, as it is for nontargeted programs. The committee, therefore, asks, “Should those trainees who do not go on to earn a doctoral-level degree be viewed as program failures?” The committee does not support this view on the grounds that when minority trainees leave a training program to become lab managers or employees in the biotech or pharmaceutical industries, they are persisting in science and are visible as such. This is value added.
It is not realistic to expect that every minority program trainee who participates in the NIH targeted programs will successfully complete doctoral-level research training. If that were the case, it would suggest that the program was merely harvesting talent rather than growing talent. Harvesting talent in this context means supporting trainees who probably would have, for a variety of reasons, “made it” regardless of support from targeted programs. Growing talent, on the other hand, refers to the promotion of science
and science careers to individuals and communities that may not otherwise have entered science. Growing nascent talent implies that some program participants will indeed say, “No, thank you” to science, but still others will become turned on to science for the first time. The latter group is an important target audience for these programs. An excellent training program will, therefore, harvest and grow talent to appreciable degrees.
The committee goes so far as to recommend that one as-yet-untapped sector for growing nascent talent is the pool of high achieving non-science majors. Many bright undergraduates major in nonscientific fields not because they do not enjoy or excel at science but because they appreciate that college may be the only time in their life when they are free to explore a discipline without regard for its vocational implications. Thus, a music or philosophy major who successfully masters core science courses, such as calculus, chemistry, and physics, may make an excellent graduate student in the biomedical sciences, by virtue of his or her broad-based education. Such students have the potential to infuse science with uncommon creativity and synthesis.58 Furthermore, because biomedical research training has clear and direct vocational applications, such students may be highly motivated to persist in biomedical research through a full career.
Minority Training Programs: What Works and What Doesn’t?
The committee concludes that underrepresented minorities are entering the biomedical workforce as a direct result of the NIH minority research training programs. Further, among trainees at all career stages, there is profound appreciation for what these programs offer and recognition of the prestige associated with being an NIH research trainee. The number-one “best feature” cited by trainee respondents across all career stages is the research experience itself. For undergraduate trainees, the acquisition of laboratory skills was a key factor. For graduate trainees, laboratory experience was important but so were graduate-level coursework, research seminars and workshops, learning how to think critically, learning to make cogent research presentations, and learning to teach science to undergraduates. Among postdoctoral and junior faculty
trainees, the opportunity to choose a subspecialty and develop research independence was the most valuable aspect of the training programs.
Among undergraduate trainees, mentoring support was cited as the second most valuable feature of the training programs. Mentoring was most often provided in four key areas: (1) improving the trainee’s research skills, (2) providing motivation and personal growth, (3) providing career guidance, and (4) promoting the trainee for scholarships and other development opportunities. Mentoring was also very important to graduate, postdoctoral, and junior faculty trainees, who reported many positive interactions and support from their mentors. Given the importance placed on mentoring by trainees across all career stage levels, NIH may wish to consider conducting a more in-depth best-practices examination of successful mentoring practices and associated behaviors. For that matter, NIH may also wish to examine best practices with regard to marketing a research career to prospective trainees. How do the best programs “close the sale”?
Financial support from the training programs was greatly appreciated by undergraduate trainees. Such support came in the form of stipends, summer research, and conference travel support. Funding was, for graduate trainees, frequently cited as a best feature. At the graduate level, a funding arrangement exists whereby NIH covers the cost of research training, including stipend and tuition support, research supplies, and benefits. In return, NIH requires that trainees refrain from taking outside jobs in order to devote 100 percent effort to the training experience. For postdoctoral and junior faculty trainees, funding was characterized as “critical and necessary.” The “protected time” that funding provided trainees at this level allowed them to achieve research independence, which is the foremost goal of these programs.
Other positive program elements that trainees mentioned include the foundation of scientific knowledge that the program provided to undergraduate trainees; the opportunities to network and collaborate with other scientists was mentioned by trainees at all levels, especially graduate trainees. Undergraduate trainees underscored the ability of the programs to help them decide whether to attend graduate school or medical school. Graduate and postdoctoral trainees frequently cited the tremendous value of learning how to prepare a competitive grant proposal. Among junior faculty trainees, the K01 award, they said, allowed them to progress to the next step in their careers, namely to obtain an R01 research grant.
Administrators of these programs mentioned that there are many more applicants to undergraduate programs than there are available positions. Thus, recruitment appears to be highly effective and attrition from undergraduate programs is minimal. This is due in part to an effective system of oversight and monitoring of undergraduate trainees’ progress. At the post-Ph.D. level, there appears to be a sharp drop-off among minority trainees. An indicator of this is the gender shift from predominantly female at the undergraduate and graduate career stages to predominantly male at the postdoctoral and
junior faculty career stages (see Appendix E). Where do the minority female trainees go? This question warrants further study by NIH.
Across the board, trainees, while extremely grateful for the funding support afforded by the training programs, uniformly stated that the levels of funding support are not sufficient and need to be increased. Undergraduates who are greatly challenged by the addition of a demanding research program to a full load of coursework must often take on additional outside work, in order to make ends meet. Program administrators call this situation a “recipe for disaster,” and it constitutes a barrier against participation in these programs for lower-income minority students. Graduate trainees have similar complaints. They are contractually prohibited from obtaining outside jobs, yet the stipend support is barely above the poverty line. In the context of the uneven health benefits afforded by these programs, this too seems like a “catch-22.” Postdoctoral and junior faculty trainees are similarly disheartened by the low stipends afforded by the training programs. This is especially true when trainees have dependents and/or live in major metropolitan areas where the cost of living vastly exceeds what the stipend offers. All trainee respondents were clear and forceful in stating that trainee stipends have to be more in line with market trends; they need to be increased in order to sustain and build student interest in research careers. This sentiment was echoed by numerous program administrators, one of whom stated that the stiffest competition faced in attracting African-American trainees to a research career comes from the salary opportunities provided by advanced health professional programs.
Mentoring, although highly cited as a positive element of the training programs, was criticized as needing significant improvement. Too many trainees reported negative mentoring experiences in the lab. Some minority undergraduate trainees were given mundane administrative tasks to perform in lieu of experiments; others experienced “benign neglect” by their mentors or, at best, a lack of encouragement. Half of the T32 minority postdoctoral fellows reported having no mentor at all, and a trend that was not replicated by nonminority T32 postdoctoral trainees. This is a red flag to which NIH should pay attention especially in the context of the already low numbers of minority trainees at this relatively advanced career stage.
Training in the biomedical sciences historically assumes that if one is trained, one will therefore be a good trainer (mentor). This conclusion does not follow. Mentoring is a skill for which academic researchers rarely receive any formalized training. The old adage, “Do as I did” does not translate well in the context of today’s diverse student populations. Such selective mentoring may indeed explain, at least in part, the homogeneity seen among this study’s trainee respondents post-Ph.D. Those who are just like their mentor are promoted. Those who are different from their mentor are not. Program administrators emphasized that in addition to the lack of mentor training, mentors receive little credit or encouragement for the time taken to mentor trainees. Grants do not provide funds that cover mentoring activities and time. Academic departments do not view mentoring as a major activity that legitimately counts toward tenure. Yet mentoring is absolutely essential to the continued growth and sustenance of
our biomedical workforce. NIH may want to take a look at this issue and consider changing the value it places on this essential activity in some concrete way.
Minority respondents to our survey provided additional clues that may bear upon their low numbers at higher career stages. Based on the survey data, which the committee believes are biased toward the most successful NIH trainees, minorities publish fewer papers than do nonminority trainees. They have greater difficulty securing post-Ph.D. employment. They report less social integration in their laboratories, and this was the case more so for minority trainees at institutions using nonminority training mechanisms. Finally, a large fraction of minority trainees believe that their minority status in some way affected their training experience. Given that one-half of the minority postdoctoral survey respondents reported having no mentor at all, one wonders what factors are at play in these training environments that affect minority trainee outcomes so profoundly.
In its efforts to train a new generation of minority research scientists, NIH is by definition effecting change, however small, in minority communities. The committee believes that increasing a community’s interest in science requires more than a single generation. Achieving buy-in to the scientific enterprise requires not only that trainees become turned on to science, but also that families and community pillars demonstrate overt support for young persons’ interests in science. The goal of producing more minority doctoral-level researchers may at first seem straightforward, but it is not. No single trainee outcome suffices for measuring program success. Interim outcome measures are germane and informative and should be taken into account when assessing the impact of these programs. The degree to which minority children can realistically envision themselves as future scientists, leaders of science, and biotech entrepreneurs is a reflection of how successful the intervention of bringing science to minority communities truly is.
The committee recommends that NIH articulate a clear and measurable training philosophy and implement change in data collection to better support rigorous evaluation and accountability efforts beyond the doctorate. In the meantime, the committee also strongly recommends that NIH commit to the continued funding of minority-targeted research training programs and implement measures to better coordinate them across the 27 institutes and centers. Finally, the committee recommends that NIH fund an independent follow-up to this assessment within five years. The committee’s detailed recommendations follow.
By the end of 2005, the NIH director should articulate a set of clear and measurable training goals and objectives specific to minority training. The director should take into account the mission of NIH and the integral role of research training in attaining both societal goals (e.g., health and well-being, the ability to support oneself
and one’s family, community development) and research goals. Such a policy should be responsive to society’s workforce needs in their broadest sense, with an understanding that contributions to society derive from all parts of the career stage pipeline.
NIH should commit to the continued funding of minority-targeted research training programs. Although the committee cannot substantiate this recommendation in quantitative terms for reasons described throughout this report, it does so in qualitative terms, using survey data that were collected from trainees and program administrators who are the programs’ primary informants. The following reasons underlie this recommendation:
These programs have added many minorities to our science workforce.
The elimination of these programs would likely diminish the number of new minority scientists entering the scientific workforce.
The trainees interviewed indicate overwhelmingly that these programs benefited them. These programs provided research experiences, financial support, and mentoring that were critical to their career success.
Mentoring is a critical part of the career development of all scientists and is particularly important for minority trainees. Trainee survey data suggest that the diversity of mentors is greater in the minority-targeted programs than in the nontargeted programs. Atkinson et al,59 found that when rating mentoring relationships, both mentors and mentees rated their relationships more positively when they were matched for race or ethnicity.
The committee recognizes two distinct and valid approaches to the development of minority research trainees. The training policy of the NIH institutes and centers in conducting these programs should emphasize the development of trainees who have already demonstrated promise in the sciences, so that they can overcome the barriers to becoming productive investigators. Two examples of minority training programs that emphasize talent harvesting include the National Institute of Mental Health Career Opportunities in Research Training and Education (COR) and the Minority Access to Research Careers (MARC) Undergraduate Student Training in Academic Research (U*STAR) programs. The NIH training policy should also emphasize the development of other trainees—those without demonstrated science promise—in order to add to the pipeline of trainees interested in pursuing science careers. An example of a minority training program that emphasizes growing talent is the Bridges to the Baccalaureate program.
The implementation of this training policy should also consider the following. NIH should more vigorously monitor the use of racial or ethnic eligibility criteria for these programs. Survey data from trainees and program administrators indicate that non-underrepresented minorities are participating in minority-targeted training programs.
NIH should also examine gender differences among its trainee participants. For example, the minority trainee population at the undergraduate level is mostly female, but their proportion, in relationship to male trainees, declines at each successive career stage, suggesting the possibility of substantial attrition among women who could have continued on as investigators. This trend is particularly striking at the graduate-to-postdoctoral transition where men, conversely, outnumber women. This trend among women is independent of race.
Finally, given comments offered by both trainees and programs administrators on this issue, the committee recommends that NIH conduct a review to ensure that the research infrastructure (i.e., lab space, lab equipment, active faculty research programs) available to minority trainees at the institution level is adequate and, if not, seek ways to further address this programmatically.
The director of each institute should designate a single individual as minority research training programs coordinator for that institute by the third quarter of FY 2005. Some institutes have a centralized training coordinator; others do not. This recommendation would provide consistency and make coordinated efforts more feasible.
The NIH training director should convene a meeting of all minority training coordinators on at least a quarterly basis, beginning with the third quarter of 2005. The goal of these meetings is to coordinate the administration of NIH minority training programs and the collection of relevant program data. Currently, the programs and the accountability for them are so fragmented that external evaluation is not possible. Given the importance of the NIH training programs to the continuation of U.S. leadership in biomedical research, coordinated efforts to develop, manage, and rigorously evaluate research training programs are needed. The collective management of minority training programs, although not intended to supplant IC independence and expertise, requires ongoing communication and cooperation across disciplinary and institutional lines. It is expected that agendas for these meetings will change over time as the collaboration improves communication and advances meaningful planning. The meetings should at a minimum address the following issues:
Clarification of NIH training policies regarding trainee recruitment and documentation of program activities and results;
Discussion of the range of IC training program characteristics;
Sharing of trainee recruitment strategies;
Identification of effective elements of IC training programs;
Review of IC evaluation results; and
Development of long term objectives for addressing workforce needs and increasing the participation of underrepresented minorities in science.
The committee of minority training program coordinators should establish appropriate guidelines and measures for evaluating NIH minority research training programs. Training program administrators should participate in an ongoing and rigorous evaluation process. By defining program outcomes and monitoring their
achievement, the ICs can better manage their programs. NIH should commit to making available all of the data needed to conduct internal or external evaluation of its training programs.
Further study of the relative effectiveness of minority-targeted versus nontargeted programs should be carried out by NIH institutes and centers under coordination from the Office of the Director. The reasons for this recommendation include the following:
The ICs should establish outcome measures for each training mechanism in a coordinated fashion. To do this, the ICs should identify and document the range of trainee outcomes that result from participation in these programs. Then, the range of outcomes should be codified as either contributing or not contributing to the consensus definition of program success. The committee is cognizant that this recommendation reflects an interactive process.
Continued integrated study of these programs can identify the best features of the programs and best practices among the recipient colleges and universities.
The director of NIH training should administer the funds for evaluation, data collection, and marketing by FY 2006. The centralized training activities should include a centralized and robust evaluation and planning activity. This approach will empower the director of NIH training to be able to coordinate accountability mandates (i.e., the U.S. Office of Management and Budget’s Program Assessment Rating Tool and the Government Performance and Results Act) with organizational policies and procedures.
The general issues reviewed in this report should be revisited periodically at the NIH level with the next report submitted by 2009. The Office of the Director at NIH should take the lead on this. The numerous, weighty, and very public issues regarding affirmative action that are raised by targeted research training programs require continuing attention by a consortium of the National Center for Minority Health and Health Disparities, the Office of Extramural Programs, the institutes and centers that fund such training programs, and the NIH Office of the Director. NIH should conduct an independent public review and accounting that will help ensure that the programs remain focused and effective. Doing so will inform the affected groups and general public of the programs’ success and ensure that funding is being used effectively, thus yielding a positive return on the nation’s investment. The committee believes that five years is a good interval for external review of the program(s), although experience may show that more frequent review would be useful. The committee further believes that the breadth and depth of the issues, compounded by the present fragmentation of many components of NIH, require that the Office of the NIH Director take the lead on this.
NIH should develop a relational database that collects a minimum data set (MDS) for all persons who receive funding as trainees, fellows, research assistants, or postdoctorates, including those programs targeted to underrepresented minorities.
The database should be maintained by the Office of the Director of Extramural Programs, headed by the deputy director and NIH research and training officer. The Office of the Director of Extramural Programs should have the overall responsibility for coordination of the database and its constituent parts.
The MDS should be a service to all institutes and contain variables that enable rigorous evaluation and assessment of training programs; institutes may add variables at their discretion.
The MDS should collect data for all trainees, including all those funded through the training mechanisms covered here, as well as for research assistants funded through R and K awards.
The Office of the Director of Extramural Training, in coordination with institute representatives, should develop a data entry system accessible from multiple sources, including external data entry points, such as grant-specific progress reports. In addition, the Office of the Director of Extramural Training should develop a user-friendly data entry form for the MDS that is web-accessible. The database and data coordination in the deputy director’s office will emulate that of a coordinated data center. Considerations of personal privacy and confidentiality must be high on the list of necessary attributes.
The Office of the Deputy Director of Extramural Training should identify data elements that help in tracking persons who received training funds—both directly and indirectly. These tracking data should be obtained at the time of initial NIH funding and should be updated periodically.
Development of the MDS, database, and data entry system should begin immediately and be completed no later that FY 2008.