The National Institute of Justice (NIJ) is the only federal research agency devoted to the control of crime and the administration of justice, which are a central component of national identity. Since its establishment as part of the emerging federal role in assisting state and local officials in these important issues, NIJ has been the central research arm of the U.S. Department of Justice (DOJ). The charge to this committee, detailed in Chapter 1, was to assess the performance of this agency and to identify changes in the agency that would allow it to better achieve its mission.
Our review has demonstrated that NIJ is central to the better understanding of how to control crime and its impact on citizens and communities and how to ensure that the system of justice is fair and effective. No other federal, state, local, or private organization can do what NIJ was created to do. During the past 40 years, no other agency has contributed more to the understanding of crime and justice efforts than NIJ. This report has documented many of these accomplishments, including in the areas of crime mapping and hot spot policing, violence against women, the role of firearms and drugs in crime, drug courts, and forensic DNA analysis, as well as the success of the data archives, fellowships, and other outreach programs. Despite the many shortcomings of NIJ that are demonstrated in this report, the nation needs an NIJ—but one that functions better than the current one. We are convinced that a “new” NIJ, defined by the recommendations that follow, will better serve the nation and instill increased confidence in its justice system operations. We encourage Congress and the executive branch to provide the funding that the tasks assigned to NIJ require.
As this report demonstrates, few major initiatives in criminal justice during the past 40 years have occurred without significant involvement of research developed and sponsored by NIJ.
These contributions are in part noteworthy because they have occurred in an agency that, as documented in this report, has lacked the level of independence, appropriate leadership, funding, and operational practices that characterize much more successful federal research agencies. The appropriate level of independence is important to ensure that a research agency has the authority to set agendas through planned deliberations with the research and practitioner communities. Otherwise, its mission, decisions, and results can be influenced by nonscientific goals and objectives, and it loses credibility with these communities. Although NIJ has had periods of good leadership, it has never had leadership with experience and prestige in the research community. This not only reinforces the problems associated with limited independence, but it also results in leadership that cannot exercise independent scientific judgment and credibly involve the research community in its work. We have demonstrated that, given its mission, the funding of NIJ has been seriously inadequate, far too variable, and too subject to earmarks that remove funding decisions from peer review and scientific standards. We also have demonstrated that basic operations at NIJ—that is, agenda setting, peer review, report review and assessment—have not been consistent with those at other successful federal research agencies. These aspects of NIJ are unlikely to be greatly improved without attention to the issues of independence and resources.
The committee gave considerable attention to the issue of how best to achieve the appropriate level of independence for NIJ. Two basic approaches were considered: (1) moving NIJ out of the Office of Justice Programs (OJP) and (2) retaining NIJ in OJP and giving it a level of independence similar to other federal research agencies. In considering these options, the committee reviewed other federal research agencies, consulted with former directors of NIJ and OJP, and raised the issue with many others who offered guidance to the committee.1 We also included a question about NIJ’s independence on our survey of researchers and practitioners. In addition, the committee considered the recommendations of two other NRC committees that issued reports on related topics during our deliberations—the report on the needs of the forensic science community (National Research Council, 2009c) and the report on BJS (National Research Council, 2009a). After careful consideration of all the evidence the committee concluded that maintaining NIJ in OJP but with substantially increased levels of independence secured by Congress offered the greatest likelihood of NIJ achieving its mission. The
committee considered whether the independence NIJ needs to be an effective science agency can be achieved in other ways (e.g., an attorney general order or an OJP policy) and concluded that nonlegislative options could not be as effective. Only Congress can provide the requisites of increased independence and the necessary oversight to ensure that specific authorities cannot be easily retracted or eroded.
Strengthening Forensic Science in the United States: A Path Forward recommended that a new federal agency should be developed outside DOJ to address all the needs of the forensic science community. The primary rationale for this approach was the belief that placing such activities in a law enforcement agency could result in science being actually biased or perceived to be biased towards law enforcement agencies.2 In Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics, the recommendation was to move the Bureau of Justice Statistics (BJS) out of OJP into a new agency in DOJ that would report directly to the Attorney General. This recommendation was based on the conclusion that political influences from OJP had and could in the future bias BJS findings and reports.
The committee concluded that the concerns that prompted these recommendations were not major reasons for the problems we have described at NIJ. We also concluded that forensic science research fits within NIJ’s research mission. Furthermore, the committee received information that convinced us that having NIJ report directly to the attorney general was not likely to reduce potential political influence; that it is unlikely that a new research and statistical agency in DOJ would be established given the costs of doing so; that moving NIJ out of OJP would make linking research and action programs less likely; and that such a move would make it less likely for Congress and the administration to move funds from action agencies to NIJ as has happened in the past and is specified in the OJP current appropriations bill. The committee concluded that keeping NIJ in OJP but with increased independence and greater involvement of the research and practitioner communities has a better chance to result in an agency that can gain the trust and confidence of Congress, the administration, and the criminal justice community. If the changes we recommend are not implemented within 5 years or if they are and the problems we have identified persist then we recommend that the idea of moving this research function should be carefully revisited.3
Furthermore, the committee concluded there are a number of substan-
As described in Chapter 3, should Congress adopt the recent recommendation of the National Research Council (NRC) (2009c) and establish a new independent federal entity responsible for forensic science activities, NIJ’s future role in sponsoring forensic science research and development activities would need to be defined accordingly.
Currently, Congress is considering establishing a national crime commission. If it is formed, then it would be the natural body to conduct this review.
tive reasons for retaining NIJ within OJP. Foremost is that this relationship allows NIJ to be organizationally closest to those for whom its research is eventually intended. This does not mean that specific research projects will have immediately useful findings. However, unlike other federal research agencies, such as the National Institutes of Health (NIH) institutes and the National Science Foundation (NSF), NIJ’s mission includes a greater focus on development and evaluation activities designed to bridge research and practice. Being in OJP provides an opportunity for NIJ to advise and participate in the development of OJP-sponsored programs and to be in a position to evaluate those programs. Because criminal justice policy makers and practitioners are much more closely tied to OJP than NIJ, NIJ can use its OJP relationship as a bridge to the field. NIJ can then use these connections to better target its research to the needs of the field, to involve practitioners and their agencies more directly in research activities, and to increase the likelihood that its research will eventually have an impact on criminal justice practices and policies.
Because of the importance of the overall OJP program to the field and to DOJ, as well as the unprecedented levels of federal funding that have been made available in recent years, the OJP Assistant Attorney General is in a position to assist NIJ in making a compelling case for a well-designed and funded research program with the Office of Management and Budget (OMB), DOJ, and Congress. The need for research is particularly apparent given the expanded scope of OJP and DOJ program efforts, their partnerships with state and local criminal justice agencies and the expectations of Congress for well-designed, evidence-based programming.
Of course, leaving NIJ in OJP means that the relationships between these agencies need to be clearly specified in the authorizing legislation for each. NIJ should be able to set its own research agenda, determine the awards it makes, set monitoring and oversight procedures for its grantees, control publication and dissemination decisions, hire and supervise its allotted personnel, present its budget to the Attorney General and Congress, and interact with appropriate DOJ and congressional entities. When OJP provides some limited centralized services, it should be done in full cooperation with NIJ and without the level of interference in research decisions and operations that have occurred too often during the history of NIJ.
To address these issues the committee makes five recommendations: (1) to ensure NIJ’s independence and improve governance, (2) to strengthen its research mission, (3) to bolster the crime and justice research infrastructure, (4) to enhance the scientific integrity and transparency of its operations, and (5) to establish a culture of self-assessment. Our analysis strongly suggests that if these recommendations are implemented, NIJ can be a great leader in national efforts to reduce crime and improve the administration of justice.
ENSURE INDEPENDENCE AND IMPROVE GOVERNANCE
RECOMMENDATION 1: The committee recommends that Congress provide for the requisite independence and authority of the National Institute of Justice (NIJ) while retaining its organizational placement within the Office of Justice Programs and the U.S. Department of Justice. Among the key issues to be considered in pursuit of this goal are a statutory advisory board, a set term of office and minimum qualifications for the NIJ director, and clear authority for NIJ to make awards and control its budget and resources.
NIJ’s efforts to fulfill its research mission to build a research infrastructure and produce a body of knowledge that can contribute to a greater understanding of crime and society’s response have met with mixed success. Much of this is due to the deficiencies that have continued to persist since the last review. To become a more effective agency, NIJ needs control over its research and development (R&D) management operations. Like the prior review by the the NRC (1977), we recognize the need for independence, improved governance, and authority to conduct its work.
Need for Independence
The issue of independence for research and statistical agencies has been hotly debated for years. The 1931 Wickersham Commission, the 1967 President’s Crime Commission on Law Enforcement and Administration of Justice, and participants at the 30th Anniversary Symposium of the President’s Crime Commission held in 1997 all grappled with it.
The 1977 NRC report pointed out that the location of NIJ in the Law Enforcement Assistance Administration had not served it well. Its lack of independence and political constraints prevented the institute from fulfilling its mission of building knowledge about crime. The report warned that unless its present form was changed to an independent research agency within DOJ, with final approval authority over all awards as well as control over its administrative budget, personnel, and detailed program review, it was unlikely to become a “significant and quality-oriented research agency” (National Research Council, 1977, p. 3). In many respects, this prophecy could be made today exactly as it was more than 30 years ago.
As this report shows, the history of NIJ reflects diminishing authority and resources, not only as a result of congressional action, but also from actions taken by its oversight agency, OJP.
From our review of other federal research agencies, we are well aware that independence is an essential attribute shared by all of them. Conducting rigorous and objective research requires freedom from control by policy
makers and others who have a stake in the research findings and their implications for policy, programs, and practice. Independence does not necessarily imply that all federal research organizations need to be free-standing, like NSF, whose single purpose is “to promote the progress of science.” What it does imply is that clear lines of separation need to be made from the policy-making organization, with a balance maintained between independence and responsiveness to problems a parent agency is addressing. This appears to be best accomplished in mission-oriented agencies when program services (funding of direct or indirect services) are separated but related to the research unit. In addition, in these situations, the research agency should report to the parent agency’s top leadership to ensure that it receives the best scientific advice on issues of concern.
An effective research organization needs to have the independence to conduct its work. The key components of independence include authority to (1) establish research and evaluation priorities; (2) control the grant-making process, from solicitation through grant approval; (3) make budgetary recommendations to the highest level of the parent agency; (4) make staffing decisions, including recruitment and hiring; and (5) issue reports and other dissemination activities. An important aspect of these powers is the ability to avoid earmarks in the budgeting process. In order to achieve the level of independence necessary for a trusted crime and justice research agency, the new NIJ that we propose will need a strong advisory board, terms of service for a qualified director, and grant-making, budgetary, and staffing authority.
While advisory boards have been used in different ways over the years at NIJ, the committee concludes that they have never functioned as a scientific advisory board should—to set agendas, review the integrity of the research operations, and assess the accomplishments of the agency with the standards of science as the guiding principles. As described in Chapter 2, these groups were a disappointment to their respective NIJ directors. Neither was able to build support for NIJ by gaining the attention of Congress, nor was it able to serve as a buffer against what was regarded as a highly politicized environment in which everyone had a different take on crime and what was needed to combat it. There was never any sustained attempt to involve the members in a policy-making or priority-setting role.
Although scientific advisory boards are used in different ways by different federal research organizations, they appear to be central to the effective operation of these agencies. Advisory boards provide oversight, assist in
setting research agendas, may have approval authority for certain awards, and conduct reviews of agency operations. Federal research agencies may have one or more boards; if they have more than one, a distinction is made between ongoing oversight boards and those created periodically to assess performance. In all instances that the committee reviewed, scientific advisory boards are overwhelmingly comprised of research scientists but also may include a small number (perhaps 25 percent of membership) of those practicing in the field. Boards are usually nominated by the research agency and approved by the head of the agency that houses the research agency or a designee. They report only to the director of the research agency or if they are advisors to a subunit, to the head of that research component.
The experience with the NIJ advisory board differs markedly from that of other research agencies, including many with an operational mission. For example, the Environmental Protection Agency (EPA) has an overall Science Advisory Board to guide its various science programs as well as individual program boards, such as the Clean Air Scientific Advisory Committee. These science boards are distinct from policy boards and each has a separate role to play in guiding the agency’s operational and technology programs. The U.S. Department of Education (DoEd) has a 15-member research advisory board that works with the assistant secretary for educational research and improvement to develop the agency’s long-term research and development agenda. The board’s membership includes some policy officials, especially local school officials, but most of its members are scientists.
The two most prominent research agencies, NSF and NIH, have advisory groups that play a major role advising on policies and a wide range of research activities, approving priorities, soliciting advice from the field, suggesting how the research can be strengthened, and assessing the work of their respective organizations. The members of both these boards are appointed by the president, have set terms that are staggered, and are selected for their preeminence in research, a particular field of scientific endeavor, or public service.
Among the 27 institutes and centers that constitute NIH, 20 of them have some kind of advisory council, committee of counselors, or panels, and several have more than one kind of advisory group.4 The most common form of advisory group is the advisory council composed of 12-18 members, who are appointed by the secretary of health and human services in consultation with the institute director and serve 4-year terms. The National Advisory Council on Drug Abuse has 18 members, of whom
For example, the National Cancer Institute has five advisory groups: (1) the President’s Cancer Panel, (2) the National Cancer Advisory Board, (3) the Director’s Consumer Liaison Group, (4) the Board of Scientific Advisors, and (5) the Board of Scientific Counselors (see http://getinvolved.nih.gov/advisory_councils.asp [accessed March 17, 2010]).
12 are experts in scientific fields and 6 are knowledgeable members of the public.5 The National Institute on Alcohol Abuse and Alcoholism (NIAAA) has 15 members, of whom 10 represent health and scientific fields (including not less than 2 individuals who are leaders in the fields of public health and the behavioral or social sciences) relevant to the activities of NIAAA.6 The remaining one-third are selected from the public and include leaders in the fields of public policy, law, health policy, economics, and management. These councils approve scientific review group or peer-review recommendations. They also provide advice on research opportunities, guidance on ways to improve approaches to stimulating research, and ideas for working more closely with constituent groups and consumers of the research. Although these councils do not make policy, there is a tradition of their advice being heeded by the institute administrators.
Similarly, in addition to the presidentially appointed National Science Board, which provides overall guidance and oversight of the National Science Foundation, each of its seven directorates also has its own advisory committee. The Directorate of Social, Behavioral & Economic Sciences (SBE) is the principal source of federal support for fundamental research on human cognition and behavior and social structures and social interaction, as well as research on the intellectual and social contexts that govern the development and use of science and technology. It has an 18-member advisory committee that meets semiannually to review and advise on the impact of programs in the disciplines and fields encompassed by the directorate, provides oversight of overall program management and performance, and advises on the impact of NSF-wide policies on the SBE scientific community. A typical agenda includes briefings on recent scientific discoveries and activities, review of human capital and succession planning issues, a discussion with representatives of professional research member organizations, and review of program assessment reports produced by committees of visitors (described in Chapter 6).7
Our committee discussed whether the Office of Research and Evaluation (ORE) and the Office of Science and Technology (OST) should also have advisory boards and concluded that NIJ is currently too small to warrant such an elaborate advisory board structure. However, subcommittees of the proposed NIJ advisory board could be designated for more specific oversight of ORE and OST, for specific grants or contracts, or for both.
In proposing an advisory board for NIJ, the committee considered
Available: http://www.nida.nih.gov/about/organization/NACDA/NACDAHome.html [accessed March 17, 2010].
Available: http://www.niaaa.nih.gov/AboutNIAAA/AdvisoryCouncil/default.htm [accessed March 17, 2010].
Available: http://www.nsf.gov/sbe/advisory.jsp [accessed March 17, 2010].
the number of NIJ constituencies and decided 15 members was a reasonable number to cover all functions and perspectives. Given the scope of the agency and the responsibilities we have outlined for the advisory committee, it would be difficult to have a smaller membership. In determining who should be the appointing authority for the advisory board, the committee carefully considered three alternatives: appointment by the president, appointment by the attorney general, and appointment by the NIJ director. NIJ’s size (organization as well as resources) seemed to preclude it as being suitable for a presidential appointed advisory board. The committee thought that giving the NIJ director authority to appoint might provide too much power and give the perception that the members were in some way beholden to the director. The prior experience of having the attorney general as the appointing authority seemed to work well. Previous advisory boards appointed by the attorney general contained a mix of highly regarded and nationally recognized researchers, practitioners, and policy makers who were familiar with NIJ and supportive of its mission.
Using the experiences of these other federal agencies as a guide, the committee suggests the following structure for the NIJ advisory board:
15 members appointed by the attorney general for 5-year staggered terms.
A slate of nominees will be recommended to the attorney general by the NIJ director following consultation with institutional groups representing important research and practitioner communities. Interested parties may also recommend nominees either via the NIJ director or to the attorney general directly.
Ten board members will be experienced researchers; five members will be from the practitioner and policy communities. Their interests and expertise should be broadly representative of criminology and criminal justice and of issues of importance to both NIJ’s social science and technology research programs.
Additional persons from other federal research agencies as well as NIJ leadership may be invited by the advisory board to serve as ex officio members as appropriate.
Governance and Scheduling
The chair and vice chair should be elected by the board for terms of 2 years.
The board should meet at least semiannually.
The board will report directly to the director, NIJ.
Powers and Responsibilities:
Advise on policies involving grant selection and administration.
Advise on the development and implementation of research priorities.
Review and advise on appropriate procedures for technical and scientific peer review.
Review and advise on NIJ’s dissemination strategy.
Review and advise on NIJ’s quality assurance activities.
Review all grants and contracts exceeding $1 million prior to their award.
Biannually prepare a report to the attorney general on NIJ’s progress.
Given the powers and responsibilities described above, the committee suggests in Box 7-1 an array of topics that might conceivably be included in meeting agendas.
Since 1979, the NIJ director has been a presidential appointee. But the original legislation says little about the requirements for that position, does not set terms, and does not provide for a careful transition from one director to another. Traditionally this role has been seen as one requiring familiarity with legal institutions, crime trends, crime policy, and the criminal justice system but not experience in directing or conducting crime and
Possible NIJ Advisory Board Meeting Topics
justice research. In short, there has never been a premium placed on having as the director of NIJ a criminal justice researcher and in its 40-year history no NIJ director can be described as such.
Successful research agencies are characterized by leaders who are recognized scientists in their field and staff who are researchers with experience and expertise in areas of their specialization. Success in the application of the science or in related areas cannot substitute for scientific accomplishments. In situations in which the research unit is in a parent organization dealing with intense political issues, terms of appointment that are fixed and that minimize the influence of political factors in retention decisions are critical (e.g., a fixed term of appointment that overlaps presidential elections).
This is in sharp contrast to other federal research agencies, such as the Institute of Education Sciences (IES), in which the director is appointed for a 6-year term designed to overlap presidential administrations. The most common terms of office are 4- and 6-year terms. The director of NSF also has a 6-year term. The head of Congress’s Office of Technology Assessment has a 6-year term, and the head of the U.S. Government Accountability Office has a 12-year term. Terms of office do contribute to a longer period of sustained professional management and decrease the likelihood of shifting priorities and interests that a new director might bring to the job.
The NRC has conducted numerous assessments of federal research programs similar to this one (e.g., National Research Council, 1992b, 2000, 2008c). These assessments have been unequivocal about the need for a top science official. In a study of EPA’s Office of Research Development, the report noted “the lack of a top science official is a formula for weak scientific performance in the agency and poor scientific credibility outside the agency” (National Research Council, 2000, p. 5). The leaders of the NIH institutes, the NSF directorates and divisions, and IES are all drawn from the scientific community. In the case of the latter, this requirement is legislatively mandated and calls for the director to be “[a] highly qualified authorit[y] in the fields of scientifically valid research, statistics or evaluation in education as well as management within such areas” (U.S. House of Representatives, 2002).
For these reasons, we recommend that Congress should set science requirements for the director and a renewable fixed term of service of 6 years. An appropriate director for NIJ shall have had experience in directing crime and justice research, be recognized as a highly qualified authority in the fields of crime and justice research (including evaluation research), and have demonstrated success in managing substantial crime and justice research efforts. A term of 6 years means that appointments will overlap presidential elections but allows for change in leadership as the requirements of the position may change. Also, during the most recent congres-
sional reorganization of a major research operation (DoEd), a 6-year term was deemed preferable.
Grant Sign-off, Budget, and Staffing Authority
NIJ’s authorizing legislation provided for sign-off authority for grants. The committee strongly believes that this authority be retained. Earlier in the report, we note that this authority has not always been recognized by the leadership of OJP. The committee reiterates the point that this authority is essential for a science agency, and any attempts to weaken or eliminate it deserve immediate attention and action.
The committee also stresses the importance of giving NIJ the authority to present its budget to DOJ and for NIJ to have its own budget line item in the departmental budget that is considered by the Office of Management and Budget (OMB) and congressional appropriation committees. Unlike other science agencies, NIJ does not have its budget separately reviewed or discussed by congressional appropriators. As a result it is constrained in its ability to advocate for its program directly with Congress. Reaching out to legislative and funding bodies to describe its research activities and useful findings should be part of any federal research agency’s advocacy strategy. This is a change from former practice. In 2003, the House appropriations subcommittee noted that DOJ had merged seven different accounts into one larger “justice assistance” heading and requested that the former accounting system be reinstated (U.S. House of Representatives, 2003, p. 36). This attempt to consolidate OJP funding into a single pool has continued in each subsequent year. Regarding submissions for fiscal year (FY) 2009, Senate appropriators commented that “the Committee again rejects the Department’s proposed merger of all OJP programs under this heading and instead has maintained the previous account structure” (S. Report 110-397, p. 64, as cited in National Research Council, 2009a, p. 235). During the FY 2009 appropriations hearings, the congressional appropriators focused their attention on OJP’s request to consolidate state and local law enforcement assistance programs. Despite numerous references by the acting OJP administrator regarding the utility of research and data, neither the NIJ budget nor that of the Bureau of Justice Statistics came up for discussion (Consortium of Social Science Associations, 2008).
The committee recommends that NIJ should be given authority for staffing, including staffing allocations and independent hiring authority. One way to exert political interference is to control the numbers of authorized staff, grade levels, and the ability to fill vacancies. Allocations are currently made across six OJP programmatic offices, and NIJ must compete with the others for them. Prior to 1979, NIJ had this authority; it also handled the substantive aspects of the recruitment and hiring process,
including candidate screening and selection. In the committee’s view, this model should be restored. NIJ needs to strengthen the scientific and technical qualifications of its staff. By having greater authority for its staffing, including recruitment, NIJ will have more flexibility to determine job and position series, grade levels, and other staffing options including fellowships to attract and hire talented and scientifically trained people. NIJ needs staff who can manage awards but also who are recognized as leaders in their areas of concentration and have the requisite scientific and technical ability to develop, manage, and assess research activities.
STRENGTHEN THE SCIENCE MISSION
RECOMMENDATION 2: To strengthen its science mission, the National Institute of Justice (NIJ) should direct its efforts toward building a body of cumulative knowledge that will assist the criminal justice field in its effort to prevent and control crime and improve the criminal justice system; sponsoring research that will improve and upgrade current scientific methods used to study crime; and supporting new areas that have heretofore been neglected due to NIJ’s incapacity to commit resources required to support projects of long duration, great complexity, and substantial expense. To improve NIJ’s ability to support research, the committee recommends that Congress remove responsibility for forensic capacity-building programs and reinstate them in other U.S. Department of Justice and Office of Justice Programs agencies, such as the Bureau of Justice Assistance and the Community Oriented Policing Services office, that have a clearly defined technical assistance mission, are closely linked to state and local criminal justice agencies, and have larger financial reserves to draw on.
NIJ has succeeded in developing a body of knowledge in important areas that are critical to preventing and controlling crime and improving the administration of justice. For the most part, however, these efforts have been heavily dependent on congressionally mandated programs or the transfer of funds from other OJP or DOJ offices. These requirements have been made without a strong science foundation.
This conclusion is particularly true of NIJ’s evaluation research portfolio. Part of its mission is to determine if proposed criminal justice innovations work. Ideally, such innovations would have a strong research base, but innovations are frequently prompted by policy and operational determinations that are only minimally derived from strong research. The extreme diversity of the ORE outcome evaluations—in topic, theory, focus, and method—points to a lack of programmatic focus on systematic knowledge building or problem solving. One reason for the lack of programmatic
focus is that NIJ frequently has had limited control over decisions of what programs to evaluate. It is often placed in the role of evaluating programs selected and funded by other offices in OJP or programs adopted by localities that do not have adequate (or any) preliminary evidence of effectiveness or grounding in scientific theory. A second and related reason has been NIJ’s failure to engage in long-term strategic planning.
NIJ-funded programs designated by earmark or congressional mandate are at best minimally related to research and detract from its research mission. In its discussion of the OST program, the committee differentiates between what it deems appropriate for a research agency (e.g., technology development) and what is inappropriate (e.g., the forensic laboratory capacity-building and forensic training activities) such as the Paul Coverdell, DNA Backlog Reduction, Solving Cold Cases, and Forensic DNA Unit Efficiency Improvement programs. These are awards to provide direct assistance to individual agencies and have nothing to do with NIJ’s research function. They now represent more than half of the budget of OST. The research mission of NIJ is being overwhelmed by these activities, and its standing as a science agency has been negatively affected. If these essentially formula grant programs are removed from NIJ, then NIJ can more effectively focus on its research mission.
As a science agency, NIJ should play a central role in efforts to develop research-based, effective strategies for preventing and treating criminal behavior in the United States in the 21st century. Its primary mission is not to evaluate OJP programs and in the process neglect the fostering of a robust program of more “basic” kinds of research. Basic research as well as applied research offers the possibility of clarifying the nature of crime and documenting aspects of the justice system. Nor is NIJ’s primary mission to advise OJP and DOJ on policy, build the capacity of line agencies, or disseminate information that is not science based. As a science agency, NIJ should set priorities for its work that are based on an accurate assessment of the research needs and possibilities at any point in time. NIJ should take an active role in defining the types of research that make sense. Among the possibilities are interdisciplinary efforts with other federal research agencies whose work has implications for solving crime and justice problems. In the description of NIJ’s research portfolio, the committee notes several examples of interagency cooperation in Chapter 3, and we think that such efforts should be encouraged and facilitated. As part of its science role, NIJ should also work with other program offices within OJP and DOJ to advise them on their program development activities, develop a strategy for identifying appropriate programs for evaluation, inform them of the policy and practice implications of their research, and identify the research needs of the field. To play this central role will require the agency to make longer term commitments of funds and staff to solving problems in specific
areas of criminal justice practice, to engage more actively with the research community in selecting priority areas and testing the feasibility of ideas as they develop, and to make multiyear commitments to researchers to work through the development process. This will entail significant organizational commitment to the goals and development of management and practices that support the process.
The committee was charged with recommending a research agenda for NIJ. Our approach to this task is to note the many recommendations that other NRC panels have made to guide the NIJ research agenda (see Appendix E), and to emphasize the need for a science-based process to set agendas, select awardees, and monitor performance. We have not specified a specific research agenda because we became convinced that proper governance and transparent processes should be established first within the agency in order to set the agenda and to resolve such issues as the levels of support for basic and applied research and for social science research and technology development. An NIJ with the autonomy and type of leadership we propose will use a strong advisory board and the many research recommendations it has already received to set both its long- and short-term agenda and priorities.
The NIJ advisory board should be one component of a broader advisory infrastructure that provides for outside scientific advice and feedback for particular programs of research as well as individual large-scale grants, as appropriate to the role of a research agency. NIJ should have a program of research to improve scientific methods in crime and justice research, including work on how to make sure experimental designs, longitudinal studies, and long-term observational research are appropriately supported.
NIJ should be provided with the authority and resources necessary to devote sustained attention to more long-term research activities appropriate for a research institute. In 1977, the NRC Committee on Research on Law Enforcement and Criminal Justice concluded that extensive knowledge about crime and criminal behaviors and the operation of the criminal justice system was necessary in order to develop fair and effective governmental responses. The need to identify questions that will advance theory, research methods, science, and practices around important issues—such as crime control theory, the etiology of criminal behavior, the factors that influence desistance from criminal behavior, and the emergence of new opportunities for crime in the face of social and technological change—remains as strong today as it was 30 years ago. NIJ will also be in a better position to justify and support research to improve scientific methods and other kinds of studies, such as surveys and longitudinal studies that have been neglected because of their duration, complexity, and expense.
BOLSTER THE RESEARCH INFRASTRUCTURE
RECOMMENDATION 3: The National Institute of Justice should undertake efforts to nurture and grow the pool of researchers involved in criminal justice research as well as activities that support the research endeavor itself. These efforts should include increasing the resources devoted to supporting graduate education for persons pursuing a career in criminology and criminal justice studies and other disciplines engaged in research and teaching on criminal justice topics, such as the Graduate Research Fellowship Program and the W.E.B. Du Bois Program, and enhancing the Data Archive Program.
Improving research on crime and justice will require more and better trained researchers in all of the sciences on which NIJ depends, most especially criminology and other social sciences and the forensic sciences. NIJ’s support for graduate students and rising academics has been effective but modest. In the past three decades, NIJ has developed and sustained a number of fellowship programs. The recipients of NIJ doctoral and young faculty fellowships have made scholarly contributions to the criminal justice literature or to NIJ’s research programs and many have remained in the criminal justice field.
Currently, NIJ lacks the resources or administrative oversight to effectively expand these programs. The committee observes that, over the years, there has been inaccurate documentation of basic and relevant information regarding the fellowship recipients. In addition, no external formal assessments of NIJ’s fellowship programs have been conducted to date. We make suggestions for enhancement of these efforts in Chapter 5.
Similarly, NIJ has operated a very effective program of archiving and disseminating data needed for crime and justice research. It has been a leader at the federal level in recognizing the importance of collecting, archiving, and making available to other researchers the data collected as a part of the projects it funds. Since 1981, NIJ has included some kind of special requirement in its research awards for grantees to submit the data collected. But too many grantees continue to ignore such requirements and NIJ has failed to monitor compliance with its requirements that grantees submit their data sets or risk nonpayment of funds or to develop a strategy that would provide the necessary support to produce quality data sets to the National Archive of Criminal Justice Data. Again, Chapter 5 outlines steps that could be taken to increase the number and quality of data submissions.
ENHANCE THE SCIENTIFIC INTEGRITY AND TRANSPARENCY OF RESEARCH OPERATIONS
RECOMMENDATION 4: The National Institute of Justice (NIJ) should revise its research operations to allow for greater transparency, consistency, timeliness, and appropriate involvement of the research and practitioner communities. In particular, NIJ should make information about its research operations and activities publicly available, easily understood, and consistent with the highest standards found in other high-quality federal research agencies
Recommendation 1 reflects our view that NIJ does not have sufficient independence, authority, or resources to fulfill its mission. We think it will take congressional and executive action to change this. But the quality of key research agency activities, such as agenda setting and making awards, does not depend on the organizational location of a research agency or the size of its budget but on good leadership. NIJ does not have proper R&D management processes in place to ensure that good planning occurs, that all research proposals are subject to high-quality peer review, that fair and impartial funding decisions are made, that reports receive careful assessments, and that the agency is doing everything it can to ensure that its research operations are of the highest quality and in keeping with its mission. In addition, NIJ’s efforts to involve researchers and practitioners in an advisory capacity have not always been consistent or appropriate, given their expertise and training.
With regard to how NIJ can improve the quality of its research operations and enhance the trust and confidence of the research community, a central element is greater transparency in processes and decisions. Current levels of transparency are not sufficient and contribute to the opinions noted in our survey results that NIJ decisions are not made on the basis of scientific criteria. From early announcement of award cycles, to greater information on proposal reviews and decisions, to improved easily available data on awards and award completion, NIJ will need to be better understood by the research and practitioner communities.
Improving Research Operations
Agenda Setting and Planning
NIJ lacks a formal and centralized strategic planning process. Agenda setting and planning are handled differently by ORE and OST, with neither well understood by its own staff or by the broader research community.
NIJ should revise its program planning and grant selection processes in ways that reflect adherence to established criteria and well-established scientific practices. The committee urges the new NIJ to establish a clear and timely planning process that includes the following elements:
A structured agenda-setting process with full participation of staff and broad outreach to researchers in the planning process for both social science and technology research.
Engagement of practitioners in the planning process to identify needs and to build support for the conduct of research as well as the adoption of findings by criminal justice agencies.
More open, transparent, and collaborative approach to identifying areas and topics for long-term funding.
Development of a strategic plan that identifies these areas and topics.
Making the strategic plan publicly available in a timely way.
Consistent and extensive oversight by external, independent, scientific reviewers.
There are numerous models of agenda setting employed by other federal research agencies that NIJ can turn to for guidance. The individual institutes of NIH engage in a very structured process that is well documented, involves the full participation of staff, and engages the research community through their participation as members of standing advisory committees and as experts in particular areas. Although there is considerable variety in the specifics, the overall process is similar and is designed to result in a strategic plan. The purpose of the strategic plan is to announce to researchers the near-term priorities—priorities that focus on “the greatest need and promise for scientific advancement” (National Institute on Alcohol Abuse and Alcoholism, 2006).
Several NIH institute division directors met with the committee, shared information about their planning processes, and described the various roles of senior leadership, program staff, and researchers. Detailed written information was also received from the Division of Epidemiology and Prevention Research (DEPR) at the National Institute on Alcohol Abuse and Alcoholism (NIAAA) that proved very helpful in fleshing out the particulars of its planning experience (see Box 7-2).
Two things stand out about the DEPR process. There was broad input from the science field at every stage, including the final review meeting of NIAAA’s Extramural Advisory Council. Ad hoc experts from several federal agencies, universities, research centers, and medical centers attended this meeting, commented on the priorities, and responded to questions. There
Planning at the Division of Epidemiology and Prevention Research
One of four divisions of NIAAA, the Division of Epidemiology and Prevention Research was newly created in 2004. Its responsibilities are similar to those of NIJ in that it plans, conducts, and supports epidemiological studies; it supports clinical programs; it analyzes ecological and situational factors related to its particular subject area; it collaborates with outside organizations; it collaborates on studies with a broad array of organizations; it sponsors scientific conferences; and it supports a wide range of grants, including research and research training.
As part of its 2-year strategic planning process, division staff developed a mission statement; identified critical issues; reported on trends and related areas impacting on the problem being studied; reviewed relevant research findings; analyzed its current portfolio; described pertinent methodological issues; developed contextual background information and information on other relevant initiatives sponsored by the broader agency; and undertook a financial analysis of funding allocations within the division. This planning process culminated with the adoption of its strategic plan that received approval by the NIAAA National Advisory Council.
was also full documentation of the process, of the analyses that had been undertaken, and of decisions reached. Upon review by NIAAA’s National Advisory Council in February 2007, the strategic plan and all written materials were posted on the NIAAA website, where it remains today.
In comparison to the DEPR description of a formal process, NIJ’s agenda planning activities have varied over time and, particularly in the case of ORE, are not well developed or formalized. NIJ should take immediate steps to develop a formal, unified, and structured planning process. It should include broad participation of the research community at various stages and in different capacities. NIJ should base its agenda on a thorough understanding of the state of the art, an analysis of its own past and current research portfolio, and extensive consultation with the science community. Only by having such a process is NIJ likely to encourage creative and innovative thinking about both research issues and methodologies.
Signaling of Research Solicitations
NIJ has not been consistent in signaling the research community about its solicitations and proposed submission dates. NIJ should provide more predictable levels and timing of funding and as much advance notice as possible about solicitations. Other federal research agencies, such as NSF, provide notice of solicitations well in advance of their deadline dates. In the case of the Law and Science Program at NSF, submission dates occur semiannually at the same time of the year. NIJ should aim for regularly timed solicitation cycles. This makes it much easier for applicants to plan around these deadlines and, in the event they apply and are not successful, to have sufficient time to revise their proposal and resubmit for the next round of funding.
Peer Review and Grant Selection Process
Much has been written about peer review and the requisites of a high-quality scientific peer-review system (National Research Council, 1992b, 2000; U.S. General Accounting Office, 1999; U.S. Office of Management and Budget, 2004). In a review of peer-review processes being used by agencies that fund education research, NRC identified the following attributes (National Research Council, 2004d):
flexibility in designing peer-review systems to meet the individual needs of an agency;
independence of the project officer from the peer-review process;
nonparticipation of practitioners in assessment of research expertise; if included, their role is to assess relevance, significance, applicability, and impact of the proposal;
a strong infrastructure: knowledgeable staff, systems for managing the logistics of peer review, technologies to support review and discussion of proposals, clear mechanisms for providing feedback, and standing panels when research priorities are clearly stable;
predictable levels and timing of funding for the process itself;
clearly delineated scoring systems and the meaning of the scoring levels defined and understood;
training on scoring for reviewers as well as training on what the peer-review process entails and how to be an effective reviewer;
qualified peer reviewers who have the research experience and expertise to judge the theoretical and technical merits of the proposals they review; and
systematic evaluation of peer review based on intended outcomes and the public reporting of the evaluation.
Measured against these attributes, the NIJ peer-review process is very weak. The committee notes the current practice of having both researchers and practitioners serve as peer reviewers and assess proposals using the same criteria. We view this as highly inappropriate given that the purpose of peer review is to assess scientific merit and arrive at consensus.
NIJ’s ability to adapt or change its peer-review process to meet its own needs is constrained because peer-review services are managed and overseen centrally under OJP. Scoring systems are not understood and do not appear to have a scientific basis. NIJ has not undertaken a systematic evaluation of its peer-review processes. Having a good scientific peer review and grant selection process in place is a critical requirement for a high-quality research program. In the federal government, although peer review is used for many purposes, its primary use is to assess the scientific merit of competitive and noncompetitive proposals. There is no written government-wide definition of peer review, but there is general acceptance that it is an “independent assessment of the technical, scientific merit of research by peers who are scientists in knowledge and expertise equal to that of the researchers whose work they review” (U.S. General Accounting Office, 1999).
Proper peer review is integral to the work and reputation of a science agency. NIJ’s authority to conduct and manage its own peer review should be clearly established and protected. NIJ should follow standards of practice for peer review reflective of a science agency. NSF and NIH are excellent models of scientific peer review. These models have been in place for many years, and although the NIH peer-review process has recently undergone a major overhaul to improve what was perceived as its shortcomings, it still retains the basic infrastructure and approach (see Box 7-3).
Grant Award Process
During the course of our study, four different persons served as the final arbiter of grant award decisions at NIJ, and it is possible that each one used different criteria and a different in-house process to make the decision. NIJ staff themselves seemed unclear as to how decisions were made. According to the perceptions of Congress reflected in recent appropriation bills and the majority of researchers who responded to the committee’s survey, the NIJ process appears to be flawed.
A research institute’s reputation is based not only on the quality of its work but also on the perception that its grant award process is fair, consistent, and transparent. NIJ needs to reestablish its credibility by signaling clearly to the research community as much information as it can regarding its funding, the amount it anticipates providing, the selection criteria it will use, and the role of the various participants in the process—in short, the basis on which research support is being provided. Documenting and
The NIH Peer-Review Model
Responsibility for peer review across NIH resides with the Center for Scientific Review (CSR), an organization under the Office of the Director of NIH. CSR is usually the initial point of contact for applications, and it also manages the review process for most of the grant applications from the 27 NIH institutes. For grant applications, there is a two-step review process: (1) a review by a Scientific Review Group (SRG) under CSR’s direction and then (2) assignment to at least one NIH institute for potential funding. This SRG usually contains 15-18 scientific experts who meet three times a year. One of the members, designated as the chair, leads the discussion. The initial or first-level review evaluates the scientific and technical merits of the application.
The second review is done by the national advisory council for the particular institute. These councils are composed of 12 or more members drawn from the scientific and lay communities. They review the initial judgments of the proposals and make recommendations for funding in light of relevance to the program and to the institute’s overall mission. In practice, the councils make few changes in the priority rankings from the initial review. Sometimes applications for funding bypass the CSR review process and are submitted directly to the individual institutes, where they undergo review by standing committees that are formed for a very specific purpose. In this case, they fall under an institute’s individual Office of External Research. Once through this stage of review, they go on to the particular national advisory council of that institute for a funding decision. It is important to note that this process involves not only grants and cooperative agreements but also contracts.
NIH is currently in the implementation phase of a multiyear process to improve its peer-review process. The goals of this effort are to engage the best reviewers; to improve the quality and transparency of review; to ensure balanced and fair reviews across scientific fields and career stages and reduce administrative burden; and to provide for a continuous review of peer review. It is an activity that NIJ would do well to emulate as it seeks improvements to its own peer-review process.
The NSF Model
Peer review is conducted differently at NSF than at NIH but it reflects the same core values of scientific competence, fairness, timeliness, and integrity. The division within NSF that most closely approximates NIJ in
terms of the kinds of research it sponsors is the Law and Social Science program, one of 16 individual programs with the Social and Economic Sciences (SES). The SES is located within SBE, one of seven Directorates at NSF.
Proposals received by the NSF Proposal Processing Unit within the Division of Grants and Agreements are assigned to the appropriate NSF program for acknowledgement and, if they meet NSF requirements, for review. All proposals are carefully reviewed by a scientist, engineer, or educator serving as an NSF Program Officer, and usually by three to ten other persons outside NSF who are experts in the particular fields represented by the proposal. Proposers are invited to suggest names of persons they believe are especially well qualified to review the proposal and/or persons they would prefer not review the proposal. These suggestions may serve as one source in the reviewer selection process at the program officer’s discretion. Program Officers may also obtain comments from assembled review panels or from site visits before recommending final action on proposals. Senior NSF staff further review recommendations for awards. Each Division has its own Advisory Panel that is comprised of scholars from multiple disciplines (anthropology, criminology, economics, political science, psychology, sociology). In the case of the Law and Social Science Advisory Panel, eleven members serve a 2-year rotation and provide written reviews that supplement the external reviews. The Panel meets to discuss proposals and propose the ranking and within that a further breakdown for actual funding. It provides advice as to whether the proposal is competitive. The program officer makes the recommendation to the Division Director as to which ones to recommend and can disagree with the Panel’s guidance. Normally final programmatic approval occurs at the division level. NSF relies on the judgment of program officers to make funding recommendations that address NSF strategic goals.
One important way in which the NSF peer-review system differs from that of the NIH is the amount of discretion the program officers have. Peer review at NSF still receives high marks because of other features it has and its overall transparency. For example, all applicants have access to an electronic system called FastLane in which they can track the progress of their applications and can view the individual peer reviews and scores as well as any summary reports, site visits, or other documents that constitute their application file. Second, NSF program officers are highly qualified researchers and academicians who see their role as encouraging good research that is responsive to NSF priorities and who work with applicants to shape them accordingly. Staff also communicate with applicants in a timely way regarding their submission and provide useful feedback that often allows unsuccessful applicants to revise pro
posals in time for the next round of applications. Third, the system is perceived as responsive to researcher needs. Detailed information on the process is available to any applicant on the NSF website. The NSF electronic system can accommodate changes in an application prior to the final submission date. Submission dates for many programs are set in advance, frequently on a semiannual schedule. Finally, there is a check on peer review in the form of program reviews that are undertaken periodically by the Committee of Visitors. This quality assurance process examines technical as well as managerial oversight of the program. In doing so, it holds program officers’ accountable for their recommendations but also serves as check on the quality of reviews, the basis for the program officers’ recommendations.
maintaining consistency in the award decision process is an important first step to rebuilding trust and confidence in NIJ.
Although NIJ regards the dissemination of research findings as a part of its mission, the review process that supports its own dissemination process is very weak. There seems to be a lack of clarity about what NIJ expects from its grantees regarding final reports. Like agenda setting and peer review, the report review process is not well understood by researchers, who are critical of the amount of time it takes for NIJ to release research reports or research findings publicly.
This situation prevails despite the fact that interest in making scientific research available to the public has never been greater. Recently, particular attention has focused on the ability of scientists working in the federal government to communicate with the media and the public. The Union of Concerned Scientists, a public interest advocacy group that investigates the extent to which scientists have been thwarted in their efforts to release their findings, has been very involved in bringing to the public’s attention various cases in which research findings had been suppressed or altered (Union of Concerned Scientists, 2007). It also proposes a model media policy that outlines responsibilities for public affairs officers as well as researchers and other staff. This policy provides that employees have the right to review, approve, and comment publicly on the final version of any proposed publication that significantly relies on their research, identifies them as an author or contributor, or purposes to represent their scientific opinion (Union of
Concerned Scientists, 2008). GAO has also assessed several research agencies’ practices regarding release of information and recommended that policies be in place to guide researchers and public affairs officials in their efforts to disseminate research to the scientific community (U.S. Government Accountability Office, 2007).
Our examination of other federal research agencies indicates that they either have complete authority over reports issued in their name or they rely on researchers publishing independently. NIJ needs to improve its research report review process as well as to support efforts of researchers to publish in peer-reviewed journals.
A common theme running through our assessment of NIJ’s research operations is the importance of transparency. Transparency implies openness, communication, and accountability. It means that the decisions of research agencies regarding agendas, award decisions, and impact (in the form of research findings and reports) should be apparent to the research community and the public. In our examination of NIJ’s research operations and comparison with the practices of other federal research agencies, the committee found a lack of transparency, which is reflected in the limitations of the information made available to grant applicants, grant awardees, and the public. It is also reflected in our interviews with NIJ staff, in which many of them explained that they did not know how funding decisions are made or the basis for them. The lack of transparency and quality of information were apparent to the committee as we attempted to access directly or be provided with certain information for this study. Despite the facts that our study meets the necessary privacy and confidentiality requirements and it was requested by NIJ itself, our efforts to gather data were often unsuccessful, either because the data did not exist, they existed in an unusable form, or barriers to obtaining the data could not be overcome by NIJ staff or leadership.
Several members of Congress have voiced concerns about the lack of transparency, raising questions about conflicts of interest and questionable practices. As mentioned in Chapter 4, the most severe criticism has come from the Department of Justice’s Office of Inspector General, which concluded that NIJ’s failure to document key aspects of its preselection and grant award process called into question the fairness and openness of its competitive grant-making process (U.S. Department of Justice, 2009b).
The lack of transparency is also reflected in the views of criminal justice researchers we surveyed. In the absence of information and feedback on how decisions are made and the criteria that are used to reach them, it is reasonable to attribute these decisions to political considerations.
In order to achieve transparency, a research organization needs the capability, authority, and resources to maintain records of its own activities that are accurate, easily retrievable, and accessible to the public within limits of the law. The committee acknowledges that NIJ is very dependent on OJP’s centralized grant management information system and the resources provided by the Office of the Chief Information Officer. However, NIJ should take responsibility for documenting its decision-making processes, and in instances in which the information is not being generated by OJP, it should develop its own documentation and records. At a minimum these records should include
Information on the life cycle of a grant, from award to final report and resultant publications. At a minimum, titles, grant amounts, principal investigators, and grant abstracts should be a part of this information.
Information on individuals who have received funding and their research records, products, citations, and professional recognition.
Financial information for all NIJ organizational units that is consistent and includes appropriated funds and expenditures and breakdowns between various categories of restricted and discretionary funds, and program topics. This information should be compiled in a way that it can be used to present a historical picture of funding.
Information on new staff hires and vacancies that indicates grade level, position, and education levels. Such information should be compiled annually.
Involving the Research and Practitioner Communities
Another theme throughout our deliberations on NIJ’s research operations is the need to clarify the important but separate roles that the research and practitioner communities should play in the research program. These roles are reflected in the proposed composition of the NIJ advisory board, in the qualifications of the NIJ director, and in the improvements that the committee recommends regarding agenda setting and planning, peer review, and report review.
NIJ should increase its efforts to involve researchers and to seek their advice in the development, implementation, and assessment of its research activities. Their advice is critical to identifying and shaping the kind of science needed to accumulate enough knowledge to answer critical policy questions. The question “What works?” requires a multilayered response and a probing into what led to success as well as what led to failure. Researchers are uniquely trained to pose these questions and seek the answers.
Practitioners also have an important role to play and through their training and expertise can provide broad policy direction to address research concerns. As consumers of the research, practitioners can also advise as to its need and relevance.
ESTABLISH A CULTURE OF SELF-ASSESSMENT
RECOMMENDATION 5: The National Institute of Justice should measure the influence of its programs on research and practice and assess the quality of operations and program-level technical and managerial matters.
NIJ’s efforts to promote or conduct assessment activities have been extremely limited. With the exception of the 1977 NRC study and this current one, there has been no other independent review of its body of work. Research on particular topics has been discussed as part of more general substantive NRC reviews (see Appendix E), but, with the exception of the NRC report on violence against women (National Research Council, 1996b, 2004a), NIJ’s own research has generally not been singled out. The committee saw little evidence of regularly scheduled systematic reviews of the ongoing research portfolio as part of a planning process or ongoing monitoring activities.
Ensuring quality in a research organization through careful monitoring is a particularly complex undertaking. It needs to take into account the quality of the research as well as the many different processes that support the research enterprise. At its highest level, assessment activities can be directed to assessing the broad influence of a program or its major achievements or to more narrow programmatic goals or benchmarks. They can include assessments of various processes and of the products themselves as well as customer satisfaction. They can also include assessment of staff performance.
NIJ will not be able to conduct these kinds of assessments without better records and a willingness to provide access. Throughout this report, we note difficulties in obtaining documentation of research operations as well as other kinds of administrative information. Some of the problem lies with the centralization of records under OJP, which is not tailored to the needs of NIJ. NIJ should take responsibility for creating record systems that will allow for detailed analyses of administrative and program funding, administrative and personnel matters, and all programmatic activities. These records need to be consistent across all NIJ offices. Most critically, it should create a basic mechanism to track usage and influence of funded research in scholarship and practice.
Considering how other federal research agencies address the issue of
assessment is instructive. First, many research agencies struggle with defining quality and then constructing a process to assess it. This is especially true in assessing the impact of research or research outcomes as a measure of quality (National Research Council, 2008b). Second, scientists outside the agency are heavily relied on, either individually or as peer reviewers, to assess quality. Peer review has many other purposes than proposal review. Many federal research agencies rely on individual peer reviewers or peer-review panels to set priorities, develop programs, and even conduct personnel evaluations. Third, individual processes, such as peer review of proposals and the report review process, should include an evaluation component that provides useful feedback on efficiency and quality. Like other well-managed federal agencies, NIJ should establish self-assessment as an ongoing activity and use it to constantly improve its operations. Furthermore, to ensure transparency, it should make the results of such assessments publicly available. This is unlikely to happen, however, unless our recommendations on leadership and independence are implemented.
NIJ is the only federal agency devoted to crime and justice research. Despite the problems noted in this report, it has accomplished much in its short history. Our assessment of NIJ supports its continuation and growth, but with legislation and actions by the executive and congressional branches that will allow it to fulfill its role as the nation’s center for improvements in crime control and prevention. When those changes happen, we are confident that NIJ’s self-assessments and periodic external assessments will describe an agency that is science driven and that is making even greater contributions to the understanding of crime and the improvement of justice.