Strengthening the National Institute of Justice (2010) / Chapter Skim
Currently Skimming:
6 Assessing Research Programs
6 Assessing Research Programs
Pages 187-210
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 187... ...
Federal research agencies serve the role of sustaining lines of research in areas of ongoing national need, and they also have the ability to redirect support when opportunities arise in new directions for knowledge and societal benefit. Historically, the
|
From page 188... ...
. Its enactment addressed demands for accountability and demonstrated accomplishments by requiring that all federal agencies, including research agencies, develop multiyear strategic plans and evaluate and report annually on their activities.
|
From page 189... ...
Performance Measures The PART assessment for NIJ includes the following program performance measures: average days until closed status for delinquent grants; number of new final grant reports, research documents, and grantee research documents published; total number of electronic and hard copy documents/publications/other requested; number of fielded technologies; and number of citations of NIJ products in peer-reviewed journals. See Table 6-1 for the recorded measures for the period 2003-2008.
|
From page 190... ...
0 STRENGTHENING THE NATIONAL INSTITUTE OF JUSTICE TAbLE 6-1 Program Performance Measures as Reported in the PART by NIJ Year Target Actual Measure: Number of Fielded Technologies 2001 Baseline 5 2002 N/A 6 2003 N/A 5 2004 N/A 8 2005 N/A 15 2006 Baseline 26 2007 25 21 2008 26 17 2009 28 2010 32 2011 35 2012 37 2013 39 Measure: Number of Citations of NIJ Products in Peer-Reviewed Journals 2003 Baseline 54 2004 55 53 2005 60 65 2006 65 176 2007 70 96 2008 70 259 2009 70 2010 Discontinued Measure: Total Number of NIJ Electronic and Hard Copy Documents/Publications/Other Requested 2003 Baseline 5,416,579 2004 5,600,000 5,616,648 2005 5,850,000 7,327,961 2006 6,080,000 3,568,919 2007 6,310,000 3,070,622 2008 7,500,000 6,953,762 2009 4,000,000 2010 4,500,000 2011 4,500,000 2012 4,750,000 2013 4,750,000 Measure: Average Days Until Closed Status for Delinquent NIJ Grants 2003 Baseline 511 2004 400 275 2005 200 81 2006 90 80
|
From page 191... ...
. Average Days Until Closed Status for Delinquent NIJ Grants.
|
From page 192... ...
, the performance data item "number of fielded technologies" represents the NIJdeveloped technologies that are transferred to the field for use by criminal justice practitioners. The original measure may have been limited to counting the number of technology prototypes produced for counterterrorism,5 interoperable communications, computer crimes, and protective equipment; however, this technology transfer measure has since been broadened and now includes publications, demonstrations, examples of commercialized 5 Itis important to note that the FY 2006 target was reset as the baseline because of the phase-out of counterterrorism funds from NIJ to DHS.
|
From page 193... ...
From the committee's very limited review, it is fair to conclude that NIJ has initiated some work that is quite impressive7 and befitting its role to identify and support research with specific forensic and law enforcement applications. However, NIJ has also supported some work that appears to be less cutting-edge development or to have impact limited to a specific locale as opposed to the broader field.8 Although NIJ seems to be engaging in relevant work in transferring technologies to the field, its inclusion of so many different kinds of activities as fielded technologies is misleading and not a useful way of measuring program outcomes.
|
From page 194... ...
According to NIJ, the citation figures were obtained from different sources over the period 2003-2008. For example, in 2003-2005, the Social Science Citation Index was used; in 2006, Sociological Abstracts and Ebsco Academic Search Premier were used; in 2007, the Social Science Citation Index (SSCI)
|
From page 195... ...
Government Accountability Office, which assessed the methodological rigor of evaluations and the quality of program monitoring for specific programs during specific periods. These assessments do 1 represent efforts to routinely Figure 6- not R01756 and consistently collect data in order to assess the quality and impact of uneditable bitmapped image research programs.
|
From page 196... ...
referenced funding support from NIJ. In the forensic sciences journals, 75 of the 6,119 articles published (1.2 percent)
|
From page 197... ...
At first glance, this analysis indicates low visibility in these highly respected and established journals. The journals we selected undoubtedly present a limited sample, because some of NIJ's funded research may be published in other, more specialized outlets in the policing, courts, corrections, and forensic science areas or in other criminological/criminal justice/ forensic science journals.
|
From page 198... ...
Because of missing information and limitations in matching grant titles with data titles, we are unable to report verifiable statistics on the 10 ICPSR is the host to NACJD, as discussed in Chapter 5. They maintain an online Bibli ography of Data-related Literature, which is a searchable database that contains over 48,000 citations of known published and unpublished works resulting from analyses of data held in the ICPSR archive.
|
From page 199... ...
Although we cannot draw any conclusions from this citation analysis because of known limitations in the sample information and the absence of baseline information for comparison, we did, however, find the publicly available NACJD/ICPSR online bibliography to be a very useful tool. This searchable database has extensive information about each data set held in the archive, including a list of resulting literature.
|
From page 200... ...
The committee does not conclude that a potentially low documentation of NIJ-funded data or citation count indicates that NIJ was not influencing research or criminal justice policy. The committee had neither the information nor the resources necessary to compare data with other time periods, projected goals, or other agencies, so the committee is unable to draw conclusions.
|
From page 201... ...
The target practitioner sample consisted of leaders and key staff in well-known organizations with an interest in criminal justice issues.14 Despite the limitations of survey findings (presented below) , the survey results gave the committee insight into areas of relative strength and weakness in NIJ performance as well as into differences in the perceptions of researchers and practitioners and offer an otherwise unavailable window on the views of the field.
|
From page 202... ...
Those in our sample who never applied to NIJ for funding did not do so more often because they thought they were unlikely to get funded. Among the practitioner sample, satisfaction with NIJ performance is considered moderate in the areas of disseminating relevant research knowledge to practitioners and policy makers, identifying research and technology needs, and maintaining fairness and openness in practices.
|
From page 203... ...
Scientists whose primary focus is on hard sciences, such as forensic sciences or equipment development, may be less likely to join than social scientists. Some researchers do not choose to join or join other criminology professional groups.
|
From page 204... ...
Information and data in the form of metrics and performance measures 15 The committee received briefings from agency directors and program division directors of several federal research agencies (see Chapter 1)
|
From page 205... ...
Although space constraints preclude a listing of all assessment efforts of other research agencies, the committee does highlight some efforts made by the Division of Behavioral and Social Research of the National Institute on Aging and the National Science Foundation (NSF) that appear promising.16 16 The National Institutes of Health (NIH)
|
From page 206... ...
, which coordinates all NIH program performance activities, which include monitoring and assessing NIH-level program performance through several federally mandated reporting mechanisms. These mechanisms include GPRA, OMB PART, and the Performance and Accountability Report (PAR)
|
From page 207... ...
The review committee found that BSR has been highly responsive to the recommendations in a previous 2004 review report and has made excellent progress. The review committee felt that BSR has been substantially transformed in just 4 years with a number of notable accomplishments (National Institute on Aging, National Advisory Council on Aging, Division of Behavioral and Social Research Review Committee, 2009)
|
From page 208... ...
They are not easily understood by a broad audience or sufficiently accurate to be credible and have not been durable to remain relatively constant over the years. As such, they are not easily linked to its mission and strategic goals.
|
From page 209... ...
This in turn will help it determine appropriate metrics for measuring its impact. The adoption of thoughtful metrics or performance measures as part of a routine approach to assessing the progress of its scientific investments is essential for a strong, more viable research organization.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.