Skip to main content

Currently Skimming:

3 Field Evaluation Experiences in Other Areas
Pages 27-48

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.


From page 27...
... . 1 The extent to which randomized controlled trials should direct education policy and practice has been the source of considerable debate, particularly if it excludes other research methods.
From page 28...
... But I think there may be some lessons learned in other federal agencies about the production and use of research that could be applicable to intelligence." Whitehurst, now a senior fellow of government studies and director of the Brown Center on Education Policy at the Brookings Institution in Washington, began his presentation by commenting that education has a history of not knowing much about what it is doing. In 1971, for example, a RAND Corporation group set out to discover which educational practices were well supported by research and came to the conclusion that there was essentially no evidence to show that any educational methods worked.
From page 29...
... Because the institute was given a degree of independence that its predecessor agency did not have, it was able to attract knowledgeable people to perform good research and protect that research from outside pressures. There was also more money provided for randomized controlled trials.
From page 30...
... In most police departments today, Lum said, decisions about policing practices and policies are not based on scientific evidence but instead are made on a case-by-case basis, using personal experience, anecdotes, even bar stories as a guide. The individuals making the decisions fall back on their own judgment, guesses, hunches, feelings, and whims, while being influenced by various outside forces, such as political pressure, lobbying
From page 31...
... For example, very few of the new technologies being used in policing -- things like license plate recognition technology, crime mapping, mobile computer terminals, and DNA field testing -- have been evaluated for effectiveness. They are tested to see if they work as they are supposed to -- to determine, for instance, if the license plate recognition systems actually do recognize license plates accurately -- but almost nothing has been done anywhere to test if these technologies are effective in, say, reducing crime rates.
From page 32...
... Finally, shifts in police culture have been vital in developing and using this research infrastructure. Not only are an increasing number of police chiefs and officers recognizing the value of science and research to their jobs, but police research groups, such as the Police Foundation and the Police Executive Research Forum, are receptive to working with academic criminologists.
From page 33...
... Furthermore, Lum said, she has found that the leadership culture in counterterrorism is not focused on science or on the role that science could play in counterterrorism. Nor are there any third parties that can play the kind of role that the Police Foundation or the Police Executive Research Forum plays in advancing the evaluation of policing practices.
From page 34...
... It was then necessary to compare the measure of mental disorder derived from the survey responses with the measure of mental distress derived from standard clinical interviews. In particular, psycholo gists use the Structured Clinical Interview for DSM-IV disorders, or SCID, to detect the presence of a clinically significant mental disorder; the researchers needed to calibrate their survey responses with the SCID responses so that they could be sure they were compatible.
From page 35...
... For example, the answers that people give to survey questions will often vary depending on whether the questions and answers are given orally or in written form. "People respond differently to sensitive items about their mental health if they are being asked by a grandmotherly type sit ting across the table versus if they are able to answer the questions on a computer or in some way that is a little more discreet." Colpe also described an international collaboration, the World Mental Health Survey Initiative, carried out by researchers in 28 countries located in all the different regions of the world and surveying a total of more than 200,000 people.
From page 36...
... legal system -- the use of eyewitness identifications. Christian Meissner, an associate professor of psychology and criminal justice at the University of Texas at El Paso, provided an introduction to the general field of psychology and law, which includes the specific topic of eyewitness recall identification, and discussed how research has helped lead to reforms of the legal system.
From page 37...
... A variety of studies were carried out to determine the various causes of wrongful conviction, Meissner said, and they have found that mistaken eyewitness identification played a role in about 80 percent of the cases (see Figure 3-2)
From page 38...
... FIGURE 3-2 Contributing causes of wrongful convictions in the first 225 DNA exonerations. NOTE: Total is more than 100 percent because wrongful convictions can have Figure 3.2.eps more than one cause.
From page 39...
... Ultimately, Meissner said, this research has led to the development of a number of procedures to improve eyewitness identification. For example, interviewing protocols have been developed that dramatically increase the amount of correct information that witnesses recall without causing concomitant increases in errors.
From page 40...
... . Four years later, the NIJ published an accompanying training manual, Eyewitness Evidence: A Trainer's Manual for Law Enforcement (Technical Working Group for Eyewitness Evidence, 2003)
From page 41...
... A closely related field is organizational psychology, which is the study of people and groups at work. Its basic application is to make organizations more effective.
From page 42...
... Usability analy sis uses human factors principles to evaluate a system and tailor it to a user, and training evaluation typology uses principles from organizational psychology and human factors to ensure that a user is able to actually use the system or perform the task and has acquired the necessary competencies. Usability Analysis Salas then spent the next several minutes of his presentation describ ing usability analysis in greater detail.
From page 43...
... Training Evaluation The second basic approach to evaluating human factors products focuses not so much on the interaction of system and user as on the effectiveness of a particular training approach for a given system. Most train ing evaluations follow a five-level model called Kirkpatrick's typology, which has been in use since the 1950s (see Figure 3-3)
From page 44...
... Figure 3.3.eps It is the easiest and least expensive to collect, since it is basically reaction data pulled from people after a training session. "The problem is that several meta-analyses have shown the industry stops there," Salas said.
From page 45...
... "It is done basically to satisfy the bean counters." Despite the general weakness of the Level 5 evaluations, Salas said that people in the area of human factors and organizational psychology generally have a good sense of what is required for effective training evaluations. It is a very robust approach, it is systematic, it is very diag nostic, and it identifies which training works, which doesn't, and what can be done about it.
From page 46...
... "In counterterrorism, for example, there isn't that much evaluation research, but Christopher Koper (from the Police Executive Research Forum) and I suggest putting all the criminal justice studies into the Crime Prevention Matrix that we developed for policing and trying to glean from it some generalizations that might be applicable to counterterrorism," she said.
From page 47...
... Yes, Lum said, there was a time when the institute was pushing for randomized controlled experiments, and then there was a time when that was not the case. Recently has there been a return to more evidence-based practices, under the current administration, and evaluation is getting much more attention.


This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More information on Chapter Skim is available.