Field Evaluation in the Intelligence and Counterintelligence Context Workshop Summary (2010) / Chapter Skim
Currently Skimming:
2 Behavioral Tools and Techniques
2 Behavioral Tools and Techniques
Pages 9-26
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 9... ...
One common thread among all of these disparate techniques, a point made throughout the workshop, is that none of them has been subjected to a careful field evaluation. DECEPTION DETECTION People in the military, in law enforcement, and in the intelligence community regularly deal with people who deceive them.
|
From page 10... ...
Two in particular were described at the workshop: voice stress technologies and the Preliminary Credibility Assessment Screening System. Voice Stress Technologies Of the various devices that have been developed to help detect lies and deception, a great many fall in the category of voice stress technolo gies.
|
From page 11... ...
But the law enforcement com munity has taken a difference approach. Despite the lack of evidence that the various voice stress technologies work, and despite the absence of any field evaluations of them, the technologies have been put to work by a number of law enforcement agencies around the country and around the world.
|
From page 12... ...
They contend that simply questioning a person with such a device present can, if the person believes that it can tell the difference between the truth and a lie, induce that person to tell the truth. Preliminary Credibility Assessment Screening System With the reliability of voice stress technologies called into question, the intelligence community needed another way to screen for deception.
|
From page 13... ...
"Red" indicates that the person had significant physiological reactions when asked significant questions, indicating that the person may have provided deceptive answers. "Green" indicates no significant physiological responses to the relevant questions, and "yellow" appears when the test was inconclusive.
|
From page 14... ...
It was carried out in Columbus, Ohio, by people recruited from the community. The scenario involved the theft of a diamond ring from an office, and the PCASS examinations were carried out by retired law enforcement officers who had been recently trained in the use of PCASS.
|
From page 15... ...
But the technique is not assumed to give a definite answer, only a conditional one. Because PCASS is used on the front lines, it has never been field tested, Veney explained.
|
From page 16... ...
"I think we can do better than put something out there that has such limitations." And Brandon commented that "if we were doing really good field validation with the PCASS" then it might well become obvious that other, less expensive methods could do at least as good a job as PCASS at detect ing deception. There are a number of important questions concerning the validity and reliability of PCASS that can be addressed only by field evaluation, and until such validation is done, the troops in the field are relying on what is essentially an unproved technology.3 PREDICTION METHODS One of the most common tasks given to intelligence analysts is to predict the future.
|
From page 17... ...
Of these studies, the experts outperformed the actuarial models in only 8, the models outperformed the experts in 65, and the performance of the experts and the models was about the same in the remaining 63. None of this shows that analysts in the intelligence community could be outperformed by predictive models, Thomason said, but it does suggest the possibility that such models can be used to improve expert judgments.
|
From page 18... ...
Alternative Competing Hypotheses The basic idea behind structured-thinking techniques, Thomason explained, is to help experts structure their thinking so that various biases are alleviated or even avoided altogether. There is a well-known, well-established psychological literature on such biases that informs the techniques.
|
From page 19... ...
I don't know." It is frustrating, he said, that although ACH was originally proposed a third of a century ago and although it has achieved "a cult-like sta tus" in the intelligence community, there have been so few studies that have tested whether and under what circumstances it actually works to improve the predictions of analysts. In part because of the intellectual isolation of the intelligence community, Thomason added, few researchers in informal logic or other areas pay attention to ACH, and so the normal scientific process that takes place in academia when a new theory or approach is suggested has not happened with ACH.
|
From page 20... ...
And since the basic psychology of ACH should work equally well on problems outside the intelligence community, it should be possible to perform the tests in various settings with various types of participants. This should offer insight into which settings and for which types of users ACH is most effective in improving expert judgment.
|
From page 21... ...
Applied Bayesian Analysis A second approach to improving prediction applies Bayesian analysis, a statistical approach that uses observations and other evidence to regularly revise and update a hypothesis. Charles Twardy of George Mason University described APOLLO, a software application that uses an advanced form of Bayesian analysis called Bayesian network modeling to help analysts predict the likely behavior of a country's leader or other persons of interest.
|
From page 22... ...
In the con ventional approach, the analysts assigned numerical probabilities based on their own judgment and intuition. The five analysts started out with varying estimates of the probability of a Soviet action -- anywhere from 10 to 80 percent -- and, as events unfolded throughout September 1969, all of them revised their estimates steadily downward until their estimated probabilities were close to zero.
|
From page 23... ...
"Even though there are tremendous numbers of studies in the psychology literature that you are better off with Bayes, you never know what is going to happen when you give it to analysts." It seems likely that the fifth analyst had somehow misunderstood how to apply the Bayesian analysis -- and that is an important piece of information that only a field study could detect. The research group's work ended in 1979, and Twardy said he has seen no evidence that work with Bayesian methods continued beyond that.
|
From page 24... ...
is created, ideally with the input of a number of analysts and experts assembled for a two-day meeting. The model traces how various eventualities affect one another, with probabilities assigned for each cause-and-effect relationship.
|
From page 25... ...
For example, if a leader's major goal is to stay in power, his responses to various events should be significantly different than if his goal is, say, to move his country toward democracy. The model helps untangle the various causes and effects and focus on what events imply about motivations.
|
From page 26... ...
"You sacrifice more validity, but you get a lot more access." And, he continued, there is no reason it wouldn't be possible to do a tiered approach, in which much of the early work is done with psychology undergraduates, and then, after the bugs are fixed, further testing is done in the intelligence academies or with working analysts. Finally, Twardy noted, one of the obstacles to incorporating APOLLO or something similar in the work of intelligence analysts -- in addition to the extra time it demands from them -- is the fact that it requires quantita tive estimates of probabilities: there is a 20 percent chance that the striking workers will back down if the army is called in; there is a 40 percent chance that the army will take over the country if the leader flees; and so on.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.