Intelligence Analysis Behavioral and Social Scientific Foundations (2011) / Chapter Skim
Currently Skimming:
4 Use of Signal Detection Theory as a Tool for Enhancing Performance and Evaluating Tradecraft in Intelligence Analysis--Gary H. McClelland
4 Use of Signal Detection Theory as a Tool for Enhancing Performance and Evaluating Tradecraft in Intelligence Analysis--Gary H. McClelland
Pages 83-100
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 83... ...
The chapter relies extensively on recent advances in assessing medical forecasts and detections, where signal detection theory specifically and evidence-based medicine more generally have led to many advances. Although medical judgment tasks are not perfectly analogous to IC analyses, there are enough strong similarities to make the examples useful.
|
From page 84... ...
As the medical examples illustrate, rigorous evaluation of physician judgments and practices using the methods proposed in this chapter have improved medical outcomes substantially. It is reasonable to expect similar benefits if these methods are applied in intelligence analysis.
|
From page 85... ...
This chapter suggests signal detection theory as a useful method for keeping score, develops some examples in the context of intelligence analysis, and describes some benefits of keeping score that have been achieved in other disciplines, such as weather forecasting and medicine. SIgNAL DETECTION THEORY Proposing signal detection theory as a method for keeping score -- to evaluate prediction quality -- in intelligence analysis, specifically, or detection and diagnosis, more generally, is neither novel nor surprising.
|
From page 86... ...
response bias -- the propensity to be cautious and overwarn (false alarms) versus avoiding crying wolf, thereby underwarning (misses)
|
From page 87... ...
The inherent problem for the analyst is to decide when the graded evidence is strong enough to issue an alert. Hit and false alarm rates Commonly used basic measures of detection performance are the hit rate -- probability of correctly alerting when a signal is present -- and the false alarm rate -- incorrectly alerting when a signal is not present.
|
From page 88... ...
They are all transformations of one another, so the choice is one of convenience. Assessing detection performance The key issue in this context is how to use the hit and false alarm rates to assess the performance of the detection system, whether that system is an electronic device (e.g., the Preliminary Credibility Assessment Screening System, or PCASS)
|
From page 89... ...
comes at the expense of a high false alarm rate. The gray circle mid-way 1 0.8 0.6 Hit Rate 0.4 0.2 0 0 0.2 0.4 0.6 0.8 1 False Alarm Rate FIgURE 4-2 Differential weighting of misses and false alarms.
|
From page 90... ...
1 0.8 0.6 Hit Rate 0.4 0.2 0 0 0.2 0.4 0.6 0.8 1 False Alarm Rate FIgURE 4-3 ROC curves representing increasing detection performance with increasing distance from the diagonal. SOURCE: Generalized from Green and4-3.eps Fig Swets (1966)
|
From page 91... ...
This was a serious false alarm. Strong experimental evidence in other contexts shows that changing the costs of misses and false alarms changes response bias and hence the rates for misses and false alarms (e.g., Healy and Kubovy, 1978)
|
From page 92... ...
. Here is an example of a detection problem that illustrates the substantial effects of base rates on percentage accuracy even when the hit rate is very high and the false alarm rate is very low.
|
From page 93... ...
Summary of Benefits of Signal Detection Theory If one is going to keep score of prediction performance, signal detection theory provides an ideal framework. Its fundamental value is separating the effects of base rates, detector accuracy, and cut-point biases motivated by avoiding either false alarms or misses.
|
From page 94... ...
provides such a metric." An important benefit of using signal detection theory to evaluate and compare performance of individuals, teams, systems, procedures, and other factors is that it would require only a minimal, almost trivial, addition to the daily activities of the typical analyst. The only additional workload for the analyst would be to produce a probabilistic or categorical prediction of the future events being analyzed.
|
From page 95... ...
For example, intelligence analysis tradecraft not evaluated adequately include alternative competing hypotheses, PCASS, and even recent communication innovations such as Intellipedia and A-Space. The scientific literature is replete with examples of conventional wisdom, often based on observational data and anecdotes that turn out to be untrue when 2 Now the Centers for Disease Control and Prevention.
|
From page 96... ...
. These specific examples probably have no direct relevance to intelligence analysis except that they demonstrate that many strong beliefs of practitioners, developed over many years of experience, are often not confirmed by scientific experimentation.
|
From page 97... ...
Although many possible measures might be used as scores, those of signal detection theory seem naturally suited to the uncertainty problems facing the IC that involve problems of low base rates, fluctuating biases toward false alarms and misses, and detector accuracy. REFERENCES Bakwin, H
|
From page 98... ...
2008. "Noisy patients" -- Can signal detection theory help?
|
From page 99... ...
2001. Elementary signal detection theory.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.