Identifying the Culprit Assessing Eyewitness Identification (2014) / Chapter Skim
Currently Skimming:
Appendix C: Consideration of Uncertainty in Data on the Confidence-Accuracy Relationship and the Receiver Operating Characteristic (ROC) Curve
Appendix C: Consideration of Uncertainty in Data on the Confidence-Accuracy Relationship and the Receiver Operating Characteristic (ROC) Curve
Pages 139-154
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 139... ...
and the confidence– accuracy relationship involve data (usually, as the proportions of participants in a given study that meet some criterion) and hence are subject to various sources of uncertainty, including measurement error, random variations from external conditions, and biases (such as the tendency to respond "conservatively" or "liberally"; see examples of these biases in Chapter 5)
|
From page 140... ...
denotes the false alarm rate.1 CONFIDENCE–ACCURACY RELATIONSHIP When authors talk about the confidence–accuracy relationship, they usually are referring to a correlation coefficient or to a slope of the line fitted to the points (C, A) , where a measure of the eyewitness' expressed confidence level C is on the x-axis, and a measure of the witnesses's accuracy A is on the y-axis.
|
From page 141... ...
However, the relationship is not "statistically significant" when assessed via a weighted linear regression (where weights are inversely proportional to either the standard errors or the variances) , nor via an unweighted Pearson correlation coefficient or a Spearman's rank correlation coefficient (which depends less on the assignment of "Low," "Medium," and "High" as 2, 5, 7, respectively, than do the other two methods)
|
From page 142... ...
For each trial with five simulated c values and their five corresponding a values, one calculates a Pearson correlation coefficient. Figure C-1 shows a plot of the five data points, with limits of one standard error on the estimated accuracies (left panel)
|
From page 143... ...
the uncertainty in an eyewitness' "expressed confidence level." The 2009 National Research Council report, Strengthening Forensic Science in the United States: A Path Forward, cited studies in which fingerprint examiners reached different conclusions when presented with exactly the same evidence at a later time.5 Quite possibly, in many of these laboratory studies on which these confidence–accuracy relationships are based, participants 5 ational N Research Council, Strengthening Forensic Science in the United States: A Path Forward (Washington, DC: The National Academies Press, 2009)
|
From page 144... ...
ROC has been used for decades in the medical test diagnostic literature. Conventionally, as noted in Chapter 5, two procedures were compared using a single diagnosticity ratio: DR = HR/ FAR = hit rate/false alarm rate, or sensitivity / (1 – specificity)
|
From page 145... ...
T Wixted, "Receiver Operating Characteristic Analysis L of Eyewitness Memory: Comparing the Diagnostic Accuracy of Simultaneous and Sequential Lineups," Journal of Experimental Psychology: Applied 18: 361–376 (2012)
|
From page 146... ...
As above, one can simulate each of the five hit rates and the five false alarm rates, with 4,000 independent trials and 1,200 participants, in such a way that the means of the five distributions of hit rates (HRs) and the means of the five distributions of false alarm rates (FARs)
|
From page 147... ...
values given by the "simultaneous" and "sequential" data in Mickes, Flowe, and Wixted, Table 3.13 The text indicates that Experiment 1A used n = 598 participants, so the simulation assumed n = 600. In Figure C-3, "M" refers to "siMultaneous," and "Q" refers to "seQuential." Note that the "M" and "Q" points fall roughly in the same pattern as in Mickes, Flowe, and Wixted's Figure 6A.14 Note the substantial overlap in the bands of "one standard deviation" surrounding each of the data points, indicating no "statistically significant" differences between the "M" (simultaneous)
|
From page 148... ...
using a third variable [different test thresholds in the medical literature; in the present context, different expressed confidence levels (ECLs)
|
From page 149... ...
different expressed confidence levels]
|
From page 150... ...
. ALTERNATIVE ANALYSIS TO CONFIDENCE-BASED ROC FOR COMPARING PROCEDURES As noted in Chapter 5, the diagnosticity ratio [hit rate/false alarm rate = HR/FAR = sensitivity/(1 – specificity)
|
From page 151... ...
Distinctive feature, two levels: present versus absent. The data are provided in their Table 3, along with 95% confidence intervals.19 Because the length of a confidence interval is proportional to the standard error, pAUC values with shorter confidence intervals correspond to smaller standard errors and hence should have higher weights.
|
From page 152... ...
"An Evaluation of Lineup Presentation, Weapon Presence, and a Distinctive Feature Using ROC Analysis," Journal of Applied Research in Memory and Cognition 3(2)
|
From page 153... ...
, and sim versus the average of SEQ2 and SEQ5 (3.90) , and consider all pairwise interaction terms among the four "main effects." All single degree-of-freedom effects remain non-significant, in either this weighted analysis or in an unweighted analysis.
|
From page 154... ...
"An Evaluation of Lineup Presentation, Weapon Presence, and a Distinctive Feature Using ROC Analysis," Journal of Applied Research in Memory and Cognition 3(2)
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.