Enhancing Human Performance Background Papers, Issues of Theory and Methodology (1988) / Chapter Skim
Currently Skimming:
Part II Intuitive Judgment and the Evaluation of Evidence
Part II Intuitive Judgment and the Evaluation of Evidence
Pages 99-148
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 99... ...
Intuitive Judgment and the Evaluation of Evidence Dale Grif f in Stanford University
|
From page 101... ...
Natural processes of judgment and decision-making are subject to systematic errors. The formal strucn~res of science -- objective measurement, statistical evaluation and strict research design -- are designed to minimize the effect of such errors.
|
From page 102... ...
More generally, the themes will include why personal experience is not a trustworthy source of evidence, why people disagree about beliefs despite access to the same evidence, and why evidence so rarely leads to belief change. The underlying message is ~is: The checks and balances of formal science have developed as protection against the unreliability of unaided human judgment.
|
From page 103... ...
The key difference between intuitive and scientific mess is that Me measurement and analysis of the scientific investigation are publicly available, while intuitive hypothesis-testing takes place inside one person's mind. Recent psychological research has examined ways In which intuitive judgment departs from fonnal models of analysis -- and in focusing on such "errors" and "biases", this research has pinpointed some natural mechanisms of human judgment.
|
From page 104... ...
Cogn~tive judgments have the same feeling of "true" -- it is difficult to believe mat our personal experience does not perfectly capture the objective world. I The systematic biases I win be discussing throughout this section operate at a basic and I automatic level.
|
From page 105... ...
The predictions from the human judges and the statistical models were then compared win the actual outcomes. The clinical judges involved in these studies were exceedingly confident that statistical models based on obvious relationships could not capture the subtle strategies that they had developed over years of personal experience.
|
From page 106... ...
This combination of demonstrably suboptimal judgments and continued confidence of me judges set the stage for the two themes of the judgment literature: What is wrong with human judgment? and Why don't people naturally realize the limitanons of human intuitive judgment?
|
From page 107... ...
models: selecting the information One reason for the superiority of statistical judgment is Hat it utilizes info~manon based on the observed quantifiable relationship between the predictors and the outcome. A prediction equation sums by identifying Hose predictors that are mean~ngfi~ in a purely statistical sense.
|
From page 108... ...
People Ming to solve logical puzzles doggedly set Out to prove their hypothesis by searching Out confirming examples, when they would be much more efficient if Hey would search for disconfirming examples (Wason, 1960)
|
From page 109... ...
Sure enough, their opponents responded to their aggressive moves with aggressive countermoves, "proving" the compenOve theory of human nature. Such biases do not need to come from strong long-standing theories, they can be created within one situation.
|
From page 110... ...
In this example, formal statistical methods meet "head to head" with intuitive judgment Remote viewing is the name given to the ability to "see" a sewing mat is physically removed from the viewer. In Me ong~nal SRI study, viewers descnbed their impressions of a number of targets and sketched an outline of those impressions.
|
From page 111... ...
Sometimes the biases of attention and memos are random and "average out" to give accurate conclusions. But because we are not aware of these biased processes as Key occur, we cannot know when to Dust our intuitive judgments.
|
From page 112... ...
Especially in He area of social perception, psychologists have demonstrated how people or things that are "eye-catching" are seen as having more influence than are over people or things who act the same way but are less noticeable (Taylor tic Fiske, 19781. The effect of salience -- the tendency to notice some things more than omen because they are brighter, louder, unique, or noticeable In some other way -- underlies many of Me failures of human judgment of probability and improbability.
|
From page 113... ...
Is a bespectacled, mild-mannered student of love poetry, more likely to be a Buck Diver or an Ivy League Classics professor? Counting the (proportionally few)
|
From page 114... ...
In He first condition, me judges were aware mat Hey had valid infonnation about the particular case, and so could safely assume that the sketches contained diagnostic information about future career choice. In the second condition, He judges were aware that the infonnation had lime validity, and should have been influenced by how many people (~e base rate)
|
From page 115... ...
The base rate of fraud and cheating In the history of commercial psychic
|
From page 116... ...
are not absolutely certain few experiences should not be enough to markedly change our beliefs in the unlikelier of the explanation. But the events are vivid, we are wining to make broad inferences on the basis of a few memorable cases, we attribute the results to the salient cause without possible base rate explanations that would reveal simple unsure sing explanations for He results -- that is, we jump to conclusions.
|
From page 117... ...
3- Intuition versus formal models: the generalizability of sample conclusions Informal inference is prone to systematic bias in data coUecdon -- either from simple costive attentional biases or from biases in retrieving information. In order to keep the effects of this biased sampling procedure minimal, there are some basic cautions Mat must be taken in Me treannent of the data.
|
From page 118... ...
The second limit in generalizing from a sample of evidence is the reliability of that sample-which depends on the size of the sample. This is explicitly taken into account in statistical models, which are very cautious about conclusions made on the bases of a few cases.
|
From page 119... ...
than Me more likely patterned altemative. If people believe that random processes cannot produce sequences that look systematic, how do they respond to patterns in random data?
|
From page 120... ...
The lesson for evaluating quantitative evidence by subjective means is clear: these basketball records are comparable to the results of a trial of some new method of performance enhancement Even if there is no systematic structure In the data, people win see meaningful pauems~ shout any theories other than the belief that random processes must look random. The most serious flaw in our understanding of randomness is the over pretation of coincidence.
|
From page 121... ...
It is difficult to perceive one's own adventure as just one element in a sample space of people,
|
From page 122... ...
Instead, human judgment seems desired as much for protecting the ego of the dec~sion-maker as generating accurate predictions and assessments. Consider the "hindsight bias", also tenned the "knew-it-ah-along" effect 0:ischhoff, 1975~.
|
From page 123... ...
When the answers seem obvious to us Her the fact, we believe that our intuitive abilities are being confirmed. People in virtually all circumstances and professions (except hose-racing handicappers and weather forecasters, who receive repeated objective feedback)
|
From page 124... ...
72~. Human judgment allows little room for uncertainty; it is set up to explain the world -- and to prevent the anxiety that comes with
|
From page 125... ...
The process of testing hypotheses Cough personal experience leads to certain common violations of me basic tenets of research design. A simple but non-obvious rule of correct research design is that relationships can only be sum ported by examining all four possible outcomes of a success/fai} trial (see Figure 2)
|
From page 126... ...
Another essence facet of research design that is neglected in He search for evidence through personal expenense is He need for experimental consul. Valid conclusions can only come when data is also collected on occasions when the manipulation of interest is not used.
|
From page 127... ...
Most techniques proposed to improve human perfonnance are actually combinations of a number of distinct interventions. In such cases, conceptual combinations of variables must be identified that are distinct from attention and demand effects in order for 'evaluation' to be sensibly applied.
|
From page 128... ...
We cannot be "blind" to our theories when collecting He data, and we always know whether each data point collected supports or weakens the evidence for a theory. Without c are fin consideration of research design, people cannot help but bias the sample of data Hey collect The risks of gathering evidence through personal experience: an example Pan of He responsibility of the Committee on Techniques for the Enhancement of Human Perfonnance has been to make site visits to various research establishments.
|
From page 129... ...
This ignores the base rate of emotional reactions in general, focuses only on the "hits", and leads to an illusory correlation between the "sign" (the in vitro leucocyte response) and the "cause" (the human emotional response)
|
From page 130... ...
Biases cannot be completely removed -- but they can be minimized by research methodology that can be publicly scn~t~n~zed and objectively evaluated. Problems in Evaluating Evidence lI: Motivationalfactors in judgment Not only are our perceptions biased towards our expectations, but we also actively distort our perceptions in order to see what we want to see.
|
From page 131... ...
They did, however, manipulate the amount of "skill demls" In the situation. They found Mat when subjects were able to choose Weir own occult symbol to send, and when the sender and receiver were able to discuss their commun~cadve technique, Hey believed that they were operating at a success rate Wee times the chance rate.
|
From page 132... ...
Those who believe that sensitivity to pain indicates a healthy heart find themselves unable to bear the discomfort for more than a minute. A few of the participants are aware that they are "cheating" on We diagnosis, but only Pose who are not aware of Weir own motivation are confident that Hey have me hearty type of heart These people are not deceiving the investigator, they are deceiving themselves.
|
From page 133... ...
People feel unpleasantly aroused when two cognitions are dissonant -- when they contradict one another -- or when behavior is dissonant with a stated belief. To avoid this unpleasant arousal, people win often react to disconfim~ng evidence by strengtherung Weir beliefs -and creating more consonant explanations.
|
From page 134... ...
Those who thought about the evidence became sufficiently distressed to be motivated to resolve the dissonance by strengthening their own beliefs. Though most of the research on dissonance theory has involved a~tud~nad or emotional commitment, financial commitment to an enterprise sets up a similar psychological system.
|
From page 135... ...
Instead, this theory claims that people infer their own motivation by observing their own behavior. Wring the process of seDing a product, a person obliges Is on clams for mat product -- md mess me sheen is content to conclude that he or she is motivated only by the money -- win likely conclude ~ he or We has very good reason to believe in the quality of We produce Beyond perceptual and judgmental biases, misunderstandings of chance phenomena and motivated distortions, lies me essential reason why personal expenence cannot be decisive: We can never determine the Due cause of our behavior or our experience.
|
From page 136... ...
This is appropriate to the extent that people are able to evaluate second-hand information rationally. But aB the attentional biases that are active in our personal experience are doubly pernicious when we evaluate processed evidence because the media further emphasizes the vivid, emotionally gripping aspects of information while ignoring or downplaying cautions and unexciting statistical summaries.
|
From page 137... ...
37 discounted, the information obtained may become part of the general knowledge of the the recipient. In the classic demonstration of this phenomenon, students were given persuasive arguments about the use of nuclear power and were told that the source of the arguments was either Pravda or an American nuclear scientist.
|
From page 138... ...
, 1983; Kurtz, 1984b) and demonstrate Me way Mat personal reports fit the requirements of the media better than caution or rigor.
|
From page 139... ...
In a more-recent extension of this idea, social psychologists at Stanford University presented proponents and opponents of capital punishment with some studies that purported to show that deterrence worked, and some studies apparently showing that capital punishment had no deterrence effect (Lord, Ross & pepper, 1979~. They reasoned that common sense must dictate mat mixed evidence should lead to a decrease in certainty in the beliefs of both partisan groups.
|
From page 140... ...
Both groups believed that the methodology that had yielded evidence supportive of Heir view had been clearly supenor, both In its relevance and freedom from artifact, to the methodology Hat had yielded non-suppo~ve evidence. In fact, however, the sum jects were evaluating exactly the same designs and procedures, win only the purposed results vaned....To put the matter more bluntly, He two opposing groups had each cons~ued me "box-score" vis a vis empirical evidence as 'one good study supporting my view, and one lousy study supporting He opposite view' -- a state of affects that seemingly justified He maintenance and even the strengthening of Heir particular viewpoint" (Ross, 1986, p.
|
From page 141... ...
These "objective" viewed were defined by the experimenters as those without personal involvement or sing options about the issues. But the partisans themselves -- if Hey are involved in college football, the capital punishment debate, party politics or the Arab-Israeli conflict -- claim to be evaluating me evidence on its own meets.
|
From page 142... ...
Articles published in the Journal of Parapsychology or the Journal of the Society for Psychical Research explicitly discuss the statistical assumptions and controlled research design used in their studies. Most active parapsychological researchers believe that the path to scientific acceptance lies Hugh Be adoption of rigorous experimental method.
|
From page 143... ...
Kurt Back (1972) titled his personal history of the human potential movement "Beyond Words" but it could have been just as accurately called "Beyond Measurement".
|
From page 144... ...
This example is often used in introductory statistics' classes because it demonstrates that good research really matters in He world. It shows how opinions based On personal experience or even uncontrolled research can cause the adoption or conirnuadon of dangerous policies.
|
From page 145... ...
Because the formal studies were publicly available, and because the quality of the studies could be evaluated on the basis of their exper~men~ method, the overall conclusions were decisive. Until the human potential movement agrees on the importance of quantitative evaluation, it will remain split into factions based on ideologies maintained by personal experience.
|
From page 146... ...
135~. Intuitive Judgment and the evaluation of evidence: A spry Personal experience seems a compelling source of evidence because it involves tile most basic processing of information: perception, abandon, and memory storage and retrieval.
|
From page 147... ...
These errors and biases are now welldocumented by psychologists and decision theorists, and the improvement of human judgment is of central concern In current research. The long-tem~ response to this knowledge requites broad educational programs in basic statistical inference, and formal decision-making, such as Pose props and examined by venous authors in Kahneman et al (1982~.
|
From page 148... ...
Omitting the investigator in question lowers the mean effect size from .23 to .22 but- raises the median effect size from .32 to .34. It seems clear that the questioned randomization of the 9 studies of this investigator cannot have contributed substantially to an inflation of the overall effect size.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.