Skip to main content

Currently Skimming:

Implications of Cognitive Psychology for Measuring Job Performance
Pages 1-26

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.


From page 1...
... Fundamentally, the measurement of job performance should be driven by modem cognitive theory that conceives of learning as the acquisition of structures of integrated conceptual and procedural knowledge. We now realize that someone who has learned the concepts and skills of a subject matter has acquired a large collection of schematic knowledge structures.
From page 2...
... The construction of tests that are diagnostic of different levels of competence in subject matter fields is a difficult task, but recent developments, including cognitive task analysis and research on the functional differences between experts and novices in a field, provide a good starting point for a theory to underpin proficiency measurement. Until recently the field of psychological measurement has proceeded with primary emphasis on the statistical part of the measurement task, assuming that both predictor and criterion variables can be generated through rational and behavioral analysis and perhaps some intuitions about cognitive processing.
From page 3...
... In this regard, cognitive psychology has produced a variety of methods that can be sources of a new set of measurement methodologies. During the past 2 years, we have been applying these techniques in developing a cognitive task analysis procedure for technical occupations in the Air Force.
From page 4...
... Something like the following, example ~ is quite common: TO TRAVEL-TO :X SUBGOAL CHECK-OUT-POSSIBILITIES IF :X WITHIN 50 MILES THEN SUBGOAL DRIVE-TO :X ELSE SUBGOAL FLY-TO :X END Because of this sort of contingent branching, an overall subgoal structure for a particular goal may not exist explicitly. Rather, it may be assembled as the goal is being achieved it is implicit in part.
From page 5...
... Expert-novice comparisons are also helpful. Also, information about novice goal structures can sometimes reveal training problems that could be corrected by specifically targeted instruction.
From page 6...
... A major part of our current work on cognitive task analysis of avionics equipment repair skills is to catalog the procedures that an airman must know in order to perform tasks within the job specialty being studied. This is done in a variety of ways and is probably the aspect of cognitive task analysis that is closest to traditional rational task analysis approaches.
From page 7...
... Combined with goal structure knowledge, selection rules are an important part of what is cognitive about cognitive task analyses. Conceptual Knowledge In performing cognitive task analyses, it is important to consider how deep or superficial the knowledge and performance are.
From page 8...
... The cognitive task analyst must attempt to determine the role played by such knowledge in successful performance. The knowledge to be examined includes scientific laws and principles as well as more informal background, such as crude mental models and metaphors for processes that are directly relevant to job tasks (e.g., what happens when a circuit is shorted)
From page 9...
... There are several ways in which this can be done. For some domains, such as electronics troubleshooting, there is an existing literature because cognitive scientists have used the domain to study expertise, have been designing tutoring systems for the domain, and have been building expert systems to supplement human expertise in the domain.
From page 10...
... Our own exploratory efforts in cognitive task analysis lead us to make two important cautionary statements about this aspect of cognitive task analysis in particular and the entire approach in general: · A cognitive task analysis stands or falls partly on the level of expertise in the target domain that is assimilated by the analysis team. This is not a chore for dilettantes.
From page 11...
... Declarative knowledge can, of course, be partitioned into categories such as goal structure, procedures, selection rules, and conceptually supporting information. When analyzing verbal protocols, it is also important to distinguish between verbal protocol content that provides a trace of declarative knowledge of a task and content that reveals the mental representations that guide performance even after it is automated.
From page 12...
... Summary Our analysis of job performance highlights the following components of skill: knowledge of the goal structure of a task skill and knowledge prerequisites for successive levels of performance procedural skills and the rules for deciding when to apply them conceptual knowledge and metaphors that support performance mental models and task representations levels of learning, from declarative to proceduralized knowledge, from rigid algorithms to flexible strategies In general? this approach to job performance is intended to avoid using performance correlates as the basic units of analysis, to instead base measurement and evaluation on an analysis of the specific cognitive procedures and conceptual representations that produce successful perfo~ance.
From page 13...
... An instrument might be very effective at picking, the right people to be taught a job without being particularly good at specifying how those people who make the cutoff for selection will differ in either their ability to learn or their post-training performance. In essence, when looking at the incumbents within a specific military job specialty, one is looking at a group whose members are chosen because they are classified in the same manner by the available selection tests as appropriate trainees.
From page 14...
... Conceptual errors would not have blocked the assembly, but the plane would not have functioned properly afterwards. Sorting Tasks Sortin:, tasks are an important exploratory tool for cognitive task analysis.
From page 15...
... Such differences as have been found seem to involve very small numbers of items that have specific ambiguities of nomenclature that only the better performers are sensitive to. Characteristically, novices put things together in a sorting task on the basis of their superficial characteristics, while experts sort more on the basis of deeper meaning, especially meaning relevant to the kind of mental models or schemes that drive expert performance.
From page 16...
... Realistic Troubleshooting Tasks In job domains that involve substantial amounts of diagnosis or other problem solving, some of the most revealing, tasks used in cognitive task analyses are those that provide controlled opportunities for the subjects to actually do the difficult parts of their jobs. We have only begun to work on this approach, but a few possibilities already present themselves, particularly with respect to metacognitive skills of problem solving.
From page 17...
... COGNITIVE PSYCHOLOGY AND MEASURING PERFORMANCE 17 This then created the possibility that we could specify in advance a set of probe questions that would get us the information we wanted about subjects' planning and other meta-co=,nitive activity in the troubleshooting, task. In this most complex troubleshooting task, there are perhaps 55 to 60 different nodes in the problem space, and we have specific meta-cognitive probe questions for perhaps 45.
From page 18...
... 18 How would you do this? Why would you do this?
From page 19...
... In one approach, the items are randomly distributed in such a way that no structure information is conveyed by the test form. Ideally, they should be at random points on the edge of a circle.
From page 20...
... Antenna |Transmitter | [Synchronizer | | Circulator | | Power | 1 Supply I | Receiver | | Indicator | FIGURE 3 Systematically arranged connections test form.
From page 21...
... These are areas in which rational task analysis has not been able to supply adequate direction to those who design training systems because of the specific lack of an understanding, of job difficulties that arise from the nature and limitations of human cognitive function. Ire commenting on the specific situations in which cognitive approaches can be useful, we address three related topics.
From page 22...
... Complex job environments require deeper cognitive analyses that can ferret out the conceptions and thinking that lurk behind observable behaviors. For the military sector, understanding the influence of the machine on work and the dimensions of intelligent human performance in work settings is very important.
From page 23...
... If performance assessment is directed toward the measurement of individual skill for purposes of improving performance, i.e., if the goal is diagnosis to prescribe instruction, then individual differences in cognition are worth some attention. A cognitive analysis that examines complex human performance in depth may uncover uniformities as well as common misconceptions or bugs that affect the learning process.
From page 24...
... Inventive testing informed by cognitive analyses could conceivably begin to shift the emphasis in technical training away from academic models of learning facts to experiential models of learning procedures. Frederiksen (1984)
From page 25...
... Specifically, the approach we favor is one of identifying the critical mental models, conceptual knowledge, and specific mental procedures involved in competent performance and then asking whether a given test allows one to reliably assess the extent of those aspects of competence. This suggests that traditional paper-and-pencil formats may have to be supplemented by hands-on testing in order to be sure that procedural skills are well established, but it also suggests that even exhibiting competent performance on the job may not predict transfer capability nor the ability to work well with nonstandard problems or work conditions.
From page 26...
... Stevens, eds., Mental Models. Hillsdale, N.J.: Lawrence Erlbaum Associates.


This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More information on Chapter Skim is available.