National Academies Press: OpenBook

Emerging Needs and Opportunities for Human Factors Research (1995)

Chapter: 11 Aiding Intellectual Work

« Previous: 10 Cognitive Performance Under Stress
Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×

11
Aiding Intellectual Work

John D. Gould

INTRODUCTION

Nearly all human intellectual work is aided by technology. In the next decade, new technology will make possible many new potential aids. The main thrust of this chapter is to structure the problem of aiding intellectual work in such a way that human factors researchers can contribute toward the development and evaluation of electronic systems that will indeed aid their users.

The Problem

The problem focused on in this chapter is how to carry out human factors research aimed at augmenting human intellectual work with electronic aids. Aids include artifacts or tools, as well as procedures, techniques, organizational structures, and facilitated, encouraged communication patterns. Most human intellectual work tends to be mentally demanding, with a priority on timeliness and with a high premium on getting the ''right information" (i.e., not just the right answer to a question asked, but information that is relevant to questions that should have been asked but were not).

Today, all human intellectual work is augmented to some degree. Electronic aids for intellectual work include ubiquitous, simple "single-function" tools, some of which are largely electronic versions of established

Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×

books (e.g., dictionaries, thesauruses, foreign language translators) and some of which are more (e.g., calculators, spelling verifiers). There are spreadsheets for business people, high-level programming languages and tools for computer application developers, Cadam and very large-scale integrated (VLSI) tools for hardware designers, and conferencing systems for group work. To help users transfer learning from one computer application to another, software developers have attempted to standardize user interface styles (e.g., Apple Corporation's Macintosh style and IBM's CUA style). Many jobs simply could not be performed without electronic aids (e.g., work related to forecasting, reservations, large databases, simulations, and much medical practice).

Aiding intellectual work performance is a broad topic. Several domains have been addressed by recent National Research Council committees and are not explicitly covered here. One report summarized evidence on whether human performance is enhanced through techniques that have their roots in academic research, as well as techniques that have their roots in commercial enterprise. The various techniques used to enhance human performance include training, pain management, stress management, expert guidance, meditation, self-help subliminal audiotapes, self-assessment techniques, mind-altering drugs, and sports-psychology techniques to aid performance under pressure (see Druckman and Bjork, 1991; Druckman and Swets, 1988). Another National Research Council report (Ferber et al., 1991) discusses how human intellectual work performance can be affected by the linkages between work and such family and personal areas as finances, marital status, child care, the need to support elderly relatives, marital stress, illness, and working at home. Other studies have reviewed intellectual work aids for (a) people with special needs (e.g., sensory and motor impairments, communication difficulties) (see Chapter 3) and (b) students in primary, secondary, and collegiate schools.

Why Is Aiding Intellectual Work Important?

There are three main reasons for understanding how to better aid human intellectual work.

It Increases Productivity and Creativity

Productivity is a national problem: many studies report that the productivity of the U.S. labor force is relatively stagnant. Some widely used electronic aids have not helped much. Nevertheless, there is the belief, supported by informal evidence, that electronic aids, if developed with users and their work organizations in mind, do enhance productivity and creativity and can allow people to work in new ways. For example, using

Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×

electronic spreadsheets, business professionals routinely run "what if" analyses in seconds, a task that previously took hours or days or that would simply not have been done a few years ago. A variety of intellectual workers use simulations to provide quick studies of situations that previously could be studied only by actually building an object itself (if indeed it could be built). Scientists move single atoms, study their effects, and produce new combinations with awesome possibilities. With new visualization systems, engineers and scientists can see how wings are affected by wind or how a thunderstorm forms and thus develop an understanding that cannot be gained from studying the thermodynamics and the underlying isolated equations themselves. Powerful visualization machines have led to a change in the design of drugs: "scientists use graphics workstations to visualize a protein in three dimensions and determine which drugs will fit best into its active sites. Such visualization is so powerful that no pharmaceutical researcher would try to do the job without it" (Pool, 1992). An entire new field (experimental mathematics) and a way of thinking for mathematicians has developed based upon these new visualization tools (Pool, 1992).

It Is Socially Responsible

The abilities of people to cope with increasing intellectual work demands and their resulting self-images are affected by tools they are given (e.g., see the section below on "Organizational Impact of a New Technology"; see also Kraut et al., 1989). Some workers are deficient in problem-solving skills, spelling, composing, calculating, language skills (including second language problems), giving instructions, and so forth. Intellectual aids have the potential to reduce these deficiencies. Intellectual aids also have the potential to help people deal with problems outside work (e.g., financial planning, finding support groups, interacting with their children), and these, in turn, impact on their work lives (Ferber et al., 1991). Human factors researchers can help identify which intellectual tasks, if aided, could lead to the most benefits (for individuals and society), and they can help identify aids that can effectively help people with specific intellectual deficiencies compensate for what is latent or missing.

Our Expanding Technology Makes It Easier to Do

Rapid advances in electronic technology provide opportunities to improve existing intellectual aids, create effective and unforeseen new ones, and reduce costs, thereby bringing heretofore expensive aids to larger audiences. The productivity and creativity of individuals and work groups can be dramatically affected, in unforeseen ways, by the faster, cheaper processors with larger storage that are becoming available. In the next decade, the

Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×

costs of electronic storage and of computer processing are expected to continue decreasing at about the same rate as they have been (about 10 to 20 times per decade). The speed of computer processing is expected to continue increasing at about the same rate as it has been (about 10 to 20 times per decade). Communications bandwidth is expected to grow even more dramatically—about 100 times during the next decade.

Human Factors Special Interest

The evaluation and creation of intellectual aids draw on a variety of disciplines involved in human factors work, including assessment, cognitive science, applied and experimental psychology, work science, social science, organizational theory, and systems theory. It is the human-centered focus and the assessment and system development methodologies pioneered by these disciplines that provide the potential contributions.

THREE RECOMMENDED HUMAN FACTORS RESEARCH NEEDS AND STRATEGIES

Three human factors research efforts are recommended to improve human intellectual aids. Although they differ in emphasis, the three research needs overlap, as do their respective research strategies and methodologies. Several studies used as examples to illustrate one human factors research strategy can, from a different angle, be used to illustrate one of the other research strategies.

Assessing Existing Intellectual Aids

Today, little experimental and formal empirical evidence exists as to the impact of various electronic aids on individual worker and organizational productivity. Human factors research should be directed at empirically and experimentally assessing the value of existing intellectual aids, particularly as related to individual and organizational productivity.

Exploring the Nature of the Tasks to be Aided

The fundamental nature of the intellectual tasks to be aided must be understood. Stated differently, the characteristics of aids that will improve worker and organizational productivity and workers' quality of life must be discovered. Human factors research should empirically study ongoing intellectual work to identify which work tasks are "aidable," the cognitive characteristics of these tasks, and the characteristics that the aids should have. There exist a variety of human factors methodologies that can be

Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×

used to conduct empirical investigations of intellectual work in creative jobs and work organizations in order to understand what types of intellectual aids would be valuable. These methods include serious observational visits to work locations, task analyses, interviews, and thinking-aloud protocols.

Participating in Interdisciplinary Efforts

The disciplines that underlie human factors research are necessary to improve existing intellectual aids and to create new useful ones. But they are not sufficient. Human factors researchers should participate in serious interdisciplinary efforts with other scientists and engineers. This is necessary both because human factors researchers have the potential to make a contribution and to help ensure that human factors research has a significant impact. This chapter provides frameworks for thinking about how to address these three research needs, suggestions for following the recommended research strategies, and examples of existing research.

RESEARCH NEED AND STRATEGY 1: ASSESS EXISTING AIDS

It is important to know the effectiveness of the various intellectual work aids. It would seem reasonable to assume from observation that, to augment their intellectual work, people are successful in using some aids (e.g., actuarial tables) but not others (e.g., whatever aids many mutual fund managers used in the last decade). Although free enterprise market forces and differential commercial success of various intellectual work aids would seem to be a good indicator of their effectiveness, this may not always be the case. The experimental results that do exist sometimes question the value of the generally accepted intellectual aid studied (see the examples of studies below; see also Attewell, 1994).

Future human factors research should assess existing, ubiquitous intellectual aids to learn how well, if at all, they aid human intellectual performance and productivity; to understand more deeply how they affect people's minds and the productivity of their work organizations; and to use the results as insights in defining new aids that would be valuable. In addition to increasing productivity, knowledge about the effectiveness of intellectual work aids might help identify their common characteristics that have long-term positive effects (e.g., making workers more productive, improving the quality of work life). These discoveries might center on characteristics of the aids themselves or on characteristics of the processes used to develop them. For example, a number of questions might be asked about aid development processes. What design processes lead to the best aids? Do the

Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×

system design processes recommended by human factors people lead to good aids (e.g., iterative design, user-centered design; see Helander, 1988)? If so, then are there more case study data in support of these design and development processes? If not, then these recommended processes can be modified.

Experimental assessment of intellectual aids could help identify the cognitive implications of widely used aids. For example, does a calculator lead students to become poorer at basic math skills, as is often suspected, or better at math skills, which one could argue on the basis of accurate feedback? Intellectual aids can have surprising, unpredictable effects and intellectual implications. A personal example is presented here: Prior to the availability of copiers, I used to make notes and know a relatively small number of journal articles in detail; now I have a more peripheral knowledge of a larger number of journal articles and, when I need to know any details, I rely on remembering the existence and location of the copies that I have stacked in my office. Experimental studies of existing aids would focus energy on tasks for which the aids are designed to help, rather than on convenient lab or toy tasks that are often used in psychological investigations. This may lead to developing new methodology and a worthwhile theory about the work under study. It also can lead either to the identification of ways to improve an aid or to ideas about the design of new aids.

Figure 11.1 summarizes key points to be kept in mind when designing experiments or empirical field studies to assess an existing intellectual aid. Although a few readers may think that some of these points are obvious, many of us are familiar with studies that fail to follow "obvious" suggestions.

Examples of Studies

A few studies are described here to illustrate how to successfully assess a potential aid.

Composing Aids

Composing by Dictation In the 1970s dictation was often used to produce office correspondence. Vendors and employers encouraged their professional employees to compose letters and memos this way; office dictation systems, including hardware, software, and required procedures, were being designed, purchased, and used. There were several generally accepted notions about dictation at that time, as summarized in Figure 11.2. To test these commonly held views, in a series of about 20 laboratory experiments, participants were given a specific topic on which to compose a letter and a specific composition method with which to do it. They composed similar

Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×

Key Methodological Points in Assessing Existing Aids

Try to pick a ubiquitous, important task and a ubiquitous, important aid.

Continually ask the question: does this aid do what it claims to do? Does it do it in the SHORT run and in the LONG run?

Figure out how the task can be studied experimentally in the laboratory or empirically in the field. Often short-term laboratory experiments are not as informative as longer-term empirical studies conducted in context.

In laboratory experiments, include appropriate experimental controls, e.g., have participants do the same work with and without the aid (if this is possible). In empirical field studies, strive to do comparative studies of multiple groups or longitudinal studies of the "aided" group, especially if newly "aided."

Use tasks, equipment, material, procedures which relevant users and vendors would agree, a priori, provide a fair evaluation and could lead to their altering their behavior depending upon the outcome.

Design experiments so that results generalize to intended users (e.g., use appropriate people as experimental participants) and to a useful range of variation in the intellectual task under study (e.g., if it is querying a database make sure enough different types of queries are studied).

Choose meaningful dependent variables, including performance ones and attitudinal ones.

Try to identify and understand theoretically the underlying mental processes involved in the tasks and use of the aid.

Provide limits of generalization of the results in the study report.

Suggest how to improve the aid studied, or the possible use of alternative ones.

Prior to starting the experimental work, write conclusions sections that vary depending upon what is found. Show them to a few friends and get their reaction. This will help you determine whether it will be possible to draw any conclusions from the study design under consideration.

FIGURE 11.1 Key points in designing experiments to assess existing intellectual aids.

letters with each method studied. Their performance and behavior were measured while they composed, and the quality of their work was measured in various ways.

The results showed that, contrary to what was generally supposed, it does not take a long time to learn to dictate with dictating equipment (Gould, 1978a). Eight college graduates who had never dictated spent part of one day learning to dictate to a machine. The next day, they dictated four business letters and hand wrote four similar business letters. The experimental results showed that they dictated and wrote letters in about the same time and with about the same resulting quality. Quality was rated by judges who viewed typed copies of the letters and who did not know which method was used to compose them.

Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×

Common Notions about Dictation in the 1970s

1.

"Dictating requires a long time to learn."

2.

"Eventually, dictating is much faster than writing. That is, potential maximum output rates, as measured informally, are 200 words per minute (wpm) and 40 wpm, respectively.

3.

"Dictating may be qualitatively superior to writing because the higher potential output rate permits faster transfer of ideas from limited capacity working memory, thus reducing forgetting through interference or decay."

4.

On the other hand, handwriting was thought to have "an advantage over dictating because it is easier to review and modify."

FIGURE 11.2 Hypotheses about different composition methods in the 1970s. SOURCE: Gould (1980:101).

A second key result was that, contrary to what was generally supposed, (Figure 11.2), people experienced at using dictation equipment did not dictate several times faster than they wrote. Indeed, eight business executives who dictated regularly for years dictated routine business letters about 60 percent faster than they wrote them (p < .001), they dictated more complex letters (e.g., essays on the Bicentennial celebration, on the issue of capital punishment, on their favorite teacher) about 25 percent faster than they wrote them (p < .01) (Gould, 1978b). These experienced dictators dictated routine business letters about 20 percent faster than did novice dictators (p < .05), and they dictated more complex letters in about the same time as did the novice dictators. Dictated and handwritten letters, when typed, were rated as similar by judges blind as to the method of composition and the experience of the authors. Part of the aid assessment strategy recommended in this chapter is to develop a theoretical understanding of users' tasks and to relate the results of experiments to that understanding. These dictation studies found that the reason people do not dictate five times faster than they write, even though they can say words five times faster than they can write them (40 wpm vs. 200 wpm), is that the generation component of composition (i.e., the actual moving of one's hand or mouth) takes only a small fraction (25 percent or less) of the total time. Planning (i.e., pausing and thinking) takes two-thirds of the total time.

A third key result was the discovery that inexperienced dictators feel it takes them longer to learn to dictate than it actually does because they think the quality of their dictation is poorer than it actually is. Novice dictators, just after composing a letter, rated their dictated letters as significantly poorer than their written letters (Gould and Boies, 1978). Upon receiving a typed copy of their dictated and written letters and incorporating proofreading and editing changes, they elevated the ratings of their dictated letters to those of their handwritten letters. Experienced dictators, on the other hand,

Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×

rated their dictated and written letters equivalently at both stages. Judges later rated the quality of the final versions of the dictated and written letters as equivalent.

A fourth key result had to do with the just-emerging method of composing by speaking "voice documents," which recipients listen to (rather than read). Experimental results showed speaking to be a much faster method of composing than dictating or writing. Authors could use syntax, phraseology, and organization that they had learned over the years to be appropriate for listening but not necessarily for reading. In addition, in contrast to dictating, authors did not have to give typing instructions, which is potentially a cognitively disrupting "secondary" task (Gould, 1980). This information on voice documents had an impact on technology. In the middle 1970s, researchers at IBM had begun work on a "super dictating system," one that allowed users to apply great editing power to their aural documents. The results of the research project described here were a key factor in IBM's decision to shift the emphasis away from a super dictation system and toward a voice messaging system (Gould and Boies, 1983). Since then, voice messaging has grown dramatically as an industry (e.g., voice mail, telephone answering machines, etc.), whereas dictation has declined.

Composing With Text Editors In the 1970s, office professionals began using text editors to compose their own memos and documents—and to create the final text versions without the aid of secretaries. This trend led to research in the late 1970s to experimentally assess the productivity of professionals using text editors to compose documents (Gould, 1982), the method of composition that has since become dominant in much American industry. In the first study, 10 professionals, who regularly used a computer-based, mainframe, line-oriented text editor to compose their own correspondence and longer documents, composed four letters with a text-editor (T letters) and four letters with handwriting (W letters). A secretary typed each W letter, and then the participants revised and proofread the typed copy.

The results questioned the productivity advantage of using the line-oriented text editors of the 1970s. The basic result was that participants spent significantly more time composing T letters (29.5 minutes) than they did composing W letters (19.2 minutes) (p <.001). There was no difference in the judged quality and effectiveness of the T and W letters (based upon ratings by judges afterwards). Participants made many more modifications to T letters (41.3) than to W letters (8.5). There was no difference in the length of T and W letters, and there were no observable differences in the style of T and W letters. The report concluded that no productivity advantages were found for professionals who composed their own letters with their own text editor.

Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×

These results seemed questionable to some, particularly because of the lack of differences in quality between the T letters, with all their additional modifications, and the W letters. A few years later, Card et al. (1984) repeated this study, using the same tasks, materials, and procedure, but using a "display-oriented" editor (called Bravo) rather than a line-oriented editor. Again, participants' T and W letters were of the same length and were judged to be of the same quality and style. Again, participants made several times more modifications to T letters than to W letters, and again these did not lead to higher quality in the T letters. However, in this experiment the composition times for the two methods were about equivalent (mean times for T and W, respectively, were 22.8 and 21.7 minutes versus 29.5 and 19.2 minutes in the Gould study). That is, the mean time for T letters was reduced to about that of W letters. Besides extending the findings to other editors, an important point for this chapter is that results showed that experiments aimed at assessing aids can lead to cumulative knowledge.

Comment Note how these experiments reflect the recommended methodological points of Figure 11.1. Important aids were studied. Participants were experienced with all methods of composition. This helps generalization to the intended audience. Appropriate tasks, that is, letters rather than, for example, sentences or minor utterances, were used. A variety of letter types were studied, which leads to greater generalization of the results. Meaningful dependent variables—time and effectiveness—were measured in a variety of ways. In real work, these variables reflect productivity. Since the goal of communication is often to persuade, having judges select the most persuasive letters in some of the studies was better than simply judging spelling, syntax, and stylistic considerations. The experimental results were repeatable, extendable, and cumulative.

Some of the work had an impact on composing technology. Human factors research should not just question the value of existing aids but, whenever possible and perhaps aided by theory, find something that is a good replacement. Speaking documents as a method of composition was so identified.

Each report contained a theoretical explanation of the results, a serious discussion of limitations, and a pointed conclusion paragraph. For example, the Gould (1982) report on text-editing systems pointed out that only one text editor was studied; that even though there were no other experimental data on the use of text editors in composing tasks, it was known that the text editor used can affect the speed of routine secretarial editing tasks (Roberts, 1979); that line-oriented editors can lead to longer editing times for very simple editing tasks than display-oriented editors (Card et al., 1979); that the results are limited to one-to-two-page letters, typical of those composed

Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×

in industry; that these were not letters in which participants were deeply ego-involved or did an extensive number of drafts.

Missing from these studies, however, was a serious discussion, based upon interviews with participants afterwards, about why they were not going to modify their behavior after learning about the results. In other words, a powerful dependent variable in assessment studies is to try to take some ''aid" away from people and see how much they resist this or fight to get it back. Related to this is the observation that many developers do not, even when appropriate, themselves use the aid that they are making for others to use.

Everyday Aids

Norman (1988) analyzed the way many "everyday" aids (e.g., doorknobs, VCRs, stoves, faucets, automobile controls) work in his popular book on the subject. The book is valuable for its motivational, theoretical, and methodological content. It is largely anecdotal, empirical, and analytical (rather than experimental) in its approach. It shows an excitement to get started, a motivation to focus on existing aids. It articulates theories of mental mechanisms involved in using common aids. It correctly describes how successful artifact design proceeds. The book finds fault with many existing implementations of intended aids, but sometimes points to successful ones as well. Norman's methods can be extrapolated to aiding intellectual work.

Behavioral and Motivational Effects of Automation

One work setting that has become increasingly automated is the flight deck of modern jet transports. Many intellectual tasks, such as flight planning and navigation, have been delegated to flight management computers, with the tasks of flying becoming increasingly centered on entering data and monitoring electronic displays. This, of course, has been done to "aid" the crew and improve airline safety. But some researchers have raised questions about unintended consequences of this automation, such as complacency, overreliance on malfunctioning automated systems, reduced job satisfaction, changes in authority relationships, and loss of skills. One recent study found no differences in technical performance (errors) between crews flying an automated version and those flying a standard version of the same aircraft on a simulated flight involving mechanical malfunctions and diversions from planned routing (Wiener et al., 1991). There are probably many broad, longer-term issues surrounding the impact of using "electronic colleagues" that go beyond performance results from laboratory studies.

Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Organizational Impact of a New Technology

The studies described above have focused more on individuals than on groups and more on outcomes than on processes. The next study shows how to focus on all four. Methodology derived from experimental psychology is not the only way to assess the effect aids have on productivity. Empirical field studies, especially if they are aimed at organizations of people, rather than just individuals, can provide another powerful approach. While lacking some of the control of laboratory experiments, empirical field studies can assess whether individual productivity, measured in the laboratory, translates into organizational productivity and can also take into account contextual variables.

Kraut et al. (1989) studied what happened to productivity and quality of working life when a large company introduced a computerized record system to replace a microfiche record system in their customer service department. Management's goal was to increase efficiency and cut labor costs. Customer service representatives, who were the highest-paid nonmanagement employees in the company, were the primary contact between the company and customers. They attempted to solve customers' problems, collect overdue bills, and sell new company services.

Kraut et al. studied 10 customer service offices, collecting survey data from each (a) prior to the introduction of the new technology, (b) one month after its introduction, and (c) three months after its introduction. This approach allowed pre/post comparisons within an office. Each month the new system was introduced into another office. Thus, Kraut et al. started studying each office at successive monthly intervals (e.g., office 1 in November, office 2 in December, etc.). This "lagged, time-series" design also allowed comparison among offices in various stages of automation.

Management's main goal was realized. Productivity of service representatives, as estimated by the service representatives themselves (more objective data were not available), increased by 50 percent. They reported that their jobs got easier. However, they also reported (Kraut et al., 1989:230) that their jobs:

became less satisfying, less interesting, and generally of poorer quality following the introduction of the computerized record system. Further-more, contact with work colleagues became a less frequent and less satisfying component of service representatives' work life. … Interview data suggest[ed] that these changes resulted partly from changes in social interaction, based upon new seating arrangements, new privacy panels, and service representatives' limited physical movement that came from coordinating information through a database rather than through word of mouth and transfer of documents.

The supervisors of these service representatives, on the other hand,

Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×

reported their jobs became much more difficult after the introduction of the computerized system. They reported that they now worked harder. Evaluation became more time-consuming, largely because obtaining the relevant data from the new system was harder than before. Overflow work could no longer be done at home; it now had to be squeezed into business hours. Supervisors no longer felt expert in the details of the work that service representatives did, although service representatives reported that their supervisors were good sources of information and responded well to pressure and uncertainty during the transition.

Comment This study by Kraut et al. (1989) is a valuable example of how to conceptualize the issues, design an empirical study of the organizational impact of a new technology, and summarize the experience. It demonstrates the value of being guided by a general theoretical view of human work that takes into account a multiplicity of factors. The results showed that the new system's effects depended on workers' jobs, the tasks they performed, and the types of offices in which they worked. The study illustrates that impact on productivity is not the only key measurement of an aid. "Both economic and social theory as well as a rich case study literature suggest that while information technology may increase productivity it can degrade the work lives of those who use it" (Kraut et al., 1989:220). Quality of work life can sometimes be neglected in formulating a study, because nearly all human factors people work for management (not labor); this has powerful consequences, often subtle, on the goals of the work.

In developing and refining group aids, it is important to consider not only how they affect group processes and team efforts, but also how these affect individual motivation, skills, and job satisfaction in the long term and how they shape organizational cultures and norms. These goals would probably require that even longer studies be carried out.

Group Work Aids

Attempts to aid collaborative group work have taken many forms, including new systems, technologies, facilities, and procedures (and the tools and infrastructure to create them) and have been aimed at many tasks, including collaborative writing, design, problem solving, consulting, and programming. The emphasis has been upon tools for interactive, real-time work, although there have been some attempts to help people who are separated in time as well as in space.

A combination of experimental laboratory work and empirical field studies to assess various aids for group work is a valuable human factors approach. One such approach was carried out by Judy and Gary Olson (Olson et al., 1992:91) at the University of Michigan:

Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×

Our research strategy began by understanding group work in the field where we noted important baseline features of group work and gathered ideas about how various kinds of technology can help. We have now moved to the laboratory to study specific phenomena such as how technologies change the flow of activities in meetings or affect the quality of the work. From these laboratory studies, we will move back to the field, assessing the use of these new technologies in the full organizational context. This strategy has the benefit that we are informed by actual practice and can eventually test ideas in the field, but also puts the burden on us to construct laboratory situations that mirror the field in important ways.

In one experiment, Olson et al. (1992) compared the work processes and performance of small groups whose members were mainly business degree holders with business experience. Some groups used a group editor that allowed all participants to see and edit the same document at the same time, and other groups used paper and pencil, white-boards, and similar traditional aids. Each group worked face to face and collaborated for 90 minutes to draft the initial requirements statement for an automated post office. The designs produced by the "aided" groups were of higher quality than those produced by the groups working with traditional methods. The "aided" groups focused more on the core issues of the problem (in other words, they did less extensive exploration of the design space). However, the "aided" groups did not like their experience in the experiment as well as the control groups.

Research interest has been dramatic. In the most recent Association for Computing Machinery conference on computer-supported cooperative work, 191 research papers were submitted (48 were selected for inclusion) (Turner and Kraut, 1992:1). The conference papers appear to reinforce the research strategies recommended above. In the area of assessment, there was a feeling that empirical studies of the impact of aids on group work need to be longer-term than studies of individual work. According to Cool et al. (1992:31), "while the basic cycle of build-study-redesign is still valid, the implementation should be more complete and robust, and the study should extend over a longer time period before one trusts conclusions about ultimate value of a new communications system." Regarding the second recommendation—study existing work situations in order to determine how best to aid them—Berlin and Jeffries (1992) did precisely that for consultants helping apprentices. Bentley et al. (1992) carried out an ethnographic study of air traffic controllers in the United Kingdom. This methodology, relatively new to systems design, uses observational methods and puts emphasis upon identifying the social and cooperative processes involved in group work.

Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×

RESEARCH NEED AND STRATEGY 2: EMPIRICALLY STUDY ONGOING INTELLECTUAL WORK

In contrast to the first strategy (assessment), in which the emphasis is upon studying both an aid and a task, here the emphasis is upon studying an intellectual work domain or task itself (which, or course, is already aided in some way, since all tasks are). This means identifying the social, organizational, physiological, and cognitive mechanisms involved in the task. The goal is to use these observations to gain insights into what would make good new aids. There is no cookbook for this transition; one hopes the studies cited below will help generate ideas. Particularly in the case of group work, deciding which task to study can be a major challenge. Issues include whether to study long-term tasks (such as effectiveness in increasing sales, morale, safety, and student learning) or short-term tasks (such as teacher-student contact hours), entire work assignments (such as changes in productivity of a group) or subtasks (such as effectiveness in communicating). The underlying belief is that mental analysis and task analysis can help in aiding intellectual work more effectively.

To this end, it is probably more important to study the work of groups and organizations than the work of individuals because (1) it is harder to do, (2) most people work in organizations, and (3) there is uncertainty about how increases in individual performance affect organizational performance (see Chapter 1).

Many formal and informal human factors methods are available for carrying out empirical studies of ongoing intellectual work. Figure 11.3

Methods for Observing On-Going Intellectual Work

Visit worker locations.

Study the work organization—understand the target group's role.

Observe workers working—with and without existing aids they may use or wish they had.

Talk with workers—about their work and any existing aids.

Have workers think aloud while they work.

Videotape workers doing their jobs with and without intellectual aids.

Try the job yourself—with and without intellectual aids.

Collect opinions from associated workers outside the target group.

Study worker-made notes attached to equipment, documentation, bulletin boards for insights into what would be a better aid.

Surveys and questionnaires.

Use computer forums to learn about other possible aids.

FIGURE 11.3 Methods for collecting data on ongoing intellectual work. SOURCE: Gould (1988).

Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×

Key Methodological Points in Identifying the Characteristics of Useful Intellectual Aids

Understand the work that you want to aid, so that the proposed aid will likely have the right characteristics.

Focus on the work itself and the workers themselves: this keeps you close to the tasks you are trying to aid.

Pay attention to the social and organizational interactions and needs.

Analyze at a sufficiently abstract level for the purpose of identifying new ways for people to work.

Understand the context of the work (e.g., what policemen actually do) so that a proposed aid (e.g., a geographic database system) will be appropriate.

Go beyond activity analyses; involve workers' personal and organizational intentions, motivations, socialization.

Will the aid reduce manufacturing and development cycle times ("design for manufacturing")?

Consider the trade-offs between integrating several intellectual aids, rather than using intellectual separate aids.

FIGURE 11.4 Key points in studying ongoing intellectual work for the purposes of identifying the characteristics of useful intellectual aids.

lists some informal ones; these are described more fully in Gould (1988), with the emphasis upon designing computer applications. In addition, there is a substantial sociological and anthropological literature on participant observation and ethnography. All of these methods emphasize coming into direct contact with workers, their organizations, and the intellectual tasks they do or could perform. Failure to do this will lead to misunderstanding the characteristics of potentially valuable aids as surely as if one attempted to define an important aid for intellectual workers who live on other planets. Figure 11.4 provides further methodological guidelines.

The studies cited below are more valuable even than the following methods and results might indicate. Nearly every one provides a thoughtful discussion of the motivation, methods, results, and their implications.

Example of Studies
Work Activity Analyses

Observational studies, using a variety of methodologies, have summarized how office workers spend their time. For example, Klemmer and Snyder (1972) had observers note specific types of work behavior exhibited in a large research and development laboratory in offices, hallways, and other areas. In addition, employees answered questionnaires on how they

Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×

spent their time. They found that office professionals in the late 1960s spent most of their work time (42 percent) communicating interactively (talking face to face or on the telephone) and less time writing (14 percent) and reading (12 percent). They spent only 3 percent of their time interacting with office machines—which is certainly less than the time office workers spend today interacting with computer terminals. Mintzberg (1973) followed five very-high-level executives for two weeks each, observing and categorizing their activities, and wrote a widely read book summarizing the findings. Panko (1992) recently reviewed over 50 of these use-of-time studies and made general suggestions for office automation technology consistent with the main results.

While these studies, and many others, have illuminated how white-collar workers spend their work time, few intellectual work aids have so far emerged based upon insights from them, especially as related to the work of groups and organizations. What has been needed for some years are studies that relate these observed activities to personal and organizational motivations and goals, trigger events, and interruptions. New methodology is needed to go beyond "activity counts" and gain insights into work processes that can be aided. Although such studies are very hard to carry out correctly, they could provide insights into potentially valuable aids.

A related approach to these activity analyses has involved detailed coding and classification of verbal communications at the level of individual utterances. This approach has been used to analyze group interactions in flight simulations and in the investigation of aircraft accidents (e.g., Kanki and Palmer, 1993; Predmore, 1991). A more holistic approach has used experienced raters to evaluate interactions in real time using specific behavioral markers to exemplify effective and ineffective processes (Helmreich et al., 1990). Both methodologies can be useful in evaluating aids since they appear to be transferable to a variety of work situations of which the researchers have a detailed knowledge.

Desk Organization

To learn more about office work, Malone (1983) interviewed 10 professional and clerical workers about the way they organized their desks and offices. Participants first gave a verbal tour of their offices, explaining what information was where and why they put it there. As an indicator of their information-retrieval ability, the participants were then asked to find certain documents (chosen by a knowledgeable co-worker as being either easy or hard to find). Malone's motivation was to obtain insights into designing natural and convenient computer-based information systems to aid office workers' productivity. His two main conclusions (1983:99), which have implications for the design of intellectual aids, were the following:

Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×

(1) A very important function of desk organization is to remind the user of things to do, not just to help the user find desired information. Failing to support this function may seriously impair the usefulness of electronic office systems, and explicitly facilitating it may provide an important advantage for automated office systems over their non-automated predecessors. (2) The cognitive difficulty of categorizing information is an important factor in explaining how people organize their desks. Computer-based systems may help with this difficulty by (a) doing as much automatic classification as possible (e.g., based on access dates), and (b) including untitled "piles" of information arranged by physical location as well as explicitly titled and logically arranged "files."

The point is not so much to focus on these particular implications of Malone's study, but rather to appreciate how he drew implications for system design from his observational methodology. A second, more general, point has to do with the distinction between the need to design electronic aids (e.g., to replace paper) that allow people to work in about the same way they now work (an approach Malone's conclusions seem to favor), versus the need to design electronic aids that allow people to work in relatively new ways.

Communication in Software Development Organizations

Probably millions of people are employed in system development work. To understand better how to aid the coordination of work activities in system development, Kraut and Streeter (1995) studied the communication practices of people in such organizations. Hundreds of software professionals in the software development divisions of one company—including managers, analysts, software engineers, programmers, testers, and documentation specialists—completed a written survey. They reported that they got much of their important information directly from other people. When they had a large network of personal contacts outside the project, information flow improved, especially when the project was uncertain. They said that interpersonal techniques for getting information from beyond their immediate work group were underused, while more formal procedures for tracking routine information were overused in terms of their value. The design implication seems clear: aids to facilitate informal communication are needed. Kraut and Streeter note that planners are especially ill-served by current coordination techniques. Importantly, Kraut and Streeter conclude with valuable organizational and technological suggestions for increasing communication across project boundaries. This type of well-designed study should be imitated by future researchers.

Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Software Development

Many studies have contributed to a shared understanding of the social, cognitive, and behavioral characteristics of the software development process (e.g., Boehm, 1988; Jones, 1986). The study by Kraut and Streeter (1995), just described, and one by Curtis et al. (1988) show how human factors researchers can investigate very large task domains through surveys, visits, observations, and interviews. Using their findings, they can then develop theoretical understandings of work organizations, which in turn can lead to ideas about the characteristics of helpful electronic aids. Curtis et al. (1988) carried out extensive visits to workplaces and interviewed many individuals involved in 17 large software development projects. They found three general classes of behavioral problems that limited development productivity: thin spread of application domain knowledge, communication and coordination breakdowns, and conflicting requirements. It is possible to design new tools for software developers that address these problems.

Diagnostic Judgments

Diagnostic judgments and decision making are intellectual activities that cut across most work domains. People frequently make diagnostic judgments to answer important questions in their work lives (e.g., which person to hire? which applicants to admit to law school? what is the likely response from this drug treatment?), as well in their personal lives, which in turn affect their work lives (e.g., should I marry this person? should we have children? what is the best form of discipline to use with my children? at which supermarket should I shop?).

Consider a clinician examining a patient who is feeling ill, an employer selecting salespeople, a teachers' committee deciding whether to refer a youngster to special classes for learning disabilities, and a graduate school admissions committee selecting students for scholarships. If these "judges" base such difficult decisions upon years of personal experience as successful diagnosticians, intuition about particular cases, and knowledge of the individuals involved, then they are using the clinical method of decision making. This is said even if they base their judgment upon one or more quantitative predictors as well.

On the other hand, sometimes people rely only on statistical predictors to make judgments (the actuarial method). "To be truly actuarial, interpretations must be automatic (that is, prespecified or routinized) and based on empirically established relations" (Dawes et al., 1989:1668). The insurance industry, for example, predicts people's life expectancy (and therefore insurance rates) based only on a combination of statistical predictors. No

Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×

intuition or human element is involved. "In the clinical method the decision-maker combines or processes information in his or her head. In the actuarial or statistical method the human judge is eliminated and conclusions rest solely on empirically established relations between data and the condition or event of interest" (Dawes et al., 1989:1668).

Nearly all of over 100 studies comparing these two general approaches have shown that actuarial methods lead to more accurate diagnostic judgments than do clinical methods (Dawes et al., 1989). The topics studied included prediction of progressive brain damage, survival time following Hodgkin's disease, differential diagnosis of psychiatric disorders, length of psychiatric hospitalization, violent behavior, parolee's behavior after release, and graduate student performance. Even when clinicians are able to combine actuarial test results with their own interview data, the predictions based only upon the test results proved superior (Dawes et al., 1989). In other words, in this case, a third approach consisting of combining the two methods does not seem to help. This conclusion is not limited to clinical and social science situations. Murphy and Brown (1984) reported that "objective" (actuarial) weather forecasting is equal to or sometimes superior to "subjective" weather forecasting. The important point is that in straightforward ways, actuarial methods have the potential to be useful electronic aids, once the variables that correlate with successful predictions about the topics being judged are established.

There are elements of all three recommended research strategies in the aggregate of these studies. One problem for human factors research is how to put this conclusion into practice—where it presumably would have real value for society in many domains. To date, the results have had little impact. Dawes et al. (1989) cite several reasons, including lack of familiarity with the evidence, the continuation of previous practices in spite of the evidence, the feeling that an actuarial approach dehumanizes people, misconception, and inflated confidence about one's own ability to make accurate predictions (including situations in which the actuarial method, although superior to clinical judgment, achieves only modest results). Actuarial methods, when applied to new domains, can seem mechanistic, antidemocratic, inhuman. One task for human factors researchers is to figure out how to get people to internalize the research conclusion in this area so that it affects their behavior. McCauley (1991) notes that, when psychologists have to select National Science Fellows from a large list of applicants, they do not make use of available actuarial predictors.

A second problem for human factors research is how to extend these results to new domains (e.g., labor disputes, employee conflicts, system development planning). Here, one needs to identify the crucial predictors (which may already be known), carry out the empirical work when necessary, figure out ways to present the data effectively to potential users, assist

Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×

them in using the data, and design and carry out the necessary follow-up studies to verify whether the actuarial method helps in the new domain of concern.

Additional Approaches

There are several other approaches to studying ongoing intellectual work. There are many variations of task analysis methodology (see Drury et al., 1987). One can apply task analysis to the work of an isolated individual or to that of individuals working within an organization. Task analysis can be limited to ''logical analysis" of all the possible steps in a work process (as with an entirely new, nonexisting work system), or it can involve behavioral observations. Task analysis has been carried out mainly in military organizations, but there are many examples in industrial settings. The same handbook containing the Drury et al. paper contains chapters describing task analyses applied to process control systems (Woods et al., 1987) and to air traffic control systems (Lenorovitz and Phillips, 1987).

Another possible approach is to carefully study, through interviews, observations, task analyses, and other methods, experienced, successful workers performing crafts of high intellectual content (e.g., a physician diagnosing, a nurse treating, a musician composing, an architect designing, a teacher teaching). The goals would be to identify the intellectual skills involved, develop a theory about the task, and identify the characteristics of an aid that would presumably be helpful. Some designers find that, in the early stages of designing and developing a complex system, they want a list of classes of subtasks that most need augmenting or even automating. The challenge is to identify in general terms, if possible, these classes (e.g., spatial orientation, data interpretation, handling radioactive materials).

Another possible approach is to identify intellectual skills that cut across a large number of human intellectual tasks and then identify the characteristics of useful aids for that work. Existing examples of such aids are electronic verifiers for spelling and even grammar and electronic calculators for arithmetic. The idea is to determine not what the tasks have in common, but what common mental operations are carried out while working on a variety of tasks.

RESEARCH NEED AND STRATEGY 3: INTERDISCIPLINARY DEVELOPMENT WORK

Human factors researchers, to date, have been more likely to assess the value of an aid developed by others (e.g., the studies under strategy 1) than to join with others to develop an aid and then carefully assess its value and

Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×

impact. There is a need for human factors researchers to become involved in interdisciplinary efforts to create superior intellectual aids. This work is needed for several reasons.

First, working alone is generally not enough. It is hard to find examples of human factors researchers who, by themselves, develop effective electronic aids. Making good electronic aids requires many disciplines. Second, many engineers and scientists have come to realize that "human factors" determine the difference between success and failure of most aids. Third, there is evidence that human factors involvement in development projects is cost-effective (Karat, 1992). Fourth, iterative testing and improvement of proposed aids are vital if the aids are to increase productivity and enhance the quality of work life.

If human factors researchers are involved, such testing and improvement are likely to be carried out much more thoroughly. Thus, the potential aid under development has a greater probability of improving organizational or individual productivity or other aspects of work lives. Chapters 4 and 5 have made similar recommendations for interdisciplinary work in, respectively, emerging medical and environmental domains, areas that offer great rewards for successful collaboration. Fifth, successfully instantiating ideas into technology is a much bigger problem than most human factors researchers uninitiated in this process realize, and it is something they often cannot carry out on their own. This is true no matter how well the ideas seem to have worked in the experimental psychology laboratory or with prototypes. The studies referred to above on decision making and those described below by Landauer et al. (1993) illustrate this. Finally, many of the disciplines underlying human factors, such as computer science and experimental psychology, have a much larger supply of techniques, procedures, and results than there is a demand for by others. This proposed advanced development strategy can identify, select, and shape the techniques that others want and the context in which they must fit.

Human factors researchers should become partners with others in designing, developing, and iteratively improving good aids. This recommended strategy is not just a plug for employment opportunities for human factors researchers or a profession-oriented expression of the importance of human factors. The idea is not just to work in support roles, as most industrial and government human factors workers now do, but to work as equal, long-term participating partners, with all the commitment, risk, and pressure that leadership demands. The idea is that as co-partners, human factors researchers should develop a sense of responsibility for the initiation and successful completion of an interdisciplinary project to produce a significant aid, with all the burdens this entails. For many human factors researchers, this requires new learning, motivations, work organizations, and professional payoff

Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×

(particularly as related to the university promotion system). It requires a new conceptualization of what constitutes legitimate research. As outlined below, there are already some situations that can serve as organizational models of how to do the work and write it up for professional credit.

This recommended interdisciplinary work—wherein end users actually use, on a regular basis, the aid that human factors researchers jointly made with other necessary disciplines—may be hard to carry out because of personal and institutional barriers. It is longer-term work than is typical of many graduate school projects and may require longer-term commitments of funding than traditional research. To be successful, it will also require professional recognition and rewards, including ammunition for gaining tenure. There has been a start toward giving this work professional recognition. Case studies describing innovative research and development efforts involving human factors researchers in advanced development work exist, including Xerox's Star system (Smith et al., 1982), IBM's Audio Distribution System (Gould and Boies, 1984), and IBM's multimedia 1984 Olympic Message System (Gould et al., 1987). It is generally accepted that these systems were innovative and have influenced others who have made similar ones. Figure 11.5 summarizes key points to keep in mind carrying out interdisciplinary development work. These points are based upon personal experience and published case studies.

Key Characteristics about Effective Multidisciplinary Development Work

Strong technical leader and mentor, not lacking greatly in people skills—a technical champion.

Small group of talented people.

Individual members do what they are good at.

Commitment to colleagues.

Drive to become famous as a group for making an outstanding aid.

Willing to do what is necessary to be successful—no 80/20 or 90/10 rule, where the remainder is left to others.

Use iterative engineering, including good development tools.

Make early prototypes.

Involve intended users from the outset.

Use the aid that you are making yourselves whenever possible (if applicable).

Follow up, and measure the value of the aid created.

FIGURE 11.5 Key points in interdisciplinary work aimed at making effective aids for intellectual work.

Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Examples of Studies
SuperBook

Many developers believe that electronic computers hold greater potential than printed books for helping people find desired information. Yet Landauer et al. (1993), working at Bellcore, noted that nearly every experimental study failed to find an advantage for various electronically presented text systems over comparable printed versions. They concluded that a significant human factors problem is to develop an electronic text system that people will use to better advantage than a standard printed book.

Because of experimental results on the use of command languages and navigational tools, as well as results related to other relevant cognitive issues they had been studying (summarized in Landauer et al., 1993), Landauer et al. felt that they had a chance to change this history. They observed that much information already exists in published books and that this is the favorite medium of most authors. So, in contrast to most development efforts, which focus on electronic books only, they focused on how to help people find information in already published books that are also stored in computers. They called this approach SuperBook. They combined "full text indexing" with a "dynamic, hierarchical table of contents" of the stored book. A review of some of their previous research is in order.

Two fundamental problems that users have in trying to find helpful information in electronic databases have to do with the command language they use and how they navigate through databases.

With respect to command languages, almost all electronic information retrieval system—be they library subject catalogs, dial-up query systems, relational databases, or computer file systems—require users to select or enter some words that must match words used as identifiers by the authors or indexers. After many observations, Landauer et al. became convinced that most occasional users are usually unsuccessful in using standard query languages or in producing Boolean expressions from ands and ors to get desired information from databases. Thus, they rejected this approach as a way of helping casual users of electronic data bases.

Why not discover what words people naturally prefer to use? When trying to devise new command names that would help novices learn word processors, the Bellcore group (Landauer et al., 1993:76) decided:

that any two people were unlikely to agree on a best term, so the common expedient of a one word command, file name or table label will usually fail to put a user in touch with data stored by someone else. … The chances that any two people would choose the same name for a command were less than twenty percent, and the most popular term was chosen by only thirty percent of potential users. This observation was extended to the use of

Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×

terms for queries in data bases of several kinds, from classified ad listings to recipe files and program names.

The Bellcore researchers built upon these observations by experimentally evaluating inventions of their own derived from their results. For example, unlimited aliasing (i.e., using every term that a sample of users apply to an item as an index to that item) led to a hit rate of over 70 percent in an interactive retrieval task on a small database of 188 recipes, with little increase in false positives. The same level of success was achieved with full text indexing (i.e., every word contained in each item in the database is part of the index for that item). Unfortunately, these results do not scale up to much larger databases because too many irrelevant items (i.e., false positives) are returned. Note how these researchers were not misled by optimistic lab results based upon limited tasks.

With respect to navigation, the Bellcore experimental studies (Landauer et al., 1993:80) found that people have fundamental problems when using menus to navigate through electronic databases:

Typically menu selection schemes have a branching factor of four to ten options; more often than not, users stray from the path to their target by the third hierarchical level. Put differently, if there are more than about thirty items in the data base, the chances of finding the right one in a single error-free traversal drop below half.

They concluded provisionally that (Landauer et al., 1993:82):

menu-traversal methods are fundamentally, and probably irremediably flawed as a primary method for information retrieval. As conceived by humans, almost all categories are essentially fuzzy; most items belong only partially or with only moderate probability to any one category, and can often fit reasonably well into several.

The Bellcore group also retained a "rather cautious use of links in the SuperBook system" (Landauer et al., 1993:129).

Thus, as the experiments showed, neither full text indexing nor a dynamic, hierarchical table of contents is sufficient by itself, but the Bellcore group thought a combination of the two might help. Full text indexing returns a superset of desired items, as they had already found, but the dynamic table of contents, they reasoned, should provide contexts to help users narrow this superset of items to those that are relevant to their intentions. So they built a prototype, and used it themselves (Landauer et al., 1993:98):

We were … perfectly delighted with our system design. We were so overwhelmed with its obvious superiority to anything we had previously seen, and so pleased with our early in-house use that we dubbed it "SuperBook." We immediately began technology transfer efforts, efforts to make

Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×

this marvelous new tool available to other organizations at Bellcore. … Fortunately, [a new group member] reminded us of some of our public pronouncements about, and past experiences with the necessity of evaluation studies.

To judge how well the SuperBook actually aided people, the Bellcore group gave college students queries which they had to answer using either a published statistics book or its electronic clone.

Using the first version of the electronic aid (version 0), participants found slightly more correct answers with SuperBook than with the original printed book, but took significantly longer to do so. So the Bellcore group made many changes to its electronic aid, most based upon analyses of its experimental data. With the next version, participants answered their queries faster and more accurately with SuperBook than with the original printed book. Based upon what they learned in this experiment, the Bellcore researchers made more changes to their electronic aid and tested it again. This third evaluation study found that participants performed even better with SuperBook.

This interdisciplinary effort illustrates many important points. No one discipline could have done all the design, implementation, testing, and transfer efforts. Human factors researchers, like experts in other disciplines, can be overenthusiastic about their own inventions when empirical data are absent. Development of a successful aid usually takes a long time. The work shows the potential payoff of interdisciplinary effort directed at a very hard problem. The work reflects elements of all three recommended research strategies.

The experimental evaluations showed the status of the aid and suggested possible improvements. And the testing still goes on: Landauer et al. (1993) have carried out several other experimental evaluations with other materials and have made SuperBook available to over 40 research labs.

Aid for Detecting Drunk Drivers

Harris (1980) designed and carried out a study to develop an aid to help police officers spot drunk drivers. In the end, Harris provided officers in patrol cars with a low-tech, effective aid—a list of driver behaviors that best predict which motorists are driving while intoxicated (DWI). More important here is the process that led up to his designing, implementing, and evaluating this aid. His study is valuable for its methodology, completeness, impact, clear write-up, and (inexpensive) aid developed to help police select which drivers to stop.

In general, six percent of drivers at night have a blood-alcohol level of 0.10 or greater (most states define this as DWI), and another 9 percent have

Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×

a blood-alcohol level between 0.05 and 0.10 (some states define this as "driving while impaired").

First, Harris studied 1,288 DWI arrests throughout the United States, analyzing the reported driver behaviors associated with each of these arrests. These driver behaviors could all be observed by police officers making the arrests (e.g., swerving, straddling) and could therefore serve as visual cues to aid in deciding whether or not to stop a potentially drunk driver. Harris also rode with officers in two states as they stopped drivers, and he noted the driving behaviors of those motorists and the conditions under which they were stopped. Nearly all of the stopped drivers (93 percent) were given a blood-alcohol test when they were stopped: 38 percent had a blood-alcohol level greater than 0.10 (DWI), and another 23 percent had a blood-alcohol level between 0.05 and 0.10. (This 61 percent is already 4 times higher than the 15 percent that could be expected by chance.) From these analyses he produced a drunk driver detection guide, a small card listing several visual cues (e.g., following too closely) and the percentage of drivers with this behavior who have a blood-alcohol level greater than .01 (e.g., 60%). The card was for police to carry with them or to attach wherever convenient. Harris then went on to study the value of this aid by studying 10 law enforcement agencies across the United States. before and after they used his guide. He found that DWI arrest rates increased significantly overall (from 66 to 74 percent; p < .01) for these 10 agencies. (No mention is made of false positives—arrests that did not lead to convictions.) Harris (1980:731) also reported:

Experienced police officers … were generally skeptical that … the guide would enhance their own DWI detection ability. However, most officers considered the guide to be a valuable aid for increasing patrol awareness of useful detection cues, training inexperienced patrol officers, preparing DWI arrest reports, and supporting court testimony.

Other Approaches

The Landauer et al. (1993) studies provide an excellent example of extending the cognitive findings of one's own group to the design of a real electronic aid. Human factors researchers could take the initiative to extend other existing behavioral and cognitive knowledge to applied domains (a task much harder than many experimental and cognitive psychologists generally suppose). For example, for some years many secretaries and professionals have used commercially available computer-executable, rule-based styles to lay out and format their documents, for example, in prespecified fonts, margins, number of columns, line spacing, headings, and graphics. Now, if there exists behavioral knowledge of how to design textbooks and related reading material to enhance learning, human factors people, working

Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×

with computer scientists, could develop a similar rule-based style to lay out documents according to this knowledge. Another example would be to extend the results on human decision making, described above, to real-world aids. The Landauer et al. (1993) studies provide an excellent example of extending one's own cognitive findings to the design of a real electronic aid.

Another approach might take existing technology or aids and combine them in behaviorally innovative ways. Perhaps the Xerox Star system (Smith et al., 1982), the highly influential graphic user interface developed at Xerox Parc in the 1970s, is an example of this. The developers took existing hardware (bit-mapped displays, mouse), existing software techniques, behavioral knowledge, and cognitive engineering models and integrated them into something much greater than the sum of the parts. Many observers agree that this work influenced subsequent graphic user interfaces made by other major corporations.

ACKNOWLEDGMENTS

I thank Liz Gould, Bob Helmreich, Bob Kraut, Tom Landauer, Ray Nickerson, Lynn Streeter, Jacob Ukelson, and Frank Yates, who provided helpful written reviews and suggestions to improve this chapter, and several other committee colleagues for additional suggestions.

REFERENCES

Attewell, P.A. 1994 Information technology and the productivity paradox. Pp. 13-53 in D.H. Harris, ed., Organizational Linkages: Understanding the Productivity Paradox. Panel on Organizational Linkages, Committee on Human Factors, National Research Council. Washington, D.C.: National Academy Press.


Bentley, R., J.A. Hughes, D. Randall, T. Rodden, P. Sawyer, D. Shapiro, and I. Sommervile 1992 Ethnographically-informed systems design for air traffic control. Pp. 123-129 in J. Turner and R. Kraut, eds., Proceedings of the Conference on Computer-Supported Cooperative Work. New York: ACM Press.

Berlin, L.M., and R. Jeffries 1992 Consultants and apprentices: observations about learning and collaborative problem solving. Pp. 130-137 in J. Turner and R. Kraut, eds., Proceedings of the Conference on Computer-Supported Cooperative Work. New York: ACM Press.

Boehm, B.W. 1988 Improving software productivity. IEEE Computer September:43-56.


Card, S.K., W.K. English, and B. Burr 1979 The Keystroke-Level Model for User Performance Time With Interactive Systems. Report SSL-79-1. Palo Alto, Calif.: Xerox Palo Alto Research Center.

Card, S.K., J.M. Robert, and L.N. Keenan 1984 On-line composition of text. Pp. 231-236 in Proceedings of Interact'84, First IFIP Conference on Human-Computer Interaction, Vol. 1. London, England: Elsevier North Holland.

Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×

Cool, C., R.S. Fish, R.E. Kraut, and C.M. Lowery 1992 Iterative design of video communication systems. Pp. 25-32 in J. Turner and R. Kraut, eds., Proceedings of the Conference on Computer-Supported Cooperative Work. New York: ACM Press.

Curtis, B., H. Krasner, and N. Iscoe 1988 A field study of the software design process for large systems. Communications of the ACM 31:1268-1287.

Dawes, R.M., D. Faust, and P.E. Meehl 1989 Clinical versus actuarial judgment. Science 243:1668-1673.

Druckman, D., and R.A. Bjork, eds. 1991 In the Mind's Eye: Enhancing Human Performance. Committee on Techniques for the Enhancement of Human Performance, National Research Council. Washington, D.C.: National Academy Press.

Druckman, D., and J.A. Swets, eds. 1988 Enhancing Human Performance: Issues, Theories, and Techniques . Committee on Techniques for the Enhancement of Human Performance, National Research Council. Washington, D.C.: National Academy Press.

Drury, C.G., B. Paramore, H.P. Van Cott, S.M. Grey, and E.P. Corlett 1987 Task analysis. Pp. 370-401 in G. Salvendy, ed., Handbook of Human Factors. New York: Wiley.


Ferber, M.S., B. O'Farrell, and L. Allen, eds. 1991 Work and Family: Policies for a Changing Work Force. Panel on Employer Policies and Working Families, Committee on Women's Employment and Related Social Issues, National Research Council. Washington, D.C.: National Academy Press.


Gould, J.D. 1978a An experimental study of writing, dictating, and speaking. Pp. 299-319 in J. Requin, ed., Attention and Performance VII. Hillsdale, N.J.: Lawrence Erlbaum Associates. 1978b How experts dictate. Journal of Experimental Psychology: Human Perception and Performance 4(4):648-661. 1980 Experiments on composing letters: some facts, some myths, and some observations. Pp. 97-118 in L. Gregg and E.R. Steinberg, eds., Cognitive Processes in Writing. Hillsdale, N.J.: Erlbaum and Associates. 1982 Writing and speaking letters and messages. International Journal of Man-Machine Studies 16:147-171. 1988 How to design usable systems. Pp. 757-789 in M. Helander, ed., Handbook of Human-Computer Interaction . Amsterdam, Netherlands: North-Holland, Elsevier Science Publishers.

Gould, J.D., and S.J. Boies 1978 How authors think about their writing, dictating, and speaking. Human Factors 20:495-505. 1983 Human factors challenges in creating a principal support office system: the speech filing system approach. ACM Transactions of Office Information Systems 4(1):273-298. 1984 Speech filing: an office system for principals. IBM Systems Journal 23:65-81.

Gould, J.D., S.J. Boies, S. Levy, J.T. Richards, and J.W. Schoonard 1987 The 1984 Olympic Message System: a test for behavioral principles of system design. Communications of the ACM 30(9):758-769.


Harris, D.H. 1980 Visual detection of driving while intoxicated. Human Factors 22(6):725-732.

Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×

Helander, M., ed. 1988 Handbook of Human-Computer Interaction. Amsterdam, Netherlands: North-Holland, Elsevier Science Publishers.

Helmreich, R.L., J.A. Wilhelm, S.E. Gregorich, and T.R. Chidester 1990 Preliminary results from the evaluation of Cockpit Resource Management training: Performance ratings of flightcrews. Aviation, Space and Environmental Medicine 61(6):576-579.


Jones, T.C. 1986 Programming Productivity. New York: McGraw-Hill.


Kanki, B.G., and M.T. Palmer 1993 Communication and crew resource management. In E.L. Wiener, B.G. Kanki, and R.L. Helmreich, eds., Cockpit Resource Management. San Diego, Calif.: Academic Press.

Karat, C. 1992 Cost-justifying human factors support on software development projects. Human Factors Society Bulletin 35(11):1-4.

Klemmer, E.T., and F.W. Snyder 1972 Measurement of time spent communicating. Journal of Communication 22:148-158.

Kraut, R., S. Dumais, and S. Koch 1989 Computerization, productivity, and quality of work-life. Communications of the ACM 32(February):220-237.

Kraut, R.E., and L.A. Streeter 1995 Coordination in software development. Communications of the ACM 38(3):69-81.


Landauer, T., D. Egan, J. Remde, M. Lesk, C. Lochbaum, and D. Ketchum 1993 Enhancing the usability of text through computer delivery and formative evaluation: the SuperBook project. In C.M. McKnight, A. Dillon, and J. Richardson, eds., Hypertext: A Psychological Perspective . New York: Horwood.

Lenorovitz, D.R., and M.D. Phillips 1987 Human factors requirements engineering for air traffic control systems. Pp. 1724-1770 in G. Salvendy, ed., Handbook of Human Factors . New York: Wiley.


Malone, T.W. 1983 How do people organize their desks: implications for designing office automation systems. ACM Transactions on Office Automation Systems 1:99-112.

McCauley, C. 1991 Selection of National Science Foundation Graduate Fellows. A case study of psychologists failing to apply what they know about decision making. American Psychologist December:1287-1291.

Mintzberg, H. 1973 The Nature of Managerial Work. New York: Harper and Row.

Murphy, A.H., and B.G. Brown 1984 A comparative evaluation of objective and subjective weather forecasts in the United States. Journal of Forecasting 3:369-393.


Norman, D. 1988 The Psychology of Everyday Things. New York: Basic Books.


Olson, J.S., G.M. Olson, M. Storrosten, and M. Carter 1992 How a group-editor changes the character of a design meeting as well as its outcome. Pp. 91-98 in J. Turner and R. Kraut, eds., Proceedings of the Conference on Computer-Supported Cooperative Work . New York: ACM Press.

Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×

Panko, R.R. 1992 Managerial communication patterns. Journal of Organizational Computing 2(1):95-122.

Pool, R. 1992 Computing in science. Science 256(April 3):44-64.

Predmore, S.C. 1991 Microcoding of communications in accident analyses: crew coordination in United 811 and United 232. Pp. 350-355 in Proceedings of the Sixth International Symposium on Aviation Psychology. Columbus: Ohio State University.


Roberts, T.L. 1979 Evaluation of Computer Text Editors. Unpublished PhD dissertation. Computer Science Department, Stanford University.


Smith, D.C., C. Irby, R. Kimball, B. Verplank, and E. Harslem 1982 Designing the star user interface. Byte 7(4):242-282.


Turner, J., and R. Kraut, eds. 1992 Proceedings of the Conference on Computer-Supported Cooperative Work. New York: ACM Press.


Wiener, E.L., T.R. Chidester, B.G. Kanki, E.A. Palmer, R.E. Curry, and S.E. Gregorich 1991 The Impact of Cockpit Automation on Crew Coordination and Communication. I. Overview, LOFT Evaluations, Error Severity, and Questionnaire Data. NASA Contractor Report No. 177587. Moffett Field, Calif.: NASA-Ames Research Center.

Woods, D., J.F. O'Brien, and L.F. Hanes 1987 Human factors challenges in process control: the case of nuclear power plants. Pp. 1724-1770 in G. Salvendi, ed., Handbook of Human Factors. New York: Wiley.

Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Page 291
Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Page 292
Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Page 293
Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Page 294
Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Page 295
Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Page 296
Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Page 297
Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Page 298
Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Page 299
Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Page 300
Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Page 301
Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Page 302
Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Page 303
Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Page 304
Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Page 305
Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Page 306
Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Page 307
Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Page 308
Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Page 309
Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Page 310
Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Page 311
Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Page 312
Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Page 313
Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Page 314
Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Page 315
Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Page 316
Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Page 317
Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Page 318
Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Page 319
Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Page 320
Suggested Citation:"11 Aiding Intellectual Work." National Research Council. 1995. Emerging Needs and Opportunities for Human Factors Research. Washington, DC: The National Academies Press. doi: 10.17226/4940.
×
Page 321
Emerging Needs and Opportunities for Human Factors Research Get This Book
×
Buy Paperback | $57.00 Buy Ebook | $45.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

This book identifies areas that represent new needs and opportunities for human factors research in the coming decades. It is forward-looking, problem oriented, and selectively focused on national or global problems, including productivity in organizations, education and training, employment and disabilities, health care, and environmental change; technology issues, including communications technology and telenetworking, information access and usability, emerging technologies, automation, and flexible manufacturing, and advanced transportation systems; and human performance, including cognitive performance under stress and aiding intellectual work.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!