Research Needs for Human Factors (1983)

As part of an engineering team, human factors specialists apply their knowledge and skills to system definition, design, development, and evaluation in order to optimize the capabilities and performance of human-machine combinations. Their task can be formidable in complex system development. For example, military standard MIL-H-46855B of the Department of Defense details the human factors requirements that must be addressed in the development of military systems; an outline of these requirements appears as Figure 7–1. The outline is also a reasonable representation of the human factors considerations that may be relevant to the development of any system.

In designing and creating systems human factors specialists use a variety of analytic and data-gathering techniques to assess problems, develop machine and human requirements and functions, and evaluate system or subsystem performance. Although many of these problems would ideally be solved with the experimental methods

used in scientific research, practicing human factors specialists rarely have the luxury of using properly counterbalanced experimental designs, with a range of levels of factors and the precise control of unmanipulated variables. This is not to minimize the importance of experimental methods which are used whenever possible and have provided much of the basic data in human factors handbooks. However, applied methods are necessary both as suplements to experimental methods, e.g., for problem analysis and structuring, and as substitutes when the pressures and constraints of the engineering design environment preclude experimental investigations.

Most practical work in human factors is done under conditions that involve the incomplete specification of system functions, complex combinations of conditions that cannot be separated or controlled, restricted sets of alternatives, limited time and opportunities for investigation, and pressure to produce definitive results quickly. From necessity, human factors specialists have evolved an armamentarium of applied methods that are appropriate to these conditions and that are unfamiliar to most academic researchers. These applied methods are formal means for acquiring or organizing information about human factors characteristics that arise in the context of system design, development, and evaluation.

Applied methods are diverse, reflecting the many purposes for which human factors information is used. Some of them come from psychology, for example, questionnaires and techniques for acquiring, summarizing, and analyzing data. Some have been borrowed, with or without modification, from other fields, such as industrial engineering and time and motion engineering. For example, analytic methods draw heavily on the engineering practice of systems analysis, which identifies inputs, outputs, the functions performed, the range of values that variables may assume, process flow, the sequence of events, and the timing of the interrelations of system components. Other methods, such as the critical incident technique and link analysis, appear to have been created by human factors specialists to meet their needs in solving particular problems.

Whatever their origins, applied methods have been developed as tools to help answer questions when there are constraints of time, dollars, and freedom of action and when experimental methods are not suitable to answer the questions that arise in system development. Although it is characteristic of applied methods that they make it

FIGURE 7–2 Applied Method Names Appearing in Keyword Lists of Articles in Human Factors Between 1976–1981

possible to acquire and produce data and information only to the degree of resolution and reliability sufficient for a particular purpose, these methods are systematic and objective procedures. That is, the procedures are repeatable and input and output data are operationally defined.

The importance of applied methods in human factors work is clear from the number of technical reports and journal articles that discuss one or more applied methods. Two recent reports (Williges and Topmiller, 1980; Geer, 1981) list human factors procedures necessary for Air Force system analysis, design, and evaluation; the latter report gives brief descriptions and critiques of approximately 48 human engineering procedures, the majority of which are applied methods. Figure 7–2 lists applied methods that appeared in keyword lists of articles published between 1976 and 1981 in Human Factors, the journal of the Human Factors Society.

Despite this wide variety of applied methods, there is general agreement among human factors specialists that we need to improve existing methods and develop new ones (Topmiller, 1981; Meister, 1982). Advances in technology, particularly in the speed, power, and memory of computers, have generated concern recently with the human factors elements of computer software. At the same time, the explosive growth of computer use, with resultant increases in the complexity and integration of system components, the automation of functions, and the use of artificial intelligence, all have profound methodological implica-

tions for the analysis and description of the role of humans and computers in such systems.

Applied methods have never previously been treated as a single topic deserving attention in its own right.^★ Consequently, information has never been gathered on the number and varieties of applied methods available and the frequency and adequacy with which they are used. The workshop held by the Committee on Human Factors, on which the discussion in this chapter is based, was an attempt by committee members and a group of acknowledged experts in applied methods to identify problems and needs with respect to applied methods. Even in the absence of data on the variety and frequency of use of applied methods, we have been able to identify several major problems and to recommend solutions, which may make substantial improvements in practice possible. Three major problems are discussed: (1) the lack of adequate documentation; (2) the limited opportunities available to learn applied methods, either in colleges and universities offering human factors courses or as part of the continuing education of human factors specialists; and (3) the lack of research to improve existing methods and to develop new methods that will provide the data and information needed in current and future practical human factors work.

The practical work of human factors specialists, unlike scientific research, does not result in an orderly progression and an orderly accumulation of knowledge. Human factors projects (i.e., participation in the design of systems) and the solution of special problems come and go in great variety. Typically work is performed, reported, and forgotten as new systems and problems develop. Codified, archival repositories of practical work—i.e., review books and articles that summarize the knowledge and procedures used in human factors applications to some point in time—are rare. As a result the historical memory of human factors methods resides largely in the heads and in the report files of practitioners. By contrast, in the literature on scientific research, the

methods used by investigators are maintained and disseminated in the curricula of university departments and preserved on library bookshelves.

As an important first step toward improving knowledge about and use of applied methods, we therefore recommend that one or more projects be initiated to compile and review the available information on applied methodologies used in human factors and related fields, such as industrial and organizational psychology, personnel selection, and instructional psychology. The object of the review would be to determine what methods have been used, how they have been used, where they are used, and what their advantages and disadvantages are. The project should also include a critical analysis of the methods. Other purposes of the review would be to structure or codify the methods and to document them for subsequent educational and research purposes.

It would also be extremely valuable to practitioners, educators, and researchers in human factors to have a compendium that codifies and provides standard or generic descriptions of applied methods that are used in practical human factors work. Development of such a compendium would require a great deal of judicious and careful effort. One of the primary difficulties would be to decide which methods are viable, valid, and useful. Because such a compendium would necessarily be an implicit endorsement of the methods described, we recommend that eight criteria be used in the selection process. Methods that meet the criteria listed below could be regarded as having sufficient stature to be of value in a variety of human factors applications:

Importance—Does the method produce needed information?

Cost—Is the method efficient in terms of effort and time?

Utility—Can procedures for using the method be easily interpreted and implemented?

Available Input Parameters—Can the necessary data be collected in a direct, objective, and reliable way?

Usable Output—Does the method produce results that are interpretable and useful for decision making?

Validity/Verification—Can or has the method been found to produce the information it is supposed to?

Theoretical Foundation—Is the method supported by accepted behavioral or measurement principles?

Robustness—Can the method be applied to a variety of problems or in different contexts?

These criteria imply that the approach to documenting standard definitions of applied methods should be conservative. That is, only those methods for which there is evidence of practicality and validity should be selected for inclusion in a compendium. Methods used in workload assessment provide an example of the importance of using these criteria. Measurement of workload is a current topic of intense research interest; consequently a large number of theories, approaches, and positions have been put forward. Since most of the recent work has not been validated through practical application, it would be inappropriate to describe them as standard, accepted methods. Older methods exist for assessing imposed workload that, while perhaps wanting in certain respects, have been proven through repeated use to be practical, reliable, and valid (Parks and Springer, 1976) and are likely to meet our criteria. Nevertheless, there will be hard choices to make in deciding what constitutes an accepted, standard form of a method.

Multiple variations of a method should probably not be included. A compendium that includes only a set of core methods that meet the criteria would be of great value for both practical work on system development and as a foundation for the education of human factors students at colleges and universities. Attempting comprehensive coverage of all variations of methods would unnecessarily complicate the task of documentation and delay the compilation, causing confusion and consequently inhibiting its acceptance. A single, solid definition of each particular method would be most useful, since by its nature an applied method undergoes some variation in each instance of its use because of the requirements and constraints of a particular project. In the meantime, additional documentation and research to extend or refine the standard methods can be carried out.

In the course of compiling a reasonably comprehensive list of the most generally known applied methods (see Figure 3), it became apparent that the methodologies could be grouped into five categories according to their purpose. Five categories of applied methodologies seem appropriate: analysis, identification of needs, data

collection, prediction, and evaluation. Each methodology appears only under one heading, although several of them are appropriate to more than one category.

The organization of Figure 7–3 is probably a useful guide to the scope of work involved in documenting applied methods. The categories reflect a sequence of methods used, from the early concept definition of a system to its evaluation. There is also a rough correlation between the difficulty and detail involved in particular methods and the stage of application in the process of system development.

Documentation of applied methods necessarily requires review of the technical literature to extract descriptions of applied methods. To expect a single or a small group of experts to adequately review and document the entire range of applied methods would be impractical; a more feasible approach would be to subdivide the work according to the five categories of purpose. The individual tasks would thereby be more tractable and make better use of the skills of individuals whose knowledge and expertise is likely to be confined to a single category rather than the full range of methods. This approach would also allow the work on each subset of methods to be performed concurrently. Whatever the approach taken, producing a compendium of standard, usable descriptions of proven applied methods would be an extremely valuable contribution to the field of human factors and consequently to the future development of human-machine systems.

Because of the dearth of information on the variety and use of applied methods in human factors work we recommend a survey of human factors practitioners concerned with the acquisition, design, development, and evaluation or modification of equipment and systems. Such a survey would determine the importance and frequency of use of existing applied methods in their work; the kind of information most needed in human factors applications for which existing applied methodologies are inadequate or nonexistent; and the methods for which descriptions and guidance for use are most needed.

The survey would provide the necessary information on which to base documentation, education, and research efforts. Review, codification, standardization, and documentation of existing methods should proceed

FIGURE 7–3 Generally Known Applied Methods Categorized by Purpose

according to the priorities of importance and frequency of use derived from the survey. Information from the survey would be useful in shaping human factors curricula in colleges and universities so that students can be trained in applied methods that they will subsequently need on the job. The continuing education needs of human

factors specialists could also be met by means of tutorials and symposia on the applied methods for which there is the greatest need for information. Finally, the results of the survey would provide a sound basis for basic research efforts to extend or improve existing methods or develop new methods to meet these needs.

Construction of the survey instrument itself would require a review of the technical literature for descriptions and definitions of applied methods, which the survey recipients would be expected to rate. The literature review would also provide additional data, complementary to the anticipated survey, on the variety and frequency of use of applied methods reflected in the technical literature. A product of this review would be a relatively comprehensive bibliography of technical reports and journal articles that discuss applied methods in more than a cursory fashion; this bibliographic information would be extremely valuable for subsequent efforts on the codification and documentation of existing methods and the initiation of research efforts to extend these methods or develop new ones.

The absence of codified information and the lack of easy access to source reports inhibits instruction in applied methods at colleges and universities that offer degree programs or courses in the field of human factors. General human factors textbooks give at best only a cursory overview of a few applied methods and present case study examples that highlight the substantive issues and results rather than the methods. There are no texts suitable either for college-level instruction or as a reference for practicing human factors specialists that adequately treat applied methods. The single exception, Research Techniques in Human Engineering (Chapanis, 1959), discusses only a limited set of methods. For the most part, instructors must rely on their own experience and the descriptions of applied methods gleaned from the technical literature to develop course material. They have no current and comprehensive reference works to develop a balanced and thorough course in applied methods.

Human factors work is diverse and is performed in many settings—i.e., military research and development centers, other government facilities, and commercial organizations. Ideally, instruction in applied methods would emphasize the methods of most use in real-life settings. Without data on the variety and frequency of use it is difficult to decide which applied methods should be taught in human factors courses at the undergraduate and graduate levels. Clearly the development of a compendium of applied methods, as recommended in the previous section, would be of substantial benefit for formal educational purposes. Until such a compendium exists and survey data is compiled on the variety, frequency of use, and capabilities of applied methods, no meaningful recommendations can be made to improve education in applied methods in colleges and universities.

Of equal concern is the lack of suitable continuing education courses in applied methods for practicing human factors specialists. The problem of inadequate methodological preparation in formal education extends to the work setting. At present it appears that many presumably well-trained human factors specialists work without adequate knowledge of applied methods, and what knowledge they do have about these methods is acquired on the job.

Currently employed human factors specialists could benefit greatly from continuing education in applied methods specifically related to their current work. Development at colleges and universities of educational programs in applied methods that provide a thorough treatment of a range of applied methods would require a substantial amount of planning and course design work. Undoubtedly the broad inception of these programs, and the realization of their eventual benefits in practice, will be some time in coming. Unlike formal education in applied methods, however, the development of courses for continuing education could be done more easily and produce more immediate positive effects. Human factors professionals are likely to be more easily educated because of their general knowledge of human factors techniques and the likelihood that they have at least a working familiarity with some applied methods. Because of their previous education and experience, continuing education courses for them can be much more practical,

with less emphasis on theoretical foundations. Based on the membership of the Human Factors Society, which numbers nearly 3,000, a reasonable estimate of the actual number of practicing human factors specialists in this country who could benefit from continuing education in applied methods is between 5,000 and 10,000.

Fostering and promoting continuing education by means of tutorials on applied methods is one of the most important and immediate ways to improve the field of human factors. Moreover, this kind of activity could most easily be initiated by military and other federal agencies charged with advancing scientific and engineering knowledge and practice. These tutorials could directly benefit human factors specialists employed by the government as well as those employed by civilian organizations that develop equipment and systems for the government. It is therefore recommended that initial tutorials on applied methods be developed and conducted under the sponsorship of one or more government agencies. While we suggest methods to be discussed in the tutorial below, it would be more prudent to base the choice on a needs analysis of the data derived from the survey recommended above.

Such a tutorial could serve several purposes besides the obvious one of improving the professional competence of human factors specialists. First, the materials generated for the tutorial would contribute to the development of standard definitions and documentation of applied methods, since the course materials would have to describe the subject methods with sufficient care and detail to allow human factors specialists to use them easily and properly. Second, the tutorials would be a means for validating a prior needs analysis of which applied methods are considered most important to human factors practitioners. Attendance at the tutorials would also help answer a more fundamental question: Is there genuine interest in learning about applied methods? Third, the initial tutorial would serve as a test to evaluate instructional methods and course structures for training in the use of applied methods.

It is suggested that the initial tutorial should consist of three parts: (1) an introductory review of the applied methodologies within each of the five categories listed in Figure 7–3; (2) a comparison of techniques within each category and a discussion of how to select the appropriate method for a particular application; and (3) detailed instruction and practical

work on a few selected methods. We suggest five particular methodologies as subjects for the initial tutorial:

Because these methods as well as others are either poorly or inconsistently defined, brief definitions of the five methods recommended for the first tutorial are given in Appendix A. It would not be practical to cover more than five methodologies at the initial tutorial; five may even be too many.

There are a number of other specific concerns relevant to the form and development of a tutorial on applied methods. Experience has shown tutorials to be only the first step in learning to use a particular technique properly. Generally, an individual needs several days of supervised application to become competent in using a particular method. Therefore, the tutorial should not be simply a symposium but rather should be a workshop in which the attendees could gain hands-on experience. A by-product of the initial tutorial would be the development and testing of the structure and effectiveness of the initial instructional methods.

A tutorial on applied methods would probably require 10 to 40 hours of planning and preparing for each hour of instructional time. Since the tutorial should include practical workshop exercises in addition to lecture, a good part of the effort of preparation would have to be devoted to development of materials. It is likely that the practicum would require one or more assistants in addition to the instructor.

An individual or small group should be selected to develop a master plan for the tutorial workshop. The primary goal would be to choose the methods to be taught in the tutorial. This determination should be based largely on the needs analysis of the data gathered from the methods survey of human factors practitioners recommended above. The individual or group should also address such issues as the number of days the tutorial should run, whether it should be conducted independently or in association with a national meeting, the estimated costs, and the selection of instructors.

The most obvious audience for the first tutorial are human factors practitioners, although the needs of other groups of professionals that could benefit from learning about applied methods, such as engineers, managers, students, and university teachers, should be considered at some point. Engineers are an important audience since they are likely to need to use applied methods in the course of system design and development and they are not likely to know where to seek information on methodologies. Managers are important because of their influential role in equipment and system development. Due to their position of authority, managers are able to influence practices of their employees. College and university teachers are a relevant audience, since what they learn would be passed on to their students. And students, especially students in engineering and human factors, are a particularly important potential audience because of their receptivity to new techniques and the apparent lack of adequate education in applied methods in colleges and universities.

The tutorial format appropriate for human factors professionals may not be suitable for these other groups. If the first tutorial proves to be beneficial to human factors specialists, it would be worthwhile to design others tailored to the backgrounds and needs of these other groups. We recommend that tutorials for these other groups be developed first for engineers and subsequently for the remaining groups.

For all audiences the tutorials should be repeated at several times and locations both to make the experience available to all who are interested and to recover the initial development costs.

Each applied method was originated to fill some particular need for information to support system design, evaluation, or problem analysis. Through a succession of repeated, successful use in different contexts, methods have evolved and have become known and accepted as tools of the trade in human factors work. Because they were developed as a means to some practical end and so vary in form depending on the situations in which they are used, there has never been very much concern about their refinement or extension. That is, an applied method has rarely been regarded as an important topic worthy of research investigation in

its own right, independent of a particular use. This lack of status is partly reflected and partly caused by the absence of standard documentation of applied methods. In addition, the people who use applied methods are practitioners and, in some sense, generalists in human factors rather than specialists in methodology. There is no body of experts who devote their careers to the study and development of applied methods rather than their actual use, as there is for experimental design and statistical analysis.

Applied methods, however, are the principal means by which human factors work is accomplished. In light of their contibution to systems work, applied methods are a sufficently important topic to deserve research attention. Advances should not depend solely on incidental efforts made by human factors specialists in the course of their work. Basic research specifically devoted to the validation, refinement, and extension of existing methods and to the development of new methods is essential.

As previously discussed, fundamental problems are the lack of documented definitions and descriptions of existing applied methods and the lack of knowledge about what information is needed in human factors work. Documentation and survey work is necessary to provide baseline descriptions and to help identify the particular problems and shortcomings of existing methods.

Without this information it is difficult to specify what research on which particular methods would have the greatest value in terms of its contribution to the improvement of human factors work. Nonetheless, we propose some existing methods as subjects deserving research attention because from our experience it is apparent that these methods are widely used, critical to system design and development work, and could be substantially improved: workload analysis; function allocation; task analysis; survey techniques; and protocol analysis.

Workload analysis is already the subject of many ongoing research programs; however, it is important enough to merit expanded support for research on workload assessment methods. While the five methods named above are, in our opinion, most deserving of research attention, the order of presentation should not be construed as

indicating priorities among them. There is insufficient knowledge about the needs of the human factors community to assign priorities.

In discussing current and future problems and trends in human factors applications to system development, Meister (1980, 1982) has identified those informational requirements of human factors specialists that imply needs for the development of new applied methods. On the basis of these suggestions, we make general recommendations for research leading to the development of five new applied methods:

1. Methods for interpreting or extrapolating task/system requirements into personnel requirements;

2. Performance measurement methods that express measures in terms relative to base rates for particular system characteristics and/or demands;

3. Training technology methods for translating task/abilities requirements into training programs;

4. System evaluation methods—static, dynamic, and comparative; and

5. Methods for describing and evaluating task or system impact on affective responses of personnel.

There is a serious disparity between the importance of applied methodologies for human factors work, particularly systems and equipment design, and the efforts being made to document and codify them in a standard manner; to educate behavioral science and engineering students in their use in colleges and universities; to provide continuing education in applied methods to working human factors specialists; and to engage in research to improve existing applied methodologies and develop new ones. It is of great importance to document what is currently known about applied methods. Increasing the accessibility of information on existing methods would be more valuable than developing new methods. What follows is a summary of our recommendations with respect to applied methods.

• Existing methodologies should be assessed and documented in a codified compendium that provides standard descriptions of the most useful applied methods. This compendium would serve both as a comprehensive and readily available source for learning about and as a basis for determining specific research needs.

• Human factors practitioners should be surveyed to determine the importance and frequency of use of existing applied methods in their work; the kinds of information most needed in human factors applications for which existing applied methods are inadequate or nonexistent; and methods for which they require descriptions and guidance for use.

• Tutorials on applied methods should be developed to meet the continuing educational needs of human factors specialists. Methods recommended for the initial tutorial are: task analysis; time line analysis; activities analysis; simulation; and information analysis.

• Basic research should be performed to improve and extend existing applied methods. Methods in need of research include: workload analysis; function allocation; task analysis; survey techniques; and protocol analysis.

• Basic research is also required to develop new methods that can provide the information needed by human factors specialists to do their work. New methods needed include: (1) methods for interpreting or extrapolating task/system requirements into personnel selection requirements; (2) performance measurement methods that express measures in terms relative to base rates for particular system characteristics and/or demands; (3) training technology methods for translating task/abilities requirements into training programs; (4) system evaluation methods—static, dynamic, and comparative; and (5) methods for describing and evaluating task or system impact on affective responses of personnel.

Chapanis, A. 1959 Research Techniques in Human Engineering. Baltimore, Md.: Johns Hopkins University Press.

Geer, C.W. 1981 Human Engineering Procedures Guide. Report AFAMRL-TR-81–35. Wright-Patterson Air Force Base, Ohio: Air Force Aerospace Medical Research Laboratory.

Meister, D. 1980 Human Factors for the Future—Trends and Speculations. In Proceedings of Symposium on Human Factors in Systems Development: Experience and Trends. Karlstadt, Sweden: National Defense Research Institute.

Meister, D. 1982 The role of human factors in system development. Applied Ergonomics 13:119–124.

Parks, D.L., and Springer, W.E. 1976 Human Factors Engineering Analytic Process Definition and Criterion Development for CAFES (Computer Aided Function-Allocation Evaluation System). Warminster, Pa.: Naval Air Development Center.

Topmiller, D.A. 1981 Methods: past approaches, current trends and future requirements. In M.J.Moraal and K. F.Kraiss, eds., Manned System Design. New York: Plenum Press.

Williges, R.C., and Topmiller, D.A. 1980 Task III: Technology Assessment of Human Factors Engineering in the Air Force. Wright-Patterson Air Force Base, Ohio: Air Force Systems Command.

Task analysis is the process of analyzing functional requirements of a system to ascertain and describe the tasks that people must perform. Task analysis has two major aspects: The first specifies and describes the tasks; the second and more important analyzes the specified tasks to determine the number of people needed, the skills and knowledge they should have, and the training necessary. Results of task analysis are used in the development of operating procedures and technical manuals and the determination of critical equipment characteristics and task demands imposed on people. The analytic method involves decomposition of task content into their constituent elements, such as stimulus input,

required response, equipment output, and feedback information.

Simulation is used (1) to allow users to experience, in advance of its operation, portions of a system that are more complex, more dangerous, or more expensive than an experiment could allow for or (2) to predict performance of systems that do not exist. Simulation is a human factors methodology only when it is combined with one of the observational or measurement methodologies. And to extrapolate the observations or measurements to the real world requires a determination of the extent to which things that affect the observations of interest are realistically portrayed in the simulation. How to make this determination (cost/transfer function, part versus whole task simulation, which things to simulate) is the key part of the technology that is still largely unresolved. In the absence of other effective means of predicting the behavioral consequences of system design, simulation is crucial.

Time line analysis organizes a detailed task list for the operational scenario and procedures into serial order and plots the times of individual tasks in sequence against a time base. It portrays sequential, parallel, repeated, and/or intermittent tasks according to what is done. The resulting accumulation of tasks and total performance time can be used to appraise:

1. The validity of the operations to be performed in contributing to system objectives?

2. The feasibility of performing required tasks within the required time;

3. Antecedent hardware and operations conditions to ensure that the requirements of each task element are met;

4. The compatibility of demands on the operator, ensuring that antecedent tasks are identified and performed, required skills and performances are feasible and practical, and difficult, complex, or conflicting demands are avoided; and

5. Workload demands, by comparing time requirements to complete a task series to the time available for completion within the constraints of a given system.

Information analysis identifies information and its flow through a system, usually as perceived from a user’s viewpoint. For example, the flow of information necessary for the operation of an office differs from the flow of documents through that office. Certain system actions occur to the information received, which in turn becomes inputs to subsequent actions. Information analyses enable human factors specialists to assess and design the information requirements of the user interfaces.

In many situations involving field environments, simulations, or mock-ups, it is desirable and useful to catalog the distribution and/or sequential dependencies of workers’ activities. In activity analysis an observer periodically or aperiodically samples the work being performed and classifies the results into a set of categories. The data may be obtained from direct observation or from video or film recording. Individual samples are then aggregated into activity frequency tables or graphs or state transition diagrams. These analyses are especially useful for documenting the way in which task requirements change with alternative system designs or environments or for estimates of relative cost effectiveness, manning requirements, or simply for understanding how individuals or groups spend their time.

This page intentionally left blank.

There was a problem loading page 163.