National Academies Press: OpenBook

Performance Assessment for the Workplace: Volume I (1991)

Chapter: Coda: The JPM Project and Accession Policy

« Previous: 9. The Management of Human Resources
Suggested Citation:"Coda: The JPM Project and Accession Policy." National Research Council. 1991. Performance Assessment for the Workplace: Volume I. Washington, DC: The National Academies Press. doi: 10.17226/1862.
×

Coda: The JPM Project and Accession Policy

Selecting personnel who will turn out to be successful on the job, particularly from a youthful and inexperienced applicant population, is a complicated business. Maintaining a standing army is also expensive. At the heart of these discussions of job proficiency, quality requirements, and trade-off models is the recognition that the public purse is not bottomless. The Department of Defense has an interest in attracting the best possible people into the military. At some point, however, the quality of the enlisted force becomes a function of what the nation can afford; high-quality personnel are more expensive to recruit and, having greater opportunities elsewhere, they are less likely than others to remain in the military for their career. In order to make reasonable budgetary decisions, Congress needs to be able to balance performance gains attributable to selecting those with better-than-average scores on the ASVAB against the costs of recruiting, training, and retaining high-quality personnel. And to improve their control over performance in the enlisted ranks, DoD and the Services need to be able to make more empirically grounded projections of their personnel quality requirements.

Although the problems are complex, and there is still room for improvement at every stage of the research and development, the results of the JPM Project to date indicate that the concept of linking selection standards to objective measures of job performance is basically sound. It appears that it will be feasible for human resource planners and policy makers to incorpo

Suggested Citation:"Coda: The JPM Project and Accession Policy." National Research Council. 1991. Performance Assessment for the Workplace: Volume I. Washington, DC: The National Academies Press. doi: 10.17226/1862.
×

rate empirical data derived from job performance into the decision process in a systematic way. Both for the Services' internal management purposes and for justification of Service quality requirements at the departmental level, this is desirable.

Phase I of the JPM Project has demonstrated that it is possible to develop remarkably reliable hands-on measures of job performance. This was one of the biggest unknowns at the start of the project because there was very little experience of large-scale testing in the job sample mode. Success in this regard seems to have been due to the fact that the test items were designed as a sequence of discrete steps that involved specific, observable actions. This facilitated the scoring. Obviously, the nature of entry-level enlisted jobs lent itself to this kind of test item in a way that managerial or more cognitively loaded jobs would not. In addition, the reliability of the hands-on tests depended on careful training of the test administrators so that they would not give cues or otherwise distort the testing situation. (Such results should not be expected to generalize to situations in which supervisors score the performance of their subordinates or teachers their students.) Daily monitoring of scoring trends both within and among raters seems to have been an effective way to guard against unusually lenient or stringent scoring.

The first phase of the JPM Project also demonstrated that reasonably valid measures of job proficiency can be developed. As anticipated, coverage of the total job domain is a problem with hands-on testing because the methodology is extremely time-consuming. Despite six to eight hours of testing time per individual, the analysis of measurement error due to the particular set of tasks selected for testing presented in Chapter 6 indicates that more time and more tasks would have produced better measures of job proficiency. In addition, there is still much to learn from the JPM data base about the relationships among the hands-on, walk-through performance tests, simulations, and written tests and about the substitutability of the less expensive and time-consuming measures for the hands-on tests. Nevertheless, the JPM Project has produced hands-on measures of job proficiency that are more credible than training grades and substantially more stable and accurate than the ubiquitous supervisor's rating.

One of the most important implications of these findings is that the Armed Services Vocational Aptitude Battery has been shown to be moderately related to something fairly close to actual performance in 23 military jobs (comprising some 40 percent of the enlisted force) —and by inference to many more. Although the AFQT is not the optimal predictor that could be obtained from the ASVAB, the median corrected correlation of .38 with hands-on performance reported in Chapter 8 has considerable practical utility. When one remembers the uncertainties occasioned by the misnorming episode of 1976-1980, this is an important datum—enough to justify the use

Suggested Citation:"Coda: The JPM Project and Accession Policy." National Research Council. 1991. Performance Assessment for the Workplace: Volume I. Washington, DC: The National Academies Press. doi: 10.17226/1862.
×

of the ASVAB in setting enlistment standards, if also modest enough to encourage the search for new predictors to add to the ASVAB.

The development of cost/performance models for setting enlistment standards, now being explored by both DoD and the Services, has great potential relevance for accession policy. Until now, the standards-setting process has been largely based on an informal process of individual judgments and negotiations among the stakeholders. The manpower management models used by military planners for other purposes have simply assumed an appropriate enlistment standard or have used surrogates at quite some remove from job performance. If the JPM performance data can be successfully incorporated into trade-off models, the models will offer policy officials useful tools for estimating the probable effects on performance and/or costs of various scenarios—say a 10 percent reduction in recruiting budgets, a 20 percent reduction in force, or a downturn in the economy. The solutions provided by such models are not intended to and will not supplant the overarching judgment that policy officials must bring to bear, but they can challenge conventional assumptions and inject a solid core of empirical evidence into the decision process. One of the early runs of the DoD cost/ performance model, for example, has suggested the counterintuitive proposition that, in a situation of economic downturn, it is cost-effective to increase recruiting budgets: the marginal costs of increasing the proportion of high-quality personnel under these conditions will be outweighed by their performance contribution to the downsized force.

There are a number of gaps in the JPM data that could limit the usefulness of the cost/performance methodology in operational settings. One of the most difficult challenges has been to find a satisfactory way to generalize the JPM results to performance in other military jobs. At least two lines of research might be fruitful. First, analysis of the personal attributes or traits required by a sample of military jobs (see Chapter 4) might provide clues for more homogeneous clusters of jobs around the JPM anchor jobs. In addition, a better understanding of the relative difficulty of military jobs has turned out to be of overriding importance in understanding the hands-on test scores and, by extension, in supporting the generalizability of the results to other, similar jobs.

We have spoken at some length in this volume of the need to provide an absolute, or competency, interpretation of performance in a particular job in order to set enlistment standards—so that policy makers can address the question of how much performance is enough. If policy makers are only interested in optimizing costs, then relative comparisons among people as provided by the JPM data will suffice. From that data, analysts can construct various groups or ability mixes (based on different cutoff scores), compute their costs and overall performance outcomes, and see if there is a cutoff (or ability distribution) that minimizes cost per unit of productivity.

Suggested Citation:"Coda: The JPM Project and Accession Policy." National Research Council. 1991. Performance Assessment for the Workplace: Volume I. Washington, DC: The National Academies Press. doi: 10.17226/1862.
×

But that analysis does not provide policy makers with a grounds for judging whether the optimal cost solution will produce an enlisted corps that can perform all first-term jobs competently. If policy makers want to be able to look at costs in the framework of the performance needed to accomplish the institutional mission, then the performance data will have to be interpretable with regard to an absolute standard of mastery or competency. By and large, the JPM measures were not developed as competency scales. However, some thought is being given to ways of providing external competency anchors for the data—say, by comparing the JPM cohort with the enlisted force in what military experts consider a very bad and a very good recruiting year and extrapolating the known job performance to the other two groups. This kind of thing seems well worth exploring.

The full implications of the job performance measurement research for military policy makers—and for civilian-sector employers—remain to be worked out in coming years. It has produced a rich body of data and a wealth of methodological insights and advances. Most important of all, the JPM Project has defined the challenges for the next generation of research on performance assessment.

Suggested Citation:"Coda: The JPM Project and Accession Policy." National Research Council. 1991. Performance Assessment for the Workplace: Volume I. Washington, DC: The National Academies Press. doi: 10.17226/1862.
×
Page 207
Suggested Citation:"Coda: The JPM Project and Accession Policy." National Research Council. 1991. Performance Assessment for the Workplace: Volume I. Washington, DC: The National Academies Press. doi: 10.17226/1862.
×
Page 208
Suggested Citation:"Coda: The JPM Project and Accession Policy." National Research Council. 1991. Performance Assessment for the Workplace: Volume I. Washington, DC: The National Academies Press. doi: 10.17226/1862.
×
Page 209
Suggested Citation:"Coda: The JPM Project and Accession Policy." National Research Council. 1991. Performance Assessment for the Workplace: Volume I. Washington, DC: The National Academies Press. doi: 10.17226/1862.
×
Page 210
Next: References »
Performance Assessment for the Workplace: Volume I Get This Book
×
Buy Paperback | $75.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Although ability testing has been an American preoccupation since the 1920s, comparatively little systematic attention has been paid to understanding and measuring the kinds of human performance that tests are commonly used to predict—such as success at school or work. Now, a sustained, large-scale effort has been made to develop measures that are very close to actual performance on the job. The four military services have carried out an ambitious study, called the Joint-Service Job Performance Measurement/Enlistment Standards (JPM) Project, that brings new sophistication to the measurement of performance in work settings.

Volume 1 analyzes the JPM experience in the context of human resource management policy in the military. Beginning with a historical overview of the criterion problem, it looks closely at substantive and methodological issues in criterion research suggested by the project: the development of performance measures; sampling, logistical, and standardization problems; evaluating the reliability and content representativeness of performance measures; and the relationship between predictor scores and performance measures—valuable information that can also be useful in the civilian workplace.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!