Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
FAIRNESS IN EMPLOYMENT TESTING Validity Generalization, M~rityIssues, and the General Aptitude Test Battery John A. Hartigan and Alexandra K. Wigdor, Eclitors Committee on the General Aptitude Test Battery Commission on Behavioral and Social Sciences and Education National Research Council NATIONAL ACADEMY PRESS Washington, D.C. 1989
NATIONAL ACADEMY PRESS 2101 Constitution Avenue, NW . - Washington, DC 2W18 NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance. This report has been reviewed by a group other than the authors according to procedures approved by a Report Review Committee consisting of members of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Frank Press is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Robert M. White is president of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Samuel O. Thier is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy's purposes of furthering kn`)wledPe and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Frank Press and Dr. Robert M. White are chairman and vice chairman, respectively, of the National Research Council. rely ~ i-~~ ~ my, EVE..= ~ of This project was supported by the Employment and Training Administration of the U.S. Department of Labor. Library of Congress Cataloging-in-Publication Data National Research Council (U.S.). Committee on the General Aptitude Test Battery. Final report: fairness in employment testing: validity generalization, minority issues, and the General Aptitude Test Battery/John A. Hartigan and Alexandra K. Wigdor, editors; Committee on the General Aptitude Test Battery, Commission on Behavioral and Social Sciences and Education, National Research Council. p. cm. Bibliography: p. Includes index. ISBN 0-309-04030-2 (paper); ISBN 0-309-04033-7 (cloth) 1. General aptitude test battery-Evaluation. 2. Employment tests-United States-Evaluation. 3. Employment tests-Law and legislation-United States. I. Hartigan, John A., 1927- . II. Wigdor, Alexandra K. III. Title. HE 5549.5.E5N38 1989 153.9'4-dc20 89-3284 1 CIP Copyright C) 1989 by the National Academy of Sciences No part of this book may be reproduced by any mechanical, photographic, or electronic process, or in the form of a phonographic recording, nor may it be stored in a retrieval system, transmitted, or otherwise copied for public or private use without written permission from the publisher, except for the purposes of official use by the U.S. government. Printed in the United States of America First Printing, May, 1989 Second Printing, August 1990
Committee on the General Aptitude Test Battery JOHN A. HARTIGAN (Chair), Department of Statistics, Yale University LORRIE A. SHEPARD (Vice Chair), School of Education, University of Colorado, Boulder MARCUS ALEXIS, Dean, College of Business Administration, University of Illinois, Chicago MANFRED EMMRICH, North Carolina State Employment Service, Raleigh LARRY V. HEDGES, Department of Education, University of Chicago IRA J. HIRSH, Department of Psychology, Washington University, and Central Institute for the Deaf, St. Louis, Mo. RICHARD M. JAEGER, School of Education, University of North Carolina at Greensboro STEPHEN P. KLEIN, The Rand Corporation, Santa Monica, Calif. ROBERT L. LINN, School of Education, University of Colorado, Boulder JOHN M. RAUSCHENBERGER, Employee Development Office, Ford Motor Company, Dearborn, Mich. MICHAEL ROTHSCHILD, Department of Economics and Dean, Social Sciences, University of California, San Diego PAUL R. SACKETT, Industrial Relations Center, University of Minnesota O. PETER SHERWOOD, Solicitor General, New York State HOWARD F. TAYLOR, Department of Sociology, Princeton University ALEXANDRA K. WIGDOR, Study Director HILDA WING, Research Associate DIANE L. GOLDMAN, Administrative Secretary . . .
Liaison Group, Members ROBERT BOLDA, Personnel Research Division, General Motors Corporation, Detroit, Mich. (Retired) RALPH G. CANTRELL, Virginia Employment Commission, Richmond WAYNE F. CASCIO, Graduate School of Business Administration, University of Colorado, Denver SUZAN CHASTAIN, Office of Civil Rights, Office of the Solicitor, U.S. Department of Labor ROBERT DELAHUNTY, Office of the Assistant Attorney General (C)vi! Rights Division), U.S. Department of Justice CONSTANCE L. DUPRE, U.S. Equal Employment Opportunity Commission (Retired) PATRICIA J. DYER, IBM Corporate Employment and Placement, Armonk, N.Y. KENNETH EDWARDS, Skill Improvement Department, International Brotherhood of Electrical Workers, Washington, D.C. MANFRED EMMRICH, North Carolina State Employment Service, Raleigh BOB FUNSTON, Oklahoma Employment Security Commission, Oklahoma City JOHN E. HUNTER, Department of Psychology, Michigan State University HENRY LEVIN, Departments of Education and Economics, Stanford University PHILIP B. LYONS, U.S. Equal Employment Opportunity Commission CHARLES F. NIELSON, Texas Instruments, Inc., Dallas, Tex. EDWARD E. POTTER, McGuiness & Williams, Washington, D.C.; Equal Employment Advisory Council v
Vi LIAISON GROUP, MEMBERS MARJORIE RAGOSTA, Educational Testing Service, Princeton, N.J.; Committee on Disabilities and Handicaps, American Psychological Association NAMBURY S. RAJU, Department of Psychology, Illinois Institute of Technology, Chicago DENNIS K. RHOADES, National Economic Commission, American Legion, Washington, D.C. WILLIAM L. ROBINSON, Lawyers' Committee for Civil Rights Under Law, Washington, D.C. WILLIAM W. RUCH, Psychological Services, Inc., Glendale, Calif. ROBERT A. SCHAERFL, U.S. Employment Service, U.S. Department of Labor FRANK L. SCHMIDT, College of Business Administration, University of Iowa NEAL W. SCHMITT, Department of Psychology, Michigan State University DONALD J. SCHWARTZ, Office of Research and Analytic Services, U.S. Equal Employment Opportunity Commission RICHARD T. SEYMOUR, Lawyers' Committee for Civil Rights Under Law, Washington, D.C. JAMES C. SHARE, Career Entry Group, U.S. Office of Personnel Management WILLIAM TRACEY, Employment Services, New Jersey Department of Labor, Trenton (Retired) DENNIS L. WARMKE, Philip Morris U.S.A., Richmond, Va. RAUL YZAGUIRRE, National Council of La Raza, Washington, D.C.
Preface The Department of Labor is considering whether to promote the use of its General Aptitude Test Battery (GATB) throughout the U.S. Employ- ment Service to screen many of the 19 million people who pass through the system annually in search of private- and public-sector jobs. This study was undertaken at the agency's request because system-wide use of the test battery for referral purposes raises important questions of public policy. In order to provide employers with the maximum benefits of testing, the Employment Service would need to refer applicants in order of test score. Such a policy, however, would severely reduce the employment opportunities of lower-scoring applicants, particularly of minority job seekers, who have lower average test scores as a group than the majority. What is the appropriate balance between anticipated productivity gains from better employee selection and the well-being of individual job seekers? Can equal employment opportunity be said to exist if screening methods systematically filter out very large proportions of minority candidates? Such an outcome would leave employers and the Employ- ment Service vulnerable to the charge of discrimination under Title VII of the Civil Rights Act of 1964. In pilot projects of the test-based referral system, conducted since 1980, Department of Labor officials have adopted a score-adjustment strategy in which each applicant's test score is computed as a percentile score within his or her own racial or ethnic group (black, Hispanic, and other). By combining within-group percentile scores and top-down . . V11
viii PREFACE selection of the applicants to be referred, the Department of Labor sought a workable compromise between productivity goals and federal equal employment opportunity policy. However, the within-group scoring strategy created its own problem: the Department of Justice questioned its legality and constitutionality on grounds of reverse discrim- ination. As a consequence of this challenge, the Department of Labor sought guidance from the National Academy of Sciences, which, through the National Research Council, has convened a committee of experts to conduct a thoroughgoing evaluation of the plan to use the GATB as the primary tool for deciding which applicants to refer to employers. The Department of Labor asked the committee to address a number of important technical questions to establish the appropriateness of using a single general aptitude test to predict performance in a large number of very different jobs. In addition, recognizing that these technical issues are surrounded by a complex web of governmental policies and legal require- ments and have serious economic and social implications, the Department of Labor asked the committee to consider the possible effects of wide- spread adoption of testing in the Employment Service-effects on em- ployers, on various categories of job seekers, and on the economic health of the country. This study is intended to help policy makers decide whether the GATB should be given a primary role in the Employment Service referral system and, in that event, to offer guidance on methods for assembling the pool of job candidates and for reporting test scores. The policy context is described in Chapters 1 and 2. Chapter 3 presents an overview of the U.S. Employment Service and its activities. In Chapter 4 we look at the quality of the GATB, and in Chapter 5 discuss critical weaknesses in the test and in the GATB research program that must be overcome if it is to be used as a primary referral tool. The principal scientific underpinnings of the Department of Labor's plan to use the GATB for referrals to all kinds of jobs, contained in the theory of validity generalization, are examined in Chapters 6 through 9. Chapter 8 looks at the accumulated body of GATB validity research and presents the committee's judgments about the degree of predictive accuracy that can reasonably be assumed for jobs not studied. Chapter 10 describes the referral system- which we call the VG-GATB Referral System to signal its dependence on validity generalization as it has been conceptualized by the research staff of the U.S. Employment Service. Chapter 11 discusses the potential effects of the system on the various Employment Service clients, and Chapter 12 analyzes the claims about economic benefits said to accrue from referring job candidates in order of test score.
PREFACE 1X Based on the technical findings presented earlier, Chapters 13 and 14 present the committee's recommendations to policy makers. In Chapter 13 the committee presents its conclusions about the fair use of employ- ment tests and makes recommendations about adjusting the test scores of minority job seekers. Chapter 14 presents the committee's most impor- tant recommendations for the use of the GATB and the design of the VG-GATB Referral System. In providing independent advice to the government on matters of science policy, the National Research Council depends on committees of volunteers chosen for their expertise, together with members of the permanent staff, to carry out its work. The members and staff of the Committee on the General Aptitude Test Battery include experts in statistics and meta-analysis, psychometrics, industrial and personnel psychology, economics, sociology, policy analysis, law, and the Employ- ment Service the expertise needed to address the broad range of technical and policy questions raised in this study. Brief biographies of committee and staff appear in Appendix C. During the course of its study, the committee has called on a great many people, who gave generously of their time, their expertise, and their insights. Because the policy decisions ultimately made by the Department of Labor with regard to the GATB will have an impact, perhaps a great impact, on the interests of a variety of individuals, groups, and institu- tionsj a carefully selected liaison group was appointed to ensure that the committee would be conversant with all relevant policy perspectives and areas of expertise. Our particular thanks go to the 28 members of this group who met with us on three occasions and provided needed informa- tion and position papers throughout. Individual members also presented occasional special briefings on specific issues before the committee. We received assistance from other quarters as well. In response to questions from the committee about employers' reactions to the VG-GATB system and within-group scoring, the Employers' National Job Service Committee developed and circulated a questionnaire that elicited over 500 responses. Our work also benefited from the cooperation of the National Rehabilitation Association and the major veterans organizations. We owe a great deal to John Hawk, personnel research psychologist in the test research division of the U.S. Employment Service, and his colleagues in the central and regional offices. In response to our sheer need to be educated about the Employment Service system and the GATB, they provided helpful briefing documents. As we came to grips with the very large undertaking required by our charge, they remained helpful and forthcoming in the face of frequent requests for data, documents long buried in the files, information about day-to-day opera- tions, and myriad other questions that came up. Staff members of the
X PREFACE Northern Test Development Field Center provided the tapes of the GATB data base and arranged for the collection of a sample of item-Ievel test data for our item-bias analysis. The Southern Test Development Field Center supplied useful information about the development of the GATB and the operations of the Employment Service system. We would also like to recognize the contributions of several consultants who helped with our data analysis: Douglas Weeks assisted committee member Robert Linn in the differential validity and differential prediction analysis; Laura Burris and Victoria Crawshaw assisted Paul Sackett with the meta-analysis of the entire GATB data base; Robin Corley assisted Lorrie Shepard with a study of item bias; Anita Tesh assisted Richard Jaeger with a synthesis of the literature on GATB properties. Glen Sueyoshi contributed a background paper on the economic effects of improved employee selection as the question is addressed in the econom- ics literature. Our acknowledgments would not be complete without special thanks to staff members who worked with the committee: Hilda Wing, who assisted with the research and writing; Diane Goldman and Carolyn Sax, who provided administrative support and kept control of the evolving manu- script; and Christine McShane, whose graceful editing defied a hectic schedule. JOHN A. HARTIGAN, Chair ALEXANDRA K. WIGDOR, Study Director Committee on the General Aptitude Test Battery
Contents Summary PART I BACKGROUND AND CONTEXT 1 The Policy Context ...................... 2 Issues in Equity and Law................. 3 The Public Employment Service........... PART II . .17 .29 .. . . . .52 ANALYSIS OF THE GENERAL APTITUDE TEST BATTERY 4 The GATB: Its Character and Psychometric Properties . . .73 5 Problematic Features of the GATB: Test Administration, Speededness, and Coachability 99 PART III VALIDITY GENERALIZATION AND GATB VALIDITIES 6 The Theory of Validity Generalization ................. 7 Validity Generalization Applied to the GATB. 8 GATB Validities..................................... 9 Differential Validity and Differential Prediction. X1 119 . . .134 149 .172
xii CONTENTS PART IV ASSESSMENT OF THE VG-GATB PROGRAM 10 The VG-GATB Program: Concept, Promotion, and Implementation I! In Whose Interest: Potential Effects of the VG-GATB Referral System ................ .191 .209 |2 Evaluation of Economic Claims 235 PART V CONCLUSIONS AND RECOMMENDATIONS )3 Recommendations for Referral and Score Reporting 14 Central Recommendations ....................... References .... APPENDIXES A A Synthesis of Research on Some Psychometric Properties of the GATB Richard M. Jaeger, Robert It inn, and Anita S. Tesh B Tables Summarizing GATE Reliabilities 325 C Biographical Sketches, Committee Members and Staff 329 .251 . .281 . .291 .303 Index..... .335
FAIRNESS IN EMPLOYMENT TESTING .~
