Skip to main content

Currently Skimming:

13 Recommendations for Referral and Score Reporting
Pages 251-280

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.


From page 251...
... First, the raw test scores for each applicant are converted into five job family scores, based on predetermined weightings of the cognitive, perceptual, and psychomotor composites. Then each of the applicant's five job family scores is converted to a percentile score, which shows the applicant's ranking with respect to others in the same ethnic or racial subgroup on a scale of 1 to 100.
From page 252...
... We could have added a third reservation, for, if the VG-GATB Referral System became a very important route to employment, policy makers would have to anticipate that at least some applicants might claim minority status at the local Job Service office in order to get the benefit of preferential score adjustments and make no such claim at the workplace, so that the meaning of the reported score would be interpreted with reference to the majority group. Despite these reservations, we conclude this chapter with the recommendation that score adjustments, possibly within-group percentile score adjustments, continue to play a role, albeit a somewhat different role, in the VG-GATB Referral System for reasons that emerge from our technical analyses of GATE data as well as considerations of social policy.
From page 253...
... We are now in a position to look again at alternative score reporting and referral models, but at this time many of the earlier assumptions can be replaced by empirical statements. Evidence presented earlier in this report establishes that the average scores of black Job Service clients are substantially lower than those of majority clients, although the difference varies somewhat by job family.
From page 254...
... . Since only group status and test scores are known for applicants, information about future job performance must be extrapolated from validity studies of job incumbents who have taken the test.
From page 255...
... Because of the existence of substantial group differences in average test scores, particularly differences between black and majority-group job applicants, many now find this definition of fairness insufficient, at least as it pertains to allotting employment opportunity. A test may be fair in predicting performance, but nevertheless predict performance rather poorly.
From page 256...
... So long as there are average group differences in test scores-and these are likely to manifest themselves whenever racially or ethnically identifiable subgroups live in circumstances of comparative disadvantage the ejects of imperfect prediction will fall more heavily on these disadvantaged minorities than on other social groups. Figure 13-1 shows why the effects of imperfect prediction fall disproportionately on groups that have lower average test scores than the majority group.
From page 257...
... His analysis, which is based on GATE validity data for 47 jobs, illustrates the effects of rejection errors and acceptance errors: many more of the successful black job incumbents in the validity studies would not have been referred had the test scores been the basis of referral; conversely, of the marginal job incumbents (those who received low supervisor ratings) , a greater proportion of whites than blacks would have been referred had test scores been used.
From page 258...
... Nevertheless, the undoubted effect of imperfect prediction when social groups have different average test scores is to place the greater burden of prediction error on the shoulders of the lower-scoring group. Is this fair?
From page 259...
... When analyzed this way, the data reveal that more white test failers than black ones would do satisfactory work if given the chance, and more blacks than whites are passing the test and proving to be unsatisfactory workers. Thus the test overpredicts black job performance and is predictively unfair to whites.
From page 260...
... Which truth is the most important truth? At this point in our history, it is certain that the use of the GATE without some sort of score adjustments would systematically screen out blacks, some of whom could have performed satisfactorily on the job.
From page 261...
... Within-group percentile score, top-down referral is referral in which a percentile score is computed for each applicant by comparing the raw score for that applicant with the scores obtained by a norm group of the same racial or ethnic identity. (Equivalently, a different linear transformation is applied to the raw test score for the different groups so that the mean and the variance of test scores are the same for all groups.
From page 262...
... We may view selection under this rule as being determined by an adjusted test score that is obtained from the original by randomly reassigning test score values above the cutoff to all examinees who initially score above the cutoff and randomly reassigning test score values below the cutoff to all examiners who score initially below the cutoff. Analysis of the Referral Rules There are two statistical computations used to assess both the gains in expected performance from using the GATB and its adverse impact on groups that tend to score low on the test.
From page 263...
... These assumptions are taken from the USES norm groups for Job Families IV and V, which include almost all jobs typically filled through the Employment Service. (The norm-group differences in average test scores between "other" and black are 1 standard deviation for Job Family IV and 0.8 standard deviation for Job Family V; since jobs are divided nearly evenly between the two families, an overall figure of .9 is assumed.)
From page 264...
... As Table 13-1 illustrates, the two referral models that incorporate score adjustments dramatically increase the percentage of minority applicants referred and yet show only small decreases in predictive power compared with the raw-score, top-down referral rule. The minimum competency model does not compare favorably on either dimension of interest, expected performance or minority presence in the referral pool.
From page 265...
... These assumptions are based on 72 GATE studies, each of which each contained at least 50 black and 50 nonminority workers, all from Job Families IV and V; the median difference in average test scores (with the majority-group standard deviation in each study set equal to 1) is 0.88; the median ratio of minority standard deviation to majority standard deviation is .90.
From page 266...
... Yet, when the validity is modest, as it is here, many of the minority applicants excluded would have performed better than many of the majority workers included. Conclusion This rule has an adverse impact on minority applicants that, in our judgment, is out of all proportion to the gains in expected job performance (as measured by supervisor ratings)
From page 267...
... When the correlation expressing validity between test score and job performance is .3, and the minority-group average test score is 1 standard deviation less than the majority-group average test score, then this score adjustment adds approximately (1 - .09) = 0.91 standard deviation to each minority score.
From page 268...
... , it is a strategy for selecting test items with the goal of eliminating group differences in test scores. Using this procedure, items are field-tested with minority- and majoritygroup members and, whenever possible, items are selected that show the least difference in the proportions correct obtained by each group.
From page 269...
... The reasons for the failure of the procedure to bring substantial reductions in group differences in test scores stem in part from the requirements of a large-scale testing program. The item pool was not stable over time, in part because test forms were periodically made public as part of the settlement.
From page 270...
... In other words, increasing the referral rates of racial and ethnic minorities will produce a concomitant reduction in the referral chances of some white applicants with higher raw test scores and somewhat greater predicted success on the job. In order to avoid that diminution in the prospects of majority-group applicants while at the same time enhancing the competitive position of minority applicants, the committee recommends the consideration of a referral rule that combines the essential features of both the raw-score,
From page 271...
... Policy makers at the Department of Labor will need to consider the potential legal risks of this referral strategy just as they do the risks of other referral plans. On a practical level, if there is a burden imposed by the Combined Rules Referral Plan, it is that the local Job Service office must deal with a somewhat larger number of people to fill a job order and the employer must consider more applicants than is absolutely necessary under either rule alone.
From page 272...
... Under the Combined Rules Referral Plan no applicant is excluded who would have been referred if the Employment Service had made the baseline 20 referrals on just total-group or just within-group percentile scores. To illustrate, Table 13-3 describes a situation in which the employer has two job openings and has asked for a referral ratio of 2:1.
From page 273...
... Differences in means or standard deviations of the applicant groups from the norm group could cause quite different referral rates and validities of the within-group score for particular jobs. If, for example, an employer set qualifications for a job that are correlated with test score, then the applicants for the job would be expected to have a smaller standard deviation in test score than the norm group, and the differences between majority-group and minority-group mean score would be expected to be lower.
From page 274...
... However, these are workers on the job, not applicants, and if there were greater differences between mean scores for applicants than for workers, the referral rates for minority and nonminority workers might be about the same. THE PROBLEM OF REPORTING SCORES The general principle that should guide policy on reporting test scores is that the employer and the applicants should be given sufficient information to make correct inferences about a candidate's likely job performance from the test score.
From page 275...
... Reporting Total-Group Percentile Scores Reporting total-group percentile scores is also potentially misleading, because the employer has no information about the levels of job performance that can be expected from a particular score. It is tempting for the employer to infer that a person at the 16th percentile of whatever norm group on the test score will be at the 16th percentile of the norm group in job performance; Employment Service literature promoting the VGGATB Referral System indicates that the most able workers within each ethnic group are being referred.
From page 276...
... As Table 13-4 shows, extreme scores on the test distribution correspond to modest scores on the expectancy distribution, reflecting the modest predictability of job performance from test score. Proposed Protocol for Reporting Scores In the committee s judgment, a combination of percentile and expectancy scores will provide job applicants and prospective employers with
From page 277...
... CONCLUSIONS Fair Use of the GATB 1. Use of GATE scores in strict top-down, rank-ordered fashion is fair in the sense that a given test score predicts about the same level of job
From page 278...
... 3. Although the GATB does not appear to be inherently biased against minority-group test takers, the undoubted effect of imperfect prediction when social groups have different average test scores is to place the greater burden of measurement error on the shoulders of the lowerscoring group.
From page 279...
... The committee recommends the continued use of score adjustments for black and Hispanic applicants in choosing which applicants to refer to an employer, because the elects of imperfect prediction fall more heavily on minority applicants as a group due to their lower mean test scores. We endorse the adoption of score adjustments that give approximately equal chances of referral to able minority applicants and able majority applicants: for example, within-group percentile scores, performance-based scores, or other adjustments.
From page 280...
... 2. We also recommend that USES study the feasibility of what we call a Combined Rules Referral Plan, under which the referral group is composed of all those who would have been referred by the total-group or by the within-group ranking method.


This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More information on Chapter Skim is available.