Performance Assessment for the Workplace, Volume II Technical Issues (1991) / Chapter Skim
Currently Skimming:
Range Restriction Adjustments in the Prediction of military Job Performance
Range Restriction Adjustments in the Prediction of military Job Performance
Pages 127-157
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 127... ...
The correlation coefficient, in particular, allows for ready comparisons of predictive validity across occupational categories as well as across different predictor and criterion measures, so that its widespread use is not surprising. In the context of military performance assessment, correlations allow for comparisons of predictive validity over time and across Services.
|
From page 128... ...
. The paper concludes with a discussion of issues related to the implementation of a set of adjustment procedures for validation studies in the military, where the choice of the reference population, choice of selection variables for making adjustments, and choice of an analytical procedure all have important consequences for the assessment of the predictive validity of present and future versions of the ASVAB.
|
From page 129... ...
In such a situation one might imagine a third variable, Z, as the explicit selection variable Z can be
|
From page 130... ...
or as a latent variable underlying the true selection process (Muthen and Joreskog, 1983~. In this case, both X and Y are referred to as incidental selection variables because the selection effects on the bivariate distribution of X and Y are indirect.
|
From page 131... ...
The scatterplot contains only the upper 50 percent of the observations on Z, and, as can be seen in the figure, the only evidence for this being a selected sample is a reduction in the vanability of the marginal distributions of X and Y Even when X and the true selection variable are as highly correlated as in this example, the effects of incidental selection yield a least-squares regression line (again depicted by the broken line)
|
From page 132... ...
, while the other represents an incidental selection variable (X)
|
From page 133... ...
Examples such as these provide a clear indication that understanding of selection effects is crucial to the interpretation of results from criterionrelated validity studies. Without risk of hyperbole, one could say that the predictive validity of a selection instrument cannot be accurately characterized unless the possible effects of sample selection are accounted for to some extent.
|
From page 134... ...
D UNBAR AND ROBERT L LINN Coping With the Effects of Selection The subtle ways in which biases introduced by sample selection affect estimates of correlations in an entire applicant pool are difficult to account for in any exhaustive way; however, it is possible to obtain useful assessments of the degree of association between a set of predictors and a criterion even with selected samples.
|
From page 135... ...
~ZX where C represents the variance-covariance matrix of the variables indicated by subscripts x and z, which refer to incidental and explicit selection variables, respectively. An asterisk is used to distinguish matrices based on the selected sample from those based on the unselected population.
|
From page 136... ...
Thus, the expression given above would be applicable in a situation where, for example, selection was explicit on a composite variable, but interest in the correlations between individual elements of the composite and the criterion existed. Where a number of explicit selection variables are known, the Lawley extension of the above three-variable formula would provide the appropriate adjustment under assumptions (1)
|
From page 137... ...
Perhaps a more realistic example would depict X as the only available proxy for the unobserved true selection variable and treat it as explicit even though it is incidental. When the two-variable formula is thus used in the setting depicted in Figure 4, the corrected estimate of Ray is .540, still smaller than the population correlation.
|
From page 138... ...
extended this finding through a more systematic set of simulated bivariate distributions in which the scatter around the regression line was reduced at the extremes of the predictor score distribution. They too found overcorrections to predominate when the selected sample represented less than half of the unselected population.
|
From page 139... ...
If the selection tests are truly incidental selection variables, then in evaluating the above tendencies toward overcorrections under particular circumstances, one must consider the dual effect of violated assumptions regarding the selection process as well as the regressions of explicit on incidental selection variables. Linn and colleagues (Linn, 1968; Linn et al., 1981)
|
From page 140... ...
. lions corrected for range restriction and population correlations, expressed as Fisher's Z equivalents.
|
From page 141... ...
In the case of explicit selection on the predictor, the approximate standard error of the corrected correlation under bivariate normality is a function of the ratio of the standard deviation in the unselected population to that in the selected sample, the magnitude of the correlation in the selected sample, and the sample size. Table 3 illustrates the relative magnitudes of the standard errors of uncorrected, SE(r)
|
From page 142... ...
As can be seen from the entries in the table, under conditions of minimal selection, the loss in precision incurred by correcting for range restriction is relatively minor. When the ratio of standard deviations, K, is slightly greater than 1, the standard error of the corrected correlation, SE(R)
|
From page 143... ...
The bias in ray, bo' and bx when X and Y are incidental selection variables provides a relatively simple case for purposes of illustration. If Z is assumed to be the only explicit selection variable, then the selection process is described by a threshold value, t, such that observations exist in the selected sample only when Z > t; in other words, t
|
From page 144... ...
two-stage approach to adjusting for selection effects involves: (1 ) estimating the nonzero conditional expectation of prediction error for individuals via probit regression of the binary variable on the selection variables and (2)
|
From page 145... ...
This may be relevant to certain types of group comparisons in the loins-Service context, where a common criterion variable exists. The general approach can also be adapted to provide an alternate means of correcting correlation coefficients, albeit at the expense of distributional assumptions about the selection variables that are not required by the PearsonLawley procedures.
|
From page 146... ...
The problem this poses for criterion-related validity studies lies in the fact that most potential selection variables are precisely those variables whose use as predictors is being validated. Under these circumstances, one might expect the standard errors of the adjusted estimates to be too large to allow for useful inferences
|
From page 147... ...
Some of the technical limitations of the Heckman approach alluded to in the above example have been studied in greater detail, with concern again centering on violated assumptions regarding the distribution of selection variables and on sampling fluctuations of the adjusted estimates. Goldberger (1980)
|
From page 148... ...
, these methods, and perhaps any method for handling selection bias, are best considered as ways of focalizing the possible effects of missing observations on outcome measures rather than as substitutes for random samples normally required for valid inferences about the characteristics of a population. IMPLICATIONS FOR PREDICTIVE VALIDITY IN A JOINT-SERVICE CONTEXT The problem of range restriction is not new to anyone concerned with establishing the criterion-related validity of selection and classification tests in the Services, although the methods for coping with it have been far from uniform over the years.
|
From page 149... ...
As indicated at various points in the preceding review, the standard PearsonLawley correction procedures are familiar to most personnel psychologists and appear to have been carefully evaluated in both analytical and empirical studies, many of the latter being performed with the specific problems of criterion-related validity in mind. The limitations of these procedures and the conditions under which they are likely to give misleading indications of predictive validity are well documented.
|
From page 150... ...
In cases where the Pearson-Lawley corrections are used, the source of estimates of the variances and covariances of selection variables in the unrestricted group, in addition to the selection process itself, determines the magnitude of any correction factor. Variations in these estimates, such as those caused by preexisting differences between potential enlisters opting for one Service over the others, result in correction factors of varying magnitudes and make the corrected values difficult to interpret.
|
From page 151... ...
The Lawley corrections given in the table assume the 10 individual subtests to be explicit selection variables, and the composite and course grades to be incidental selection variables. The adjusted values given in column A were obtained by using the variances and covariances of subtest standard scores for the total Youth Population (U.S.
|
From page 152... ...
D UNBAR AND ROBERT L LINN TABLE 5 Corrected Predictive Validities of the Marine Corps Clerical Composite in Nine Clerical Training Programs Based on 1980 Youth Population and Subtest Standard Deviations in Selected and Unselected Groups Panel (a)
|
From page 153... ...
However, the relative magnitudes of the corrected validity coefficients across training programs remain the same regardless of the proportion of the reference group that is deleted. It should be noted that the regular pattern of steadily decreasing estimates of predictive validity for increasing degrees of truncation in the proposed reference population is to be expected when the selection variables (here ASVAB subtests)
|
From page 154... ...
CONCLUDING REMARKS The effects of selection on correlation coefficients and regression parameters place the personnel psychologist on the horns of a classic statistical dilemma. To retain observed values gives an extremely misleading view of the relationship between predictor and criterion variables, but to correct observed values places one at the mercy of assumptions that will not be strictly satisfied in practice and of an added degree of uncertainty in estimating population values.
|
From page 155... ...
When a summary statistic is needed to describe the predictive validity of a selection instrument in a variety of settings, these approaches can provide a more complete assessment of the relationship between that instrument and the relevant measures of on-thejob performance. REFERENCES Air Force Human Resources Laboratory 1982 Aptitude Index Validation of the Armed Services Vocational Aptitude Battery (ASVAB)
|
From page 156... ...
Psychological Bulletin 69:69-73. 1983a The Pearson correction formulas: implications for studies of predictive bias and estimates of educational effects in selected samples.
|
From page 157... ...
1985 Effects of Truncating a Reference Population on Corrections of Validity Coefficients for Range Restriction. Research Memorandum 85-40.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.