Performance Assessment for the Workplace, Volume II Technical Issues (1991) / Chapter Skim
Currently Skimming:
Alternatives to the Validity Coefficient for Reporting the Test-Criterion Relationship
Alternatives to the Validity Coefficient for Reporting the Test-Criterion Relationship
Pages 158-206
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 158... ...
The effectiveness of a testing program in selecting appropriate individuals depends on how well the test scores correspond with some objective measure of actual performance. The relationship of test scores to an objective performance measure is called predictive validity.
|
From page 159... ...
Each data set includes a test score and a performance score for 1,000 individuals. To simplify the interpretation of these data sets, both the test and the performance measure range from a minimum of 0 to a maximum of 50.
|
From page 160... ...
Although the tests have the same validity, it is apparent from comparison of the scatter plots that the two distnbutions are very different. Reporting the validity coefficient alone for either of these data sets would not reveal the abnormality of the distribution.
|
From page 161... ...
For data set A, the predicted performance score for individuals with test scores in the 0-5 interval would be 3.3, while the prediction for individuals in the 46-50 interval would be 46.5. For the least valid test, data set C, there is relatively little difference in predicted performance score between test score intervals.
|
From page 162... ...
Since the validity coefficient is based on the mean and standard deviation, this method provides a direct display of the information contained in the validity coefficient. Figures Sa-e show box-and-whisker plots for data sets A-E using the mean and standard deviation.
|
From page 163... ...
Figures 6a-e show box-and-whisker plots based on percentiles for data sets A-E. In the first interval of data set A (Figure 6a)
|
From page 164... ...
The proportions of successful individuals are given at five-point test score intervals for these data sets. For data set A (Figure 7a)
|
From page 165... ...
In Figures 10a-e, the proportion of successes is plotted at 10 five-point test score intervals for the sample data sets. By comparing these plots to the plots in Figures 9a and 9b, it is apparent that the test in sample provides the best overall discrimination between successes and failures.
|
From page 166... ...
As with other display methods, the relative validities of data sets A-C are apparent if Figures 16a-c are compared, and the abnormalities in data sets D and E are visible in Figures 16d and e. In Figure 17, successful performance has been defined again as a performance score over 25, and the test score intervals have been reduced to five 10-point intervals.
|
From page 167... ...
True positives are entered in the upper right quadrant and true negatives in the lower left quadrant. Figures l9a-e show 2 x 2 tables for the sample data sets.
|
From page 168... ...
For example, the trends in the relationships in all of the sample data sets are still apparent if the lowest two test-score intervals are deleted in all of the ten-interval test score figures presented here. In summary, while the validity coefficient is an important part of test evaluation, alternative methods exist for displaying the test-criterion relationship.
|
From page 172... ...
172 ox *
|
From page 174... ...
.... 2= 6-10 0-5 6 = 26-30 7= 31-35 3=11-15 8=36-40 4= 16-20 9= 41-45 5=21-25 10=46-50 10 FIGURE 2 Plot of mean performance score at five-point test score intervals for data sets A, B
|
From page 175... ...
Q 5 10 o o o O o ! I I J I i I 1 2 3 4 5 Test Score Interval ~ 1 1 1 1 1 1 1 2 3 4 5 Test Score Interval o 1 2 4 5 Test Score Interval a-' 20 ~ 15 -° 10 a)
|
From page 176... ...
, ~ , , , 1 2 3 4 5 d Test Score Interval , completed box-and-whisker Lo 5 1 2 4 Test Score Interval
|
From page 177... ...
ct 30 o a) Q 20 10 o 1 2 3 4 i 1 1 1 1 1 1 1 1 1 6 7 Test Score Interval 1 = 0-5 2= 3= 11-15 4= 16-20 5= 6 = 26-30 6-10 7 = 31-35 8 = 36-40 9 = 41-45 10= 46-50 177 8 9 10 FIGURE Sa Box-and-whisker plot of data set A using the mean and standard deviation.
|
From page 178... ...
7~ 50 4Q 30 o to 20 10 7 8 g 10 Test Score interval 1 = Q-S 2 = ~~D 3= 4= 78 f7-~S 5 = 6 = 28-JO 7= J/-~ J6-~O =~S 10 =FIGURE 5b Box-and-wbisker plot of dale set B using the mean and sl~d~ deviation.
|
From page 179... ...
1 = 0-5 2= 6-10 3= 11-15 4= 16-20 5 = 21-25 6 = 26-30 7 = 31-35 8= 9 = 41-45 1 0 = 36-40 46-50 179 FIGURE Sc Box-and-whisker plot of data set C using the mean and standard deviation.
|
From page 180... ...
ALLRED d 40 Cal ~ 30 o a) Do 20 10 o 1 2 3 4 5 6 7 8 9 10 Test Score Interval 1 =0-5 2= 6-10 3= 11-15 4= 16-20 5 = 21-25 6 = 26-30 7= 31-35 8 = 36-40 9 = 4 1-45 10 = 46-50 FIGURE Sd Box-and-whisker plot of data set D using the mean and standard deviation.
|
From page 181... ...
. 181 FIGURE 5e Box-and-whisker plot of data set E using the mean and standard deviat~on.
|
From page 182... ...
`q~ 1 = 0-5 6 = 26-30 2= 6-10 7= 31-35 3= 8= 9= 5 = 21-25 10 = 4= 16-20 36-40 4 1 -45 46-50 FIGURE 6a Box-and-whisker plot of data set A using the median and percentiles.
|
From page 183... ...
~ 30 o a) [L 20 10 o 1 2 1 ~ 4 5 Test Score Interval 6 7 8 9 10 1 = 0-5 2= 6-10 3= 11-15 4= 16-20 5 = 21-25 6 = 26-30 7 = 31-35 8= 36-40 9= 41-45 10= 46-50 1 1 183 FIGURE 6b Box-and whisker plot of data set B using the median and percentiles.
|
From page 184... ...
Cat ct 30 a) [L 20 10 o ; 3 4 5 _ ~ 10 6 7 8 Test Score Interval I___ 1 = 0-5 6= 2 = 6-10 7 = 3= 11-15 8 4= 16-20 9= 5 = 21-25 10 = = 26-30 3 1-35 36-40 4145 46-50 FIGURE 6c Box-and-whisker plot of data set C using the median and percentiles.
|
From page 185... ...
REPORTING THE TEST-CRITERION RELATIONSHIP d 50 40 a) C' cat 30 o 0 Q 20 10 o 185 _ _ _ 1 2 3 4 5 6 7 8 9 10 Test Score Interval .l _ 1 = 0-5 2= 6-10 3= 11-15 4= 16-20 5 = 21-25 6 = 26-30 7 = 31-35 8= 36-40 9= 41-45 1 0 = 46-50 FIGURE 6d Box-and-whisker plot of data set O using the median and percentiles.
|
From page 186... ...
C' os 30 E o a' 20 _ 10 O ] ~ 2 1 1 1 1 1 6 7 8 9 10 3 4 ~ Test Score Interval 7 8 1 = 0-5 2= 6-10 3= 11-15 4= 16-20 5 = 21-25 6 = 26-30 7= 31-35 8 = 36-40 9 = 4 1-45 10= 46-50 FIGURE be Box-and-whisker plot of data set E using, the median and percentiles
|
From page 187... ...
NOTE: Success is defined as a performance score greater than 25. b · ~ 0 10 20 30 40 50 60 70 80 90100 46-50 1 6-20 11-15 92 41-45 110 36-40 1 09 31-35 1 09 26-30 1 12 21-25 1 03 98 93 91 6-10 0-5 83 FIGURE 7b Expectancy chart for data set B: proportion of successful individuals in each test score interval.
|
From page 188... ...
NOTE: Success is defined as a performance score greater than 25. d 0 1 0 20 30 40 50 60 70 80 901 00 46-50 41 -45 36-40 31 -35 50 63 75 85 90 101 1 1-15 140 6-10 0-5 138 FIGURE 7d Expectancy chart for data set D: proportion of successful individuals in each test score interval NOTE: Success is defined as a performance score greater than 25.
|
From page 189... ...
. FIGURE 8a Expectancy chart for data set A: proportion of successful individuals at each cutting score.
|
From page 190... ...
ALLRED b @. ~ 0 10 20 30 40 50 60 70 80 90 100 46 41 36 31 26 21 16 11 6 92 202 311 420 532 635 733 826 917 None 1 000 FIGURE 8b Expectancy chart for data set B: proportion of successful individuals at each cutting score.
|
From page 191... ...
e . ~ 0 10 20 30 40 50 60 70 80 90100 46 41 36 31 26 21 16 t1 6 130 311 463 594 697 766 835 892 956 None 1 000 FIGURE Be Expectancy chart for data set E: proportion of successful individuals at each cutting score.
|
From page 192... ...
and a nondiscriminating test (9b)
|
From page 193... ...
. 1 2 3 4 5 6 7 8 9 10 Test Score Interval d l l l l l l l l l l 1 2 3 4 5 6 7 8 9 10 Test Score Interval FIGURE 10 Plots of proportion of successful individuals in ten test score intervals for data sets A-E.
|
From page 194... ...
1_ · ' 1 ~ _1 1 ~ o ~ ·. 1 1 1 · 1 1 1 0 6 11 1621 2631 3641 46 Test Cutting Interval d 0 6 11 162126313641 46 Test Cutting Interval FIGURE 11 Plots of proportion of successful individuals at ten cutting scores on test for data sets A-E.
|
From page 195... ...
who succeed and not selected (NS) who would have failed at ten cutting scores on test for data sets A-E.
|
From page 196... ...
ALLRED it:= 41-50 31-40 1 1 1-20 1 1 0-10 d Let 31-40 21-30 1 1-20 1 -— 1 0-10 202 1 218 ~3 3 3 I_ 0- 1 11- 1 21- 1 31- 1 41-1 10 1 20 1 30 1 40 1 50 1 1 1 1 61 42 1 ~ 27~;~ 3 1 21 1 33 1 31 1 13 1 1 1 1 14 1 31 1 27 1 21 1 7 40 1 28 1 17 1 13 1 3 ~ — _:ram_ . 1_~T~ 1,~ - 10 1 20 1 30 1 40 1 50 1 1113_L 1 1- 12517~-4 50 1 191 1 1 3 TO 1 1 1 1 1 ' ~ 255 1 8 1 22 1 24 1 24 1 22 281 1 31 1 23 1 21 1 19 1 7 FIGURE 13 Expectancy tables showing percent receiving, each criterion score for five intervals of test and criterion.
|
From page 197... ...
ma_ L_ - 1 -- 1 - 1 -~_ d @~ 58 57 1 100 1 100 _ ~42 . i o- 11 I 10 20 __ ~ __ ~ 35 =~ _ e 5_ | 1 15 1 48 49 51 1120 100 1120 53 47 21 30 59 41 31 40 57 100 43 21 30 21- 31 30 40 ~ ~ l ~ l 197 64 36 41 so 31- 41 40 50 65 FIGURE 14 Expectancy tables showing percent succeeding and failing at five test score intervals for data sets A-E NOTE: Success is defined as a performance score greater than 25.
|
From page 198... ...
_ over 25 l ~ 48% 52% 250r below _ _ ~3~ 60% 40% over 25 250r below 99% 1% over 25 25 or below over 25 FIGURE 15 Expectancy tables showing percent succeeding and failing at test cutting score of 25 for data sets A-E. NOTE: Success is defined as a performance score greater than 25.
|
From page 199... ...
21 9 16 147 54 l 20 _, =. 18 3 3 19 17 48 22 8 1 14 49 19 3 60 9 2 3 4 5 6 _ ~ ~ ~ ~ _ in ~ ~=e _ 1 = 0-5 6 = 26-30 2=6-10 7=31-35 3= 11-15 4= 16-20 5 = 21-25 8= 36-40 9= 41-45 10 = 46-50 199 FIGURE 16a Frequency tables for ten intervals of test score and performance score for data set A
|
From page 200... ...
c LINDA J ALLRED 2 3 5 6 5 4 6 10 11 10 16 18 5 14 15 17 10 15 14 18 14 13 20 19 13 19 12 14 17 10 11 8 11 7 4 2 5 3 = _ 3 4 5 6 ~ = , for ten intervals of test score and performance score 10 — 12 11 12 16 18 13 17 21 23 9 5 5 9 8 11 10 14 12 16 9 8 7 10 11 12 15 11 13 13 12 11 7 7 9 3 10 10 11 Al 12 11 9 6 6 7 8 9 4 9 l4 10 8 10 5 5 7 8 5 8 14 9 14 10 13 I+ t 2 12 11 10 8 11 8 10 8 5 6 1 18 14 13 14 13 10 8 8 _ 3 · · ~ ~ 3~ 1 ~ FIGURE 16c Frequency tables for ten intervals of test score and performance score for data set C
|
From page 201... ...
I ,3 ~ : it' 1 116 . L: 1 16 1 21 13 16 8 1 1 :..[ it: 123 1 19 16 1 9 1 6 1 I L Low 18 16 16 3 1 14 1 1 9 1 10 1 FIGURE 16e Frequency tables for ten intervals of test score and performance score for data set E
|
From page 202... ...
ALLRED 152 1 1 1 10 11- 2t20 30 13 31 40 166 41 50 41 Ma 79 112 1120 125 153 90 65 21 30 154 48 31 40 41 50 :~ ~ 97 O10 93 98 1120 115 100 129 89 31 40 72 41 50 149 O10 108 1 160 106 83 113 31 40 , Sir 33~1 108 O10 126 1120 25 147 21 30 137 146 31 40 41 50 201 110 41 50 l it. FIGURE 17 Frequency tables showing number succeeding and failing at five test score intervals for data sets A-E.
|
From page 203... ...
REPORTING THE TEST-CRITERION RELATIONSHIP FN TN Reject TP FP Accept TP= true positive FP= false positive TN= true negative FN= false negative Proportion of Correct Decisions 203 TP I TN TP + TN A FP ~ FN compel deck i Genierah form of the frequency table, computation of proportion of
|
From page 204... ...
~ 376 156 214 over 25 335 302 25 or below 1 250r below ~3 1 over 25 362 Over 25 Proportion of correct decisions .90 .67 .56 .66 .66 FIGURE 19 Frequency tables showing number succeeding and failing at test cutoff score for 25 for data ses A-E. NOTE: Success is defined as a performance score greater than 25.
|
From page 205... ...
NOTES: Success is defined as a performance score greater than 25. Test cutoff score of 25 determines levels of test score.
|
From page 206... ...
1 1 A .95 .91 .86 .80 40 .62 B .52 .39 .28 .20 .13 .15 .11 .11 .07 .56 .44 .39 .31 C .19 D .65 44 . .39 E .65 .58 .49 .33 .28 FIGURE 21 Effect of restriction of range by preselection on validity coefficients for data sets A-E.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.