Measuring Literacy Performance Levels for Adults (2005) / Chapter Skim
Currently Skimming:
4 Determining Performance Levels for the National Assessment of Adult Literacy
4 Determining Performance Levels for the National Assessment of Adult Literacy
Pages 87-107
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 87... ...
On the basis of our review, the committee decided to embark on a process for defining a new set of performance levels for the National Assessment of Adult Literacy (NAAL)
|
From page 88... ...
In this chapter, we take up the first three questions and describe our process for determining the number of performance levels and their purposes. In Chapter 5, we discuss our procedures for determining the descriptions of the levels and the associated cut scores.
|
From page 89... ...
· The lowest level was so broad that it made it difficult to identify the truly nonliterate population. · Having three literacy scales crossed with five performance levels produced so much information that it was difficult to present the results to policy makers and others.
|
From page 90... ...
Discussion Sessions Sponsored by the Committee The committee arranged for several opportunities to obtain feedback from various stakeholders regarding the ways NALS results were used, the ways stakeholders anticipate using NAAL results, and the types of information that stakeholders would like to see included in reports of NAAL results. We collected information about the types of inferences that might be based on NAAL results, the policy and program decisions that might be made, the number of performance levels needed to support these inferences and uses, and the types of performance-level descriptions that would communicate appropriately to the various audiences for NAAL results.
|
From page 91... ...
NCES staff members present at the forum pointed out that the data files would be made available and score data provided so that users could group the scores based on the score ranges used for the 1992 performance levels or any other grouping that fit their particular needs. With regard to qualitative names for the levels, some favored labels for the levels, noting that this can provide a means for succinctly and accurately communicating the meaning of the levels (e.g., satisfactory literacy skills, deficient literacy skills)
|
From page 92... ...
We hypothesized that such analyses might reveal break points in the distribution of literacy scores at which individuals were at an unacceptably high risk for encountering social and economic hardships. This type of information might lead to obvious choices in performance levels, standards, or both.
|
From page 93... ...
In the end, however, we came to realize that the background questions did not provide the information needed for the analyses we had hoped to conduct. Overall, the relationships between literacy scores and the background variables did not suggest obvious break points that could be used in defining performance levels.
|
From page 94... ...
In the end, we concluded that the background information could not be used by itself to identify categories of literacy skills but could be used to evaluate the reasonableness of cut scores resulting from a more typical standard-setting procedure. In Chapter 5, we therefore use the results of our analyses as a complement to a standard- setting procedure using the test items themselves, rather than as an alternative to such a standard setting procedure.
|
From page 95... ...
(1993) , shows the percentages of adults who, according to federal guidelines, were poor or near poor or who had received food stamps in the year prior to the assessment at each of the five 1992 performance levels for prose literacy.
|
From page 96... ...
. Comparison of these boxplots shows that prose literacy scores tend to increase as annual income increases.
|
From page 97... ...
The boxplots indicate that approximately three-quarters of the people receiving AFDC or food stamps scored below 380, and three-quarters of the people receiving interest or dividend income scored above 275. To the extent that it is appropriate to link literacy level in a causal way to the set of behaviors that ultimately influence an individual's financial success, this figure suggests that a cut score somewhere in the 273 to 380 range might be a rough dividing line between individuals who are experiencing functional difficulties and individuals who are experiencing financial success.
|
From page 98... ...
(1996) , we examined the mean quantitative score for occupa 3 The method of identifying the occupations of NAAL respondents, obviously crucial to examining the relationships between literacy scores and occupational classification, depends on accurate classification of a respondent's narrative description of their occupation into a
|
From page 99... ...
Nonetheless, while the information does not seem to be useful in determining specific performance levels or identifying cut scores, it does demonstrate how opportunities to enter into white-collar, higher paying occupations increase as literacy skills increase. That is, for those at higher literacy levels, opportunities are readily accessible; for those at lower levels of literacy, the opportunities to obtain higher paying jobs are more limited.
|
From page 100... ...
100 MEASURING LITERACY: PERFORMANCE LEVELS FOR ADULTS TABLE 4-1 Occupation with at Least 30 Respondents in NALS, 1992 JOB Quantity Level 1 None Level 2 Janitor 234 Sewing-machine operator, semiautomatic 243 Orderly 251 Construction worker II 253 Bus driver 257 Cook 261 Physical therapy aide 264 Cashier II 273 Level 3 Teacher aide II 276 Farmworker, livestock 277 Truck driver, heavy 278 Clerk, general 278 Mail-distribution-scheme examiner 285 Sales clerk 285 Waiter/waitress, formal 285 Nurse, licensed practical 286 Carpenter 289 Chef 289 Correction officer 291 Automobile mechanic 292 Manager, retail store 296 Assistant construction superintendent 297 Manager, property 297 Manger, food service 298 Teller 299 Secretary 303 Legal secretary 306 Nurse, office 306 Poultry farmer 307 Disbursement clerk 307 Superintendent, construction 311 Police officer I 311 Manager, department 315 Sales agent, insurance 316 Caseworker 319 Sales agent, real estate 322 Director, educational program 323 Teacher, physically impaired 324
|
From page 101... ...
To the extent that it is appropriate to link literacy level in a causal way to the set of behaviors that ultimately influences an individual's occupational choice, Figure 4-6 suggests that a cut score somewhere in the range 291-301 might be a rough dividing line between individuals who work in occupations that require minimal formal education and training (and hence are lower paying) and individuals who work in occupations that require formal education and training (and hence are higher paying)
|
From page 102... ...
102 ps. ts s i ent grou i c s rs to omp.
|
From page 103... ...
. This observation led the committee to question the need for reporting three separate literacy scores.
|
From page 104... ...
. If differential associations were found, there would be empirical support for using the separate dimensions to guide decision making about adult literacy policy and programs.
|
From page 105... ...
We designed a process for determining the performance levels that was iterative and that integrated information obtained from several sources: our analyses of NALS literacy and background data, feedback from stakeholders, and a review of the test items. This process is described below.
|
From page 106... ...
To reinforce this, we have intentionally avoided the use of the term "proficient" in the labels for the performance levels. RECOMMENDATION 4-1: The 2003 NAAL results should be reported using five performance levels for each of the three types of English literacy: nonliterate in English, below basic literacy, basic literacy, intermediate literacy, and advanced literacy.
|
From page 107... ...
Feedback from stakeholders emphasized the usefulness of creating levels for NAAL aligned with the NRS levels. Although it was not possible to establish a clear one-to-one correspondence between NAAL performance levels and the NRS levels, there appears to be a rough parallel between nonliterate in English and the NRS beginning literacy level; between below basic and the NRS beginning basic and low intermediate levels; and between basic and the NRS high intermediate level.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.