Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
1 Introduction I n 1992, the U.S. Department of Education sponsored the first nation- wide assessment of adultsâ English literacy skills. This assessment, called the National Adult Literacy Survey (NALS), was a household survey of a nationally representative sample of 26,000 adults aged 16 years and older. A decade later, the Department of Education implemented plans to revise and readminister the assessment. The revised assessment, renamed the National Assessment of Adult Literacy (NAAL), was designed to pro- duce some new information while retaining enough consistency to allow comparisons with the 1992 results to evaluate trends in adultsâ literacy skills over the ensuing decade. The department requested that the National Research Council (NRC) provide advice about creating performance levels for reporting the assessment results. This report details the work and find- ings of the Committee on Performance Levels for Adult Literacy. In this chapter, we first provide a description of the problem and then lay out the context for the committeeâs work and its approach to its charge. PROBLEM STATEMENT NALS measured literacy skills using a wide array of tasks that reflected the types of materials and demands that adults encounter in their daily lives. The assessment measured three types of literacy: (1) prose literacy was a measure of skill in using information presented in textual formats (e.g., a newspaper article); (2) document literacy reflected skill in using informa- tion presented in graphs, figures, and tables (e.g., a bus schedule); (3) and quantitative literacy measured skill with using and performing arithmetic 15
16 MEASURING LITERACY: PERFORMANCE LEVELS FOR ADULTS operations on numbers presented in texts or documents (e.g., in a bar graph). NALS was intended to profile and describe the English literacy skills of adults in the United States. It was not designed, however, to indicate whether or not any particular level of skills was adequate. That is, the test develop- ment process did not involve identifying the level of skills adults need in order to function adequately in society. Thus, it was not intended to sup- port inferences about what constitutes an adequate level of literacy. To provide information that could be more easily understood and used by policy makers and the public, the test designers grouped scores into five categories, or performance levels (called NALS Level 1, NALS Level 2, NALS Level 3, etc.). Brief descriptions of the levels were provided, and the results were reported by scores as well as by the percentage of those sur- veyed whose scores fell into each performance level. The performance levels used for reporting NALS results were intended simply to describe adultsâ literacy skills, not to suggest a level of perfor- mance that could be regarded as sufficient to function in society. When findings from the 1992 survey were released, however, many unsupported inferences were made about the results. The five performance levels were interpreted and discussed by policy makers, the media, and the public as if they represented standards for the level of literacy that adults should have. Some, for example, referred to the lowest two levels as âinadequate,â so low that adults with these skills would be unable to hold a well-paying job (Gray, 1993; Kaplan, 1993). As many as 47 percent of adults in the United States, or about 90 million people, fell into these bottom two levels. In his September 8, 1993, press release, Secretary of Education Richard W. Riley stated that âthe vast majority of Americans . . . do not have the skills they need to earn a living in our increasingly technological society,â which led to such headlines as âLiteracy of 90 Million is Deficientâ (Jordan, 1993). The results of the assessment and these unsupported conclusions about the results provoked widespread controversy in the media and among experts on adult literacy about the extent of literacy problems in the country. According to the assessment designers, however, NALS was not in- tended to provide firm answers to questions about the literacy skills essen- tial for individuals to succeed in society. The procedures for designing the test did not involve identifying what adults need to know and be able to do to adequately function in society and to obtain a well-paying job. The performance levels were designed to demonstrate the range of literacy de- mands placed on adults as part of their daily lives but not to shed light on the types of literacy demands associated with particular contexts of life. For example, the performance levels were not intended to support conclusions about the specific levels of English literacy required to obtain, remain, or advance in a particular occupation, to manage a household, or to obtain
INTRODUCTION 17 legal or community services (Kirsch et al., 1993). In addition, the process for determining the performance levels did not involve procedures typically used for setting standards. Thus, although policy makers and the press interpreted NALS performance levels as if they represented standards for what adults should know and be able to do, these sorts of interpretations are not aligned with the purpose of the assessment. In preparation for the release of the 2003 results, the Department of Education decided to focus specific attention on how NAAL results should be reported and interpreted. The department sought advice from the NRC and requested that its Board on Testing and Assessment (BOTA) recom- mend performance levels to use in reporting NAAL results. The Committee on Performance Levels for Adult Literacy was established to provide this advice. The committee included individuals with a broad array of expertise, including adult education, educational measurement and standard setting, English for speakers of other languages, literacy and literacy measurement, psychology, political science, public health, sociology, statistics, and survey methodology. The committee was charged to: â¢ Review and evaluate the processes and procedures used for deter- mining the performance levels for the 1992 NALS and â¢ Recommend a set of performance levels for the 2003 NAAL that are valid and appropriate and permit comparisons between the 1992 and 2003 results. In this report, we use several terms that need to be defined: Performance level: a range of test scores that reflect similar levels of knowledge, skills, and capabilities as measured on a test. Performance-level description: the description of the knowledge, skills, and capabilities test takers need to demonstrate in order to be classified into a specific performance level. Cut score: the score that separates one performance level from another performance level. Standard setting: the procedures used to determine the cut scores. Many standard-setting methods rely on judgments made by a set of panelists selected for their expertise in subject areas evaluated on the test; they use established systematic procedures for collecting these judgments. Other methods make use of statistical information describing discrimi- nation based on the test scores between existing externally defined groups (e.g., masters and nonmasters). In addition, there are a number of agencies involved in this work:
18 MEASURING LITERACY: PERFORMANCE LEVELS FOR ADULTS â¢ National Center for Education Statistics (NCES): the statistical agency of the U.S. Department of Education that provided oversight for NALS and NAAL. â¢ Educational Testing Service (ETS): one of the contractors to NCES that assisted with work on NALS. â¢ American Institutes for Research (AIR): one of the contractors to NCES that assisted with work on NAAL. â¢ Westat: one of the contractors to NCES that assisted with work on both NALS and NAAL. â¢ NRC: the organization that NCES requested to recommend perfor- mance levels for NAAL. â¢ BOTA: the NRC board that provided oversight for the Com- mittee on Performance Levels in Adult Literacy. OVERARCHING GOALS When BOTA agreed to take on this project, it set an overarching goal for the committeeâs work: to demonstrate a process for determining perfor- mance levels and the associated cut scores that represents exemplary prac- tice. The board provided oversight for the committeeâs work and encour- aged the committee to approach the process in an open manner that would enhance public understanding of the results and to use a scientifically in- formed process that would result in performance levels that are valid and defensible. Throughout its work, the committee sought to model exemplary prac- tices. However, as with much in the assessment field, one must always strike a balance between idealistic goals and practical constraints. For any testing program, practical and resource considerations limit efforts to strive for the ideal. For the committee, the challenge was to design a process for developing performance levels retroactively. To explain, the committeeâs work began late in 2002, just before the actual data collection for NAAL was scheduled to begin. By that time, most of the decisions about NAAL had already been made. This placed certain limitations on our work and substantially narrowed the range of options available to us as we tried to develop new performance levels. For example, when we began our work, development of the tasks that make up the literacy assessment was finished, and the background questionnaire admin- istered in conjunction with NAAL had already been finalized. This meant that our choices about new performance levels were limited due to prior decisions about test development, the scope of the content and skills to be covered, and the range of difficulty of the items included on the assessment. In addition, any analyses we chose to conduct to evaluate the relationships between new performance levels and background characteristics (i.e., to
INTRODUCTION 19 appraise the reasonableness and validity of the cut scores associated with the performance levels) were restricted by predetermined choices about the questions included on the background questionnaire. Developing performance levels retroactively, as the time constraints on the committee required, clearly is not best assessment practice. Despite our reservations, we accepted the charge to assist the Department of Education with the challenging problem of communicating about adultsâ literacy skills and improving understanding and interpretation of NAAL findings. LIMITATIONS ON INFERENCES ABOUT LITERACY SKILLS Prior decisions during the test development phase and the process of developing administrative procedures also affect the inferences that can be based on NALS and NAAL results, regardless of the performance levels used for reporting. The first limitation is posed by the definition of literacy that guided test development. As has long been recognized by researchers and practitioners in the field of literacy studies, literacy can be defined in a variety of ways. In recognition of the understanding that literacy is not a unitary construct, it has become customary in the field to speak of âmul- tiple literaciesâ rather than âliteracyâ (see Wagner, 2004, for further dis- cussion of this issue). It follows, then, that any assessment will be able to test some types of literacy but not others. The version of literacy that is tested through NAAL is based on an information-processing view of reading and cognition. Therefore, the diffi- culty of test items is varied along such parameters as the density of informa- tion and the complexity of the text structure, factors that would affect the ease or difficulty of reading. For example, if the information required to answer a question about a paragraph is found in the first sentence of that paragraph, the literacy task is presumed to be easier than if a person is required to read further or to sort through distracting information. This information-processing view of what makes reading easy or difficult, or more or less complex, shaped what the test is able to measure. There are other theoretical understandings of literacyâfor example, the view of read- ing as a process of constructing meaning in interaction with a text rather than just extracting meaning from a text. It is important to recognize that the particular theoretical stance that underpins NAAL will have implica- tions for the development and analysis of test items. Furthermore, the NAAL items were designed to measure literacy ac- cording to an information-processing model for a distinct set of purposesâ for what might be called âfunctionalâ literacy or the everyday kinds of tasks that people may encounter in the course of their daily lives. Such tasks may include reading a bus schedule, deciphering an advertisement, or filling out a form. These are valuable and necessary types of literacy, but they
20 MEASURING LITERACY: PERFORMANCE LEVELS FOR ADULTS exclude other purposes for literacy, such as reading literature for pleasure, engaging in literacy-related activities for religious purposes, or studying lengthy and complex texts in order to acquire new knowledge. Nor does NAAL ask respondents, in the context of functional literacy, to do very much writing. Many everyday literacy tasks do not, in fact, require as much writing as reading. Still, literacy has come to include, in many popular as well as academic usages, the production of text rather than only the recep- tion of it. This understanding of literacy is not captured by NAAL. Finally, it is important to remember that although the test simulates materials and activities that adults may encounter in their daily lives, it does not capture how they may actually engage with those materials in a real- world setting. A good example of this distinction is that NAAL requires participants to work alone, without help or assistance, whereas adults in real life can engage in literacy tasks jointly or in collaboration. Investigating how adults engage with literacy in everyday life would require different research methods. Another limitation posed by the definition of literacy that guided test development has to do with the now ubiquitous presence of computers and information technology in the lives of most adults and children and the impact of these on literacy. NAAL is a paper-and-pencil test and therefore not designed to assess adultsâ performance on computer-mediated literacy tasks. This means, for example, that the kinds of reading required to navi- gate the Internet or interact with hypertextual or hypermedia reading and writing environments are not represented. Moreover, the assessment measures only English literacy skills. Neither NALS nor NAAL was designed to yield information about individuals who are not literate in English but who are fully literate in another language. Literacy is not language specific, however, and English literacy is not the only literacy that matters. At the United States-Mexico border, for ex- ample, as well as in cities with a high percentage of immigrants (e.g., San Francisco, Los Angeles, Chicago, New York, Seattle, Miami), biliteracy, the ability to read and write in two languages, affords a number of social and economic benefits. In terms of the job opportunities available to them, individuals in California and Texas who are monolingual in English or in another language are at a distinct disadvantage compared with bilingual adults. When literacy assessments test only literacy in English, they ignore the literacy skills of immigrants and refugees who are able to read and write in their native language, skills that should be considered when a nation with the diversity of the United States paints a picture of the literacy abilities of its citizens.
INTRODUCTION 21 COMMITTEEâS APPROACH TO THE CHARGE The committee began its work with a review of empirical studies on the relationships between NALS literacy scores and data collected via the back- ground questionnaire (see Smith, 2003). A draft of this paper was ready prior to the committeeâs first meeting and served as the starting place for much of our subsequent work. Between December 2002 and October 2004, the committee held six full committee meetings. The first three meetings focused on gathering informa- tion about NALS and NAAL: how literacy is defined for the two assess- ments and the tasks used to measure these skills, how the performance levels were determined for NALS, who is included in the sample of respon- dents, how the assessment is administered, and what sorts of data are collected on the background questionnaire administered in conjunction with the assessment. The contractor responsible for identifying the sample, collecting the data, and determining the sampling weights for both assess- ments was Westat. The contractor responsible for developing the test and analyzing and reporting the results was the ETS in 1992 and the AIR in 2003. Our information gathering included a review of available written documentation about the 1992 procedures (primarily from Campbell, Kirsch, and Kolstad, 1992; Kirsch et al., 1993; Kirsch, Jungeblut, and Mosenthal, 2001) as well as conversations with those who worked on the 1992 NALS and the 2003 NAAL. After the second meeting, the committee prepared and issued a letter report to address three areas of concern, two related to initial screening procedures for determining survey participants and whether they would take the low-level Adult Literacy Supplemental Assessment or not, and the other related to sampling procedures. The letter report can be found at http://www.nap.edu/catalog/10762.html. To enhance our understanding of the assessment, the committee ar- ranged to observe the training for NAAL interviewers, the individuals who would be visiting households to collect data. In addition, several committee members accompanied NAAL interviewers to sampled households to ob- serve the actual administration of the assessment. The committee also gath- ered information about how literacy is defined for international assess- ments and about how policy strategies are implemented in other countries. The committeeâs fourth meeting in February 2004 included a public forum to hear from stakeholders regarding the ways in which NALS results are used, the ways stakeholders anticipate using NAAL results, and the types of information that stakeholders would like to see included in reports of NAAL results. At this meeting, the committee provided samples of per- formance levels and their descriptions and asked for stakeholdersâ reac- tions. They also solicited feedback from directors of adult education in
22 MEASURING LITERACY: PERFORMANCE LEVELS FOR ADULTS states that had subsidized additional sampling during the NAAL data col- lection process in order to obtain state-level NAAL results. The list of the stakeholders who were contacted and who participated in the committeeâs activities appears in Appendix A. The committee also sponsored two standard-setting sessions, one with the 1992 data and test questions and one with the 2003 data and test questions, to obtain judgments about cut scores for new performance levels and to receive additional feedback about performance-level descriptions. Participants in these standard settings included adult educators (directors, coordinators, and teachers of adult education services), adult literacy re- searchers, middle and high school classroom teachers, industrial and orga- nizational psychologists, and human resources specialists. OVERVIEW OF THE REPORT Background information about the measurement of adult literacy, about the two assessments (NALS and NAAL), and about adult education is presented in Chapter 2. In Chapter 3, we review the process and proce- dures for determining the five performance levels used to report the 1992 NALS results. Chapters 4 and 5 describe our processes for developing new performance-level descriptions and determining the scores that separate the performance levels (cut scores). In Chapter 6, we discuss strategies for reporting and communicating about results for NAAL and suggest ways for using the results. The final chapter contains our suggestions for im- proving future assessments of adultsâ literacy skills.