Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Appendix A The Committeeâs Public Forums on Performance Levels for NAAL Public ForumâFebruary 27, 2004 Panelists Cynthia Baur, U.S. Department of Health and Human Services Beth Beuhlmann, U.S. Chamber of Commerce, Workforce Preparation Richard Colvin, Hechinger Institute Leslie Farr, Ohio State University Milton Goldberg, Education Commission of the States (ECS) Richard Long, International Reading Association Christopher Mazzeo, National Governors Association Gemma Santos, Miami Dade Public Schools Tony Sarmiento, Senior Service America, Inc. Linda Taylor, Comprehensive Adult Student Assessment System Robert Wedgeworth, Proliteracy Worldwide Participants Joan Auchter, GED Testing Service/American Council on Education Justin Baer, American Institutes for Research (AIR) Amy Baide, Department of Homeland Security Sandra Baxter, The National Institute for Literacy (NIFL) Jaleh Behroozi, The National Institute for Literacy (NIFL) Martha Berlin, Westat 203
204 APPENDIX A Peggy Carr, Department of Education, National Center for Education Sta- tistics (NCES) June Crawford, The National Institute for Literacy (NIFL) Elizabeth Greenberg, American Institutes for Research (AIR) Ricardo Hernandez, Office of Vocational and Adult Education (OVAE) Shannon Holmes, U.S. Conference of Mayors Eugene Johnson, American Institutes for Research (AIR) Linda Johnston Lloyd, Health Resources & Services Administration (HRSA) Michael Jones, Office of Vocational and Adult Education (OVAE) Cheryl Keenan, Office of Vocational and Adult Education (OVAE) Irwin Kirsch, Educational Testing Service (ETS) Andy Kolstad, National Center for Education Statistics (NCES) Mark Kutner, American Institutes for Research (AIR) Mariann Lemke, National Center for Education Statistics (NCES) Anne Lewis, freelance journalist Lennox McLendon, National Adult Education Professional Development Consortium Wendy Mettger, Mettger Communications Leyla Mohadjer, Westat Gerri Ratliff, Department of Homeland Security Lyn Schaefer, GED Testing Service Peggy Seufert, American Institutes for Research (AIR) Sondra Stein, The National Institute for Literacy (NIFL) Lynn Thai, Department of Homeland Security Peter Waite, Proliteracy America Dan Wagner, National Center on Adult Literacy Maria White, Department of Health and Human Services Sheida White, National Center for Education Statistics (NCES) Kentaro Yamamoto, Educational Testing Service (ETS) Representatives from State Departments of Adult Educationâ April 22-23, 2004 Bob Bickerton, Massachusetts Steve Coffman, Missouri Donna Cornelius, Massachusetts Cheryl King, Kentucky Tom Orvino, New York Ann Serino, Massachusetts Reecie Stagnolia, Kentucky Linda Young, Oklahoma
PUBLIC FORUMS ON PERFORMANCE LEVELS FOR NAAL 205 Stakeholder Questions and Alternative Versions of Performance-Level Descriptions for the National Assessment of Adult Literacy 1. In what ways did you use the results from the 1992 NALS? What were the strengths and weaknesses of these performance levels? To what extent did these performance levels provide you with the information that you needed? 2. NAAL measures skills in the areas of prose, document, and quanti- tative literacy. To what extent is it useful and informative to have different performance level descriptions for each area? Are results from the three areas of literacy used differently? If so, how? 3. The attachment presents three alternative versions of performance- level descriptions for the prose literacy scale. Sample 1 is simply a reformat- ted version of the existing performance-level descriptions with 5 levels. Sample 2 is a 4-level model, and Sample 3 is a 3-level model. Please com- ment on how many levels are needed. What types of decisions are made at the various levels? What are the critical distinctions that need to be made? 4. Level Labels: The three samples present different labels for the levels. Sample 1 uses numbers (Col. 2). Samples 2 and 3 use phrases as labels (Col. 2). In addition, Sample 3 presents a narrative description of the label (Col. 3). Please comment on these alternative labels. What types of labels are useful and informative? Feel free to make suggestions for alterna- tive labels. 5. Level Descriptions: The three samples present different ways of describing the skills represented by the performance level. Sample 1 de- scribes the tasks associated with the level (Col. 3). Sample 2 describes what an average respondent who scores at this level should be able to do (Col. 3). Sample 3 (Col. 4) describes what the average respondent who scores at this level is able to do and not able to do in probabilistic terms (i.e., likely, not likely). Please comment on these alternative ways of describing the skills associated with the levels. What types of descriptions are useful and infor- mative? Feel free to make suggestions for alternative descriptions. 6. Sample Tasks: The three samples present different ways of exem- plifying the tasks respondents who score at the level should be able to do. Samples 1 and 2 (Col. 4) are similar and provide examples drawn from actual assessment. Sample 3 (Col. 5) attempts to generalize from assessment tasks to real world tasks. Please comment on the extent to which these exemplifications are useful and informative. 7. Relationships Between Prose Scores and Background Data: Samples 2 and 3 present the relationships between NAAL scores and key real-world factors as measured on the background questionnaire. Sample 2 (Col. 5) uses societal factors (income, education, voting) and Sample 3 (Col. 6) uses reading related factors. (Please be aware that the percentages in-
206 APPENDIX A cluded in these examples are purely hypothetical. If we were to recommend this format, the percentages would be based on analyses with NAAL data.) Please comment on the utility of this type of information.
PUBLIC FORUMS ON PERFORMANCE LEVELS FOR NAAL 207 SAMPLES ARE ON THE FOLLOWING PAGES
208 Sample 1: Five-Level Model, Based on Current PLDs for Prose Literacy (Col. 1) (Col. 2) (Col. 3) (Col. 4) Level Label Description of tasks Sample NAAL tasks associated with the level I Level 1 Most of the tasks in this level require the reader â¢ Locate one piece of information in a sports to read relatively short text to locate a single article. piece of information which is identical to or â¢ Identify the country in a reading passage. synonymous with the information given in the â¢ Underline sentence explaining action stated in question or directive. If plausible but incorrect short article. information is present in the text, it tends not to be located near the correct information. II Level 2 Some tasks in this level require readers to locate â¢ Underline the meaning of a term given in a single piece of information in the text; government brochure on supplemental security however, several distractors or plausible but income. incorrect pieces of information may be present, â¢ Locate two features of information in sports or low-level inference may be required. Other article. tasks require the reader to integrate two or more â¢ Interpret instructions from an appliance pieces of information or to compare and warranty. contrast easily identifiable information based on a criterion provided in the question or directive. III Level 3 Tasks in this level tend to require readers to make â¢ Write a brief letter to complain about an error literal or synonymous matches between the text and on a bill. information given in the task, or to make matches â¢ Read a news article and identify a sentence that that require low-level inferences. Other tasks ask provides interpretation of a situation. readers to integrate information from dense or â¢ Read lengthy article to identify two behaviors lengthy text that contains no organizational aids that meet a stated condition.
such as headings. Readers may also be asked to generate a response based on information that can be easily identified in the text. Distracting information is present, but is not located near the correct information. IV Level 4 These tasks require readers to perform multiple- â¢ State in writing an argument made in a feature matches and to integrate or synthesize newspaper article. information from complex or lengthy passages. â¢ Explain differences between two types of More complex inferences are needed to perform employee benefits. successfully. Conditional information is frequently â¢ Contrast views expressed in two editorials on present in tasks in this level and must be taken into technologies available to make fuel-efficient cars. consideration by the reader. V Level 5 Some tasks in this level require the reader to search â¢ Compare approaches stated in a narrative on for information in dense text which contains a growing up. number of plausible distractors. Others ask readers â¢ Summarize two ways lawyers may challenge to make high-level inferences or use specialized prospective jurors. background knowledge. Some tasks ask readers to â¢ Interpret a phrase from a lengthy news article. contrast complex information. 209
210 Sample 2: Four-Level Model for Prose Literacy (Col. 1) (Col. 2) (Col. 3) (Col. 4) (Col. 5) The average respondent who scores at this level should be Sample of NAAL tasks the average Relationships with societal Level Label able to: respondent should be able to: factors I Below Basic â¢ Identify letters and numbers. â¢ Locate specific information on a â¢ >50% of being poor or â¢ Point to orally presented words. food label. near poor â¢ Read aloud words, phrases. â¢ Identify a specific word in an â¢ >50% chance of not or short sentences. advertisement. having a U.S. high school â¢ Locate an address in an diploma advertisement. â¢ <50% chance of voting in most recent election II Basic â¢ Locate a single piece of â¢ Locate one piece of information â¢ 25-50% of being poor or information in a brief written in a sports article. near poor text that uses organizational â¢ Identify the country in a reading â¢ 50% chance of having at aids such as headings. passage. least a high school â¢ Make simple inferences. â¢ Interpret brief instructions from diploma â¢ Provide appropriate answers to warranty information. â¢ 50-60% chance of voting questions that require integrating in most recent election two or more pieces of information.
III Intermediate â¢ Generate a response based on â¢ Write a brief letter to complain â¢ 10-25% of being poor or information that is easily about an error on a bill. near poor identifiable in dense or lengthy â¢ Read a magazine article and â¢ 50% chance of having at text. identify the facts that support a least some college â¢ Integrate information from specific inference. â¢ 60-70% chance of voting dense or lengthy text. in recent election IV Advanced â¢ Search for information that is â¢ State in writing an argument â¢ <10% of being poor or in dense texts. made in a newspaper article. near poor â¢ Integrate or synthesize â¢ Explain the differences between â¢ 50% chance of having at information from complex texts. two types of employee benefits. least an associates college â¢ Perform complex inferences. â¢ Contrast views expressed in two degree editorials. â¢ 70-80% chance of voting in recent election NOTES: NAAL background data will be analyzed to provide information about the likely relationships between literacy levels and the three societal factors included in Column 5; the current figures are for demonstration purposes only. Many of these descriptions may not generalize to English language learners (ELL). ELLs may be literate in languages other than English, and the relationships with societal factors (Col. 5) may be different for ELLs than for native English speakers. The response mode may also affect ELLsâ performance, since writing skills in English may develop at a slower pace than reading or speaking skills. 211
212 Sample 3: Three-Level Model for Prose Literacy (Col. 1) (Col. 2) (Col. 3) (Col. 4) (Col. 5) (Col. 6) Description Sample real-world tasks of label for The average the average respondent individuals who score respondent who scores at this level should be Relationship to reading- Level Label at this level: at this level: able to do: oriented factors I Minimally Are not able to â¢ Is likely to be able to â¢ Place signature in â¢ >80% chance of never Literate in independently handle identify letters and proper place on reading the newspaper English most of the tasks of numbers; point to words a form. â¢ >80% chance of needing daily living that when they are presented â¢ Follow a simple a lot of help with printed require literacy skills orally; orally read recipe. information in English. individual words, â¢ Identify an address phrases, or short in an advertisement. sentences. â¢ Is not likely to able to read connected text. II Somewhat Should be able to â¢ Is likely to be able to â¢ Follow directions on â¢ 40-80% chance of never Literate in independently handle locate a single piece of a medicine bottle. reading the newspaper English some of the tasks of information in a brief â¢ Read aloud to a â¢ 40-80% chance of daily living that require written text with child in preschool needing a lot of help literacy skills in organizational aids, such or elementary grade. with printed English. as headings; to integrate information two or more pieces of information; and to compare and contrast easily identifiable information.
â¢ Is not likely to be able to integrate or synthesize information from dense text with no organizational aids. III Fully Should be able to â¢ Is likely to be able to â¢ Read and understand â¢ <10% chance of never Literate in independently handle integrate or synthesize New York Times. reading the newspaper English most of the tasks of information from dense â¢ Understand a â¢ <10% chance of needing daily living that require or lengthy text that proposition on a a lot of help with printed literacy skills in contains no organizational ballot. information English. aids; and to generate a response based on information that can be identified in the text. NOTES: NAAL background data will be analyzed to provide information about the likely relationships between literacy levels and the reading oriented factors in Column 6; the current figures are for demonstration purposes only. These descriptions may not generalize to English language learners (ELL). ELLs may be literate in languages other than English, and their performance on real world tasks (Col. 5 and 6) may be different than for native English speakers. The response mode may also affect ELLsâ performance, since writing skills in English may develop at a slower pace than reading or speaking skills. 213