Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
MEASURING LITERACY PERFORMANCE LEVELS FOR ADULTS Committee on Performance Levels for Adult Literacy Robert M. Hauser, Christopher F. Edley, Jr., Judith Anderson Koenig, and Stuart W. Elliott, editors Board on Testing and Assessment Center for Education Division of Behavioral and Social Sciences and Education
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, N.W. Washington, DC 20001 NOTICE: The project that is the subject of this report was approved by the Govern- ing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineer- ing, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropri- ate balance. This study/publication was supported by Cooperative Agreement No. R215U990016 between the National Academy of Sciences and the U.S. Department of Education. Additional funding was provided by an award from the Presidentsâ Committee of The National Academies. Any opinions, findings, conclusions or recommendations expressed in this publication are those of the authors and do not necessarily reflect the views of the organizations or agencies that provided support for the project. Library of Congress Cataloging-in-Publication Data Measuring literacy : performance levels for adults / Committee on Performance Levels for Adult Literacy ; Robert M. Hauser ... [et al.], editors. p. cm. Includes bibliographical references. ISBN 0-309-09652-9 (pbk.) â ISBN 0-309-55015-7 (pdf) 1. Functional literacyâUnited StatesâEvaluation. I. Hauser, Robert Mason. II. National Research Council (U.S.). Committee on Performance Levels for Adult Literacy. LC149.7.M4 2005 302.2â²244âdc22 2005021251 Additional copies of this report are available from the National Academies Press, 500 Fifth Street, N.W., Lockbox 285, Washington, DC 20055; (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area); Internet, http:// www.nap.edu. Printed in the United States of America Copyright 2005 by the National Academy of Sciences. All rights reserved. Suggested citation: National Research Council. (2005). Measuring literacy: Perfor- mance levels for adults. Committee on Performance Levels for Adult Literacy, R.M. Hauser, C.F. Edley, Jr., J.A Koenig, and S.W. Elliott, editors. Board on Testing and Assessment, Center for Education. Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Acad- emy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engi- neers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineer- ing programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Wm. A. Wulf is presi- dent of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academyâs purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Coun- cil is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. Wm. A. Wulf are chair and vice chair, respectively, of the National Research Council. www.national-academies.org
COMMITTEE ON PERFORMANCE LEVELS FOR ADULT LITERACY CHRISTOPHER F. EDLEY, Jr. (Co-Chair), School of Law, University of California, Berkeley ROBERT M. HAUSER (Co-Chair), Center for Demography of Health and Aging, University of WisconsinâMadison JUDITH A. ALAMPRESE, Abt Associates Inc., Bethesda, MD MICHAEL X. DELLI CARPINI, Annenberg School for Communication, University of Pennsylvania VIVIAN L. GADSDEN, National Center on Fathers and Families, University of Pennsylvania ANDREW J. HARTMAN, Independent Consultant, Denver, CO GLYNDA A. HULL, Graduate School of Education, University of California, Berkeley REBECCA A. MAYNARD, University Trustee Professor of Education and Social Policy, University of Pennsylvania LORRAINE M. McDONNELL, Department of Political Science, University of California, Santa Barbara LARRY MIKULECKY, Language Education Department, Indiana University ROBERT J. MISLEVY, Department of Measurement, Statistics, and Evaluation, University of Maryland NORMAN G. PETERSON, Satisfaction Performance Research (SPR) Center, Inc., Minneapolis, MN JOHN P. POGGIO, Department of Psychology and Research in Education, University of Kansas RIMA E. RUDD, Department of Health and Social Behavior, Harvard School of Public Health MARY JANE SCHMITT, TERC, Cambridge, MA DAVID M. THISSEN, Department of Psychology, University of North Carolina, Chapel Hill HEIDE SPRUCK WRIGLEY, LiteracyWork Associates, San Mateo, CA JUDITH A. KOENIG, Study Director STUART W. ELLIOTT, Senior Program Officer CONSTANCE F. CITRO, Senior Program Officer LORI HOUGHTON WRIGHT, Program Officer ANDREW E. TOMPKINS, Research Assistant LYNNE STEUERLE SCHOFIELD, Research Assistant MICHAEL DECARMINE, Senior Project Assistant v
BOARD ON TESTING AND ASSESSMENT 2003-2004 EVA L. BAKER (Chair), The Center for the Study of Evaluation, University of California, Los Angeles LORRAINE MCDONNELL (Vice-Chair), Department of Political Sciences, University of California, Santa Barbara, CA LAURESS L. WISE (Vice-Chair), Human Resources Research Organization, Alexandria, VA CHRISTOPHER F. EDLEY, JR., Harvard Law School EMERSON J. ELLIOTT, Consultant, Arlington, VA MILTON D. HAKEL, Department of Psychology, Bowling Green State University ROBERT M. HAUSER, NAS, Institute for Research on Poverty, Center for Demography, University of Wisconsin, Madison PAUL W. HOLLAND, Educational Testing Service, Princeton, NJ DANIEL M. KORETZ, Graduate School of Education, Harvard University EDWARD P. LAZEAR, Stanford University, Stanford, CA RICHARD J. LIGHT, Graduate School of Education and John F. Kennedy School of Government, Harvard University ROBERT J. MISLEVY, Department of Measurement and Statistics, University of Maryland JAMES W. PELLEGRINO, University of Illinois, Chicago, IL LORETTA A. SHEPARD, School of Education, University of Colorado, Boulder KENNETH I. WOLPIN, Department of Economics, University of Pennsylvania STUART W. ELLIOTT, Director LISA D. ALSTON, Administrative Coordinator vi
Foreword I n the summer of 2002, the National Center for Education Statistics approached the Board on Testing and Assessment (BOTA) of the Na- tional Academies with a request for assistance in setting performance standards for their upcoming assessment of adultsâ literacy skills. This was a unique request for BOTA. Over the years, BOTA had explored and provided advice on a variety of issues related to setting performance stan- dards on educational achievement testsâfrom discussions of the standards- based reform movement in education and its effects on various groups of student to recommendations for best practice in setting performance stan- dards for the National Assessment of Educational Progress. Undertaking the process of actually setting performance standards, however, was a new endeavor for BOTA. Setting performance standards is an inherently judgmental task. The process involves determining the number and nature of the performance levels used for reporting the test results (such as âproficientâ or âbelow basicâ), the descriptions of the levels, and the test scores used to demark the range of scores associated with each performance level. A variety of standard-setting procedures are documented in the measurement literature, procedures that lay out the methodologies and best practices, but all ulti- mately rely on the judgments of testing experts, policy makers, and other stakeholders and users of the test results. The answers to questions such as âHow much literacy is enough?â or âWhat constitutes a literacy prob- lem?ââeither for an individual or for society as a wholeâare not exclu- sively technical. Bringing scientific principles to the process of collecting and summarizing judgmental information was a daunting challenge. vii
viii FOREWORD Consistent with its mission, BOTA accepted this request. Formed to provide scientific advice to policy makers and the public about critical issues related to testing and assessment, BOTA draws on the interdiscipli- nary expertise of its members to bring a wide variety of perspectives to bear on such complex problems. Members of BOTA welcomed this opportunity to demonstrate a scientific approach to the problem and offer advice about two critical and timely issues: how to help describe and understand the literacy skill levels of adults in this country and how to set performance standards in a meaningful and technically valid way. Under the auspices of BOTA, the Committee on Performance Levels for Adult Literacy was formed as an interdisciplinary panel of 17 members with expertise in the areas of adult education and adult literacy, economics, educational measurement and standard setting, law, political science, and sociology. BOTA remained actively involved with this work, with four members of BOTA serving on the committee, two of whom served as committee co-chairs. Members of BOTA provided ongoing oversight for the work in formulating the committeeâs charge and overall approach to its tasks, identifying individuals to serve on the committee, offering feedback to the committee, and reviewing the draft report and recommendations. The committee was convened in December 2002 and held six meetings. During the course of its work, the committee solicited feedback from stake- holders using a variety of mechanisms, including a public forum held in February 2004. The committee also convened two standard-setting ses- sions, in July and September 2004, which involved experts in adult literacy, adult education, teaching, and other relevant fields. This report presents the findings and recommendations that resulted from these activities and the committeeâs deliberations. It is BOTAâs hope that this report will be of use to a variety of audiences: the U.S. Department of Education in its final decision making about performance standards for its adult literacy assess- ments and plans for future assessments; policy makers and practitioners in the adult literacy field as they make programmatic decisions; and the psy- chological measurement community as they grapple with the complex tech- nical and judgmental issues involved in the task of setting valid perfor- mance standards in similar situations. BOTA extends its sincere appreciation to the committee for its hard work on this challenging project, and particularly to Christopher Edley, Jr., and Robert Hauser, who served as co-chairs. Lauress L. Wise, Chair Board on Testing and Assessment
Acknowledgments T he work of the Committee on Performance Levels for Adult Literacy benefited tremendously from the contributions of many people, and the committee is grateful for their assistance and support. First, we wish to acknowledge the National Center for Education Sta- tistics (NCES), which sponsored this project. We think that the leadership of the NCES was wise, both to pursue improvements in standard setting andâas a statistical agencyâto choose an impartial external body to estab- lish its reporting standards. The committee thanks Gary Phillips for his willingness to initiate the study and extends its heartfelt thanks to Peggy Carr for her interest in this important topic and her constant support throughout the project. During the course of this project other NCES staff members, including Sheida White and Andrew Kolstad, gave generously of their time. We thank each of them for the wealth of information they provided and their prompt answers to all of the committeeâs questions. The American Institutes for Research (AIR) served as contractor to NCES for work on National Assessment of Adult Literacy (NAAL), and many of its staff were generous with both advice and assistance. Mark Kutner was an invaluable resource to the committee, and we are grateful for his responsiveness to all of the committeeâs requests. We also thank AIR staff members Stephan Baldi, Elizabeth Greenburg, and Eugene Johnson for their ongoing assistance and readiness to answer the committeeâs questions. Special thanks are also due to Irwin Kirsch, who led the work on the 1992 National Adult Literacy Survey (NALS) at the Educational Testing Service. Irwin was a tremendous resource to the committee as they worked to reconstruct and understand procedures that had been used to determine ix
x ACKNOWLEDGMENTS the performance levels for the earlier literacy assessment. His fact-checking of report text that documented these processes was a great help to the committee. Early in its tenure, the committee commissioned a literature review of studies conducted on NALS. We thank M. Cecil Smith of Northern Illinois University for his thorough review of the literature, which provided an important part of the foundation for the committeeâs work. The committee held an information-gathering meeting to learn about international assessments of adult literacy, and we are grateful to presenters at this meeting, including Mariann Lemke of the U.S. Department of Edu- cation; Scott Murray then of Statistics Canada; and Irwin Kirsch of Educa- tional Testing Service. At the fourth meeting, the committee convened a public forum. The insights provided by the participants were very useful in helping the com- mittee determine the performance levels. For this, we are grateful to Cynthia Baur, U.S. Department of Health and Human Services; Beth Beuhlmann, U.S. Chamber of Commerce, Workforce Preparation; Richard Colvin, Hechinger Institute; Leslie Farr, Ohio State University; Milton Goldberg, Education Commission of the States; Anne Lewis, freelance journalist; Richard Long, International Reading Association; Christopher Mazzeo, National Governors Association; Gemma Santos, Miami Dade Public Schools; Tony Sarmiento, Senior Service America, Inc.; Linda Taylor, Com- prehensive Adult Student Assessment System; and Robert Wedgeworth, Proliteracy Worldwide. Representatives from five state departments of adult education also provided feedback about performance levels and about how NAAL results would be used in their states. We thank Bob Bickerton, Donna Cornelius, and Ann Serino, Massachusetts Department of Education; Steve Coffman, Missouri Department of Education; Cheryl King and Reecie Stagnolia, Ken- tucky Department of Education; Tom Orvino, New York Department of Education; and Linda Young, Oklahoma Department of Education. The committee is indebted to the individuals who assisted with the bookmark standard-setting sessions, held in July and September 2004. We particularly thank Richard J. Patz, of R.J. Patz, Inc., who led the standard- setting procedures. His expertise and guidance were key to the success of the standard settings. We are also grateful to Jeff Hauger and April Zenisky, University of Massachusetts, Amherst, and Andrew Poggio, University of Iowa, who assisted with managing the standard-setting sessions. Their as- sistance was key in making the sessions run smoothly. Special thanks are due to the many individuals who served as panelists for the bookmark standard-setting sessions. The committee truly appreci- ates their hard work and keen insights. The panelists included Eunice Askov,
ACKNOWLEDGMENTS xi Pennsylvania State University; Marjorie Ball, Mississippi State Penitentiary; Roxanne Bauer, Indianapolis Public Schools; Michelle Blantz, South Geor- gia Technical College; Rhodella Brown, Daytona Beach Community Col- lege, Florida; Miriam Burt, Center for Applied Linguistics; Laura Chenven, AFL-CIO Working for America Institute; Suzanne Cimochowski, EASTCONN; Marie Cora, Hotspur Partners, LLC; Christopher Coro, Northampton Community College, Pennsylvania; Susan Cowles, Oregon State Department of Community Colleges and Workforce Development; Shari Crockett, Regional Office of Education, Illinois; Lansing Davis, New Jersey State Employment and Training Commission; Kim Donehower, Uni- versity of North Dakota; Suzanne Elston, Bradley County Adult Education, Tennessee; Leslie Farr, Ohio State University; Sharon Floyd, Saginaw Pub- lic Schools; Janet Geary, North Kansas City School District; Karen Gianninoto, Salisbury State University; Kimberly Gibson, Sierra College; Suzanne Grant, Arlington Public Schools, Virginia; Anne Greenwell, Jefferson County Public Schools, Kentucky; Christina Gutierrez, T.C. Wil- liams High School, Virginia; Nancy Hampson, San Diego Community Col- lege District; James Harris, Caliber Associates; Roberta Hawkins, Shorewood High School, Washington; Fran Holthaus, Upper Valley Joint Vocational School, Ohio; Sally House, Central Mississippi Correctional Facility; Brenda Jeans, Beauregard Parish School Board, Louisiana; Paul Jurmo, New York University; Judy Kihslinger, Waukesha County Techni- cal College, Wisconsin; Terry Kinzel, Big Bend Community College, Wash- ington; Jaqueline Korengel, Commonwealth of Kentucky; Nathan Kuncel, University of Illinois at Urbana-Champaign; Diane Lindahl, Western Wis- consin Technical College; Ardith Loustalet, St. Vrain Valley School Dis- trict, Colorado; Alfredo Lujan, Monte del Sol Charter School, New Mexico; Sanford Marks, Community College of Southern Nevada; Peggy McGuire, University of Tennessee; Maureen Meehan, University of Illinois at Chi- cago; Doug Molitor, 3M; Donald Mott, Wilson Mott & Associates, North Carolina; Vivian Mott, East Carolina University; Bill Muth, U.S. Federal Bureau of Prisons; Connie Nelson, Massachusetts Worker Education Roundtable; Donna Nola-Ganey, Louisiana Department of Education; Peg Perri, Western Wisconsin Technical College; Rebecca Rogers, Washington University, St. Louis; Teresa Russell, independent consultant; Sally Sandy, Parkway School District, Missouri; Kathleen Santopietro Weddel, Colo- rado Department of Education; Diane Schroeder, St. Charles Community College, Missouri; Don Seaman, Texas Center for the Advancement of Literacy and Learning; Jane Siveria, Florida Department of Education; Cristine Smith, World Education, Inc.; Maggie Sokolik, University of Cali- fornia, Berkeley; Linda Stacy, Owens Community College, Ohio; Linda Taylor, Comprehensive Adult Student Assessment System; Ray Thompson,
xii ACKNOWLEDGMENTS Middle Georgia Technical College; Patricia Thorpe, University of Phoenix; Fran Tracy-Mumford, Delaware Department of Education; Karen Valbrun, Georgia State Department of Technical and Adult Education; Denise Weiner, Delaware Department of Education; Lynne Weintraub, Jones Li- brary; Ira Yankwitt, Literacy Assistance Center; and Linda Young, Okla- homa State Department of Education. Senior staff members of the National Research Council (NRC) helped the committee move this project forward. Michael Feuer enthusiastically backed the project and lent his wisdom and advice at key stages. Patricia Morison provided sage advice throughout this project and made valuable comments on several versions of the report. Eugenia Grohmanâs knowledge and experience with NRCâs procedures and the committee process were of great assistance. We thank Christine McShane for her expert editing assis- tance and Yvonne Wise for her work in moving this report through the publication process. The committee is indebted to Kirsten Sampson Snyder for ably guiding the report through the NRC review process. Special thanks are due to Michael DeCarmine for his masterful han- dling of the logistical aspects of this project. In addition to handling the responsibilities associated with organizing committee meetings, Michael very capably managed the logistics of holding the July standard-setting session with 45 participants. He was also of great assistance to the commit- tee by attending and reporting on his observations of the training sessions for NAAL survey administrators. We also thank Lisa Alston, who provided support throughout the project. We are grateful to Dorothy Majewski, who assumed responsibility for organizing the second standard-setting session. The committee also appreciates the assistance of Teresia Wilmore and Dionna Williams, who ably stepped in to assist at various stages of the project. Many other NRC staff contributed to the success of this project. We thank Connie Citro for her wealth of experience with standard setting in other contexts. Connie provided advice throughout the project. The com- mittee sincerely appreciates the analytical assistance provided by Lynne Steuerle Schofield. Lynneâs statistical expertise and careful attention to de- tail contributed greatly to the substance and quality of this report. Lori Houghton Wright played a major role in organizing and managing the standard settings and contributed greatly to their overall success. We also appreciate Loriâs assistance in helping to produce this report. We thank Andrew Tompkins for his work in observing and reporting on the training procedures for NAAL interviewers and his shrewd research assistance. It has been most rewarding for us to work with our coeditors, Judith Koenig and Stuart Elliott. They kept everyone on track, drew wisely on various areas of expertise to create an effective division of labor within the committee, initiated and coordinated contacts with all of the parties to the
ACKNOWLEDGMENTS xiii project, and shouldered the major share of report preparation. We cannot imagine more professional and enjoyable colleagues. Above all, we thank the committee members for their dedication and outstanding contributions to this study. They drafted text, prepared back- ground materials, reviewed numerous versions of this report, and gave generously of their time throughout the course of this three-year project. Both of us are novices in comparison to any of them in matters relating to adult education, adult literacy, and the measurement of literacy. Their varied expertiseâand their patience with usâwere essential to the report and to our growing appreciation of the importance and complexity of these matters. This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with proce- dures approved by the NRCâs Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evi- dence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the delibera- tive process. We wish to thank the following individuals for their review of this report: Terry C. Davis, Department of Medicine and Pediatrics, Louisiana State University Health Sciences Center-Shreveport; Reynaldo F. Macias, Department of Chicana and Chicano Studies and the CÃ©sar E. ChÃ¡vez Center for Interdisciplinary Instruction, University of California, Los Ange- les; Mark D. Reckase, Departments of Counseling, Educational Psychol- ogy, and Special Education, Michigan State University; Stephen Reder, Department of Applied Linguistics, Portland State University; Loretta A. Shepard, School of Education, University of Colorado at Boulder; Sondra G. Stein, Policy Oversight, Equipped for the Future (EFF) Work Readiness Credential, Washington, DC; Sandy Strunk, Community Education, Lancaster Lebanon Intermediate Unit 13, East Petersburg, PA; Andrew Sum, Center for Labor Market Studies, Northeastern University; Daniel Wagner, National Center on Adult Literacy/International Literacy Institute University of Pennsylvania; Lauress (Laurie) Wise, Presidentâs Office, Hu- man Resources Research Organization (HumRRO), Alexandria, VA. Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations nor did they see the final draft of the report before its release. The review of this report was overseen by P. David Pearson, Gradu- ate School of Education, University of California, Berkeley, and Stephen E. Fienberg, Department of Statistics, Carnegie Mellon University. Appointed
xiv ACKNOWLEDGMENTS by the NRC, they were responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Re- sponsibility for the final content of this report rests entirely with the authoring committee and the institution. Christopher F. Edley, Jr., and Robert M. Hauser, Co-Chairs Committee on Performance Levels for Adult Literacy
Contents EXECUTIVE SUMMARY 1 1 INTRODUCTION 15 Problem Statement, 15 Overarching Goals, 18 Limitations on Inferences About Literacy Skills, 19 Committeeâs Approach to the Charge, 21 Overview of the Report, 22 2 ADULT LITERACY ASSESSMENTS AND ADULT EDUCATION 23 Literacy Demands and the Need for Assessments, 23 Literacy Assessments, 25 Uses of Results of Adult Literacy Assessments, 31 3 DEVELOPING PERFORMANCE LEVELS FOR THE NATIONAL ADULT LITERACY SURVEY 50 Background on Developing Performance Levels, 50 Development of NALS Tasks, 52 Development of Performance-Level Descriptions and Cut Scores, 54 Choice of Response Probability Values, 69 Mapping Items to Performance Levels, 71 Conclusion, 72 Technical Note, 73 xv
xvi CONTENTS 4 DETERMINING PERFORMANCE LEVELS FOR THE NATIONAL ASSESSMENT OF ADULT LITERACY 87 Stakeholder Views, 88 Relationships Between Literacy Scores and Background Characteristics, 92 Developing Policy-Relevant Performance Levels, 105 5 DEVELOPING PERFORMANCE-LEVEL DESCRIPTIONS AND SETTING CUT SCORES 108 The Bookmark Standard-Setting Method, 111 Bookmark Standard Setting with 1992 Data, 117 Bookmark Standard Setting with 2003 Data, 121 Results from the Standard-Setting Sessions, 130 Contrasting Groups Standard-Setting Method, 146 Adjusting the Bookmark Cut Scores, 159 Difficulties with the Upper and Lower Ends of the Score Scale, 161 6 COMMUNICATING AND USING THE RESULTS OF LITERACY ASSESSMENTS 167 Communicating Results, 167 Communication Strategies, 175 Examples of Ways NAAL Results May Be Used, 177 7 RECOMMENDATIONS FOR FUTURE LITERACY ASSESSMENTS 182 Demand-Side Analysis of Critical Skills, 183 Improving the Assessment of Quantitative Skills, 188 Improving the Information Collected About Adult Non-English Speakers, 191 Rethinking and Broadening the Definition of Literacy, 193 Conclusion, 195 REFERENCES 196 APPENDIXES A The Committeeâs Public Forums on Performance Levels for NAAL 203 B Examination of the Dimensionality of NALS 214 C July 2004 Bookmark Standard-Setting Session with the 1992 NALS Data 221 D September 2004 Bookmark Standard-Setting Session with the 2003 NAAL Data 285 E Biographical Sketches of Committee Members and Staff 327