NATIONAL ACADEMY PRESS
2101 Constitution Avenue, N.W. Washington, DC 20418
NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.
The study was supported by The Pew Charitable Trusts (award 96000217-000), The Spencer Foundation (award 199700156), The William T. Grant Foundation (award 97179797), and the U.S. Department of Education (award R305U960001). Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the view of the organizations or agencies that provided support for this project.
International Standard Book Number 0-309-06534-8
Suggested citation: National Research Council (1999). Testing, Teaching, and Learning: A Guide for States and School Districts. Committee on Title I Testing and Assessment, Richard F. Elmore and Robert Rothman, editors. Board on Testing and Assessment, Commission on Behavioral and Social Sciences and Education. Washington, DC: National Academy Press.
Additional copies of this report are available from
National Academy Press ,
2101 Constitution Avenue, N.W., Washington, DC 20418. Call (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area). This report is also available online at http://www.nap.edu
Printed in the United States of America
Copyright 1999 by the National Academy of Sciences. All rights reserved.
COMMITTEE ON TITLE I TESTING AND ASSESSMENT
RICHARD F. ELMORE (Chair),
Graduate School of Education, Harvard University
EVA L. BAKER,
Center for Research on Evaluation, Standards and Student Testing, University of California, Los Angeles
RUBEN A. CARRIEDO,
San Diego City Schools
URSULA CASANOVA,
College of Education, Arizona State University
ROBERTA J. FLEXER,
Department of Education, University of Colorado at Boulder
ELLEN C. GUINEY,
Boston Plan for Excellence in Public Schools
KATI P. HAYCOCK,
The Education Trust, Washington, D.C.
JOSEPH F. JOHNSON, JR.,
Collaborative for School Improvement, Charles A. Dana Center, University of Texas at Austin
SHARON LYNN KAGAN,
Child Study Center, Yale University
FAYNEESE MILLER,
Center for the Study of Race and Ethnicity in America, Brown University
JESSIE MONTANO,
Minnesota Department of Children, Families and Learning, St. Paul
P. DAVID PEARSON,
College of Education, Michigan State University, East Lansing
STEPHEN W. RAUDENBUSH,
School of Education, University of Michigan, Ann Arbor
LAUREN B. RESNICK,
Learning Research and Development Center, University of Pittsburgh
WARREN SIMMONS,
Annenberg Institute for School Reform, Brown University
CHARLENE G. TUCKER,
Connecticut Department of Education, Hartford
ROBERT ROTHMAN, Study Director
DOROTHY MAJEWSKI, Project Assistant
BOARD ON TESTING AND ASSESSMENT
ROBERT L. LINN (Chair),
School of Education, University of Colorado, Boulder
CARL F. KAESTLE (Vice Chair),
Department of Education, Brown University
RICHARD C. ATKINSON, President,
University of California
PAUL J. BLACK,
School of Education, King's College, London, England
RICHARD P. DURÁN,
Graduate School of Education, University of California, Santa Barbara
CHRISTOPHER F. EDLEY, JR.,
Harvard Law School, Harvard University
RONALD FERGUSON,
John F. Kennedy School of Public Policy, Harvard University
ROBERT M. HAUSER,
Institute for Research on Poverty, University of Wisconsin
PAUL W. HOLLAND,
Graduate School of Education, University of California, Berkeley
RICHARD M. JAEGER,
Center for Educational Research and Evaluation, University of North Carolina
BARBARA M. MEANS,
SRI International, Menlo Park, California
LORRAINE McDONNEL,
Department of Political Sciences, University of California, Santa Barbara
KENNETH PEARLMAN,
Lucent Technologies, Inc., Warren, New Jersey
ANDREW C. PORTER,
Wisconsin Center for Education Research, University of Wisconsin, Madison
CATHERINE E. SNOW,
Graduate School of Education, Harvard University
WILLIAM L. TAYLOR, Attorney at Law,
Washington, D.C.
WILLIAM T. TRENT, Associate Chancellor,
University of Illinois, Champaign
VICKI VANDAVEER,
The Vandaveer Group, Inc., Houston, Texas
LAURESS L. WISE,
Human Resources Research Organization, Alexandria, Virginia
KENNETH I. WOLPIN,
Department of Economics, University of Pennsylvania
MICHAEL J. FEUER, Director
VIOLA C. HOREK, Administrative Associate
LISA ALSTON, Administrative Assistant
Preface
Standards, assessment, and accountability have become a common concern of public policy toward education at the federal, state, and local levels. The reasons for this concern are deeply embedded in the politics and economics of the education sector over the past two decades—rising public expenditures, increasing centralization and equalization of education funding, and increasing concern among policy makers at all levels of government for the health and competitiveness of the American economy. Whatever one's position on the specifics, there is no avoiding the imperative for clearer definitions of the outcomes of schooling and clearer accounting for results.
In the midst of this debate, in 1994, the Congress reauthorized Title I, the largest single federal program for elementary and secondary education in the United States. The congressional debate around reauthorization of Title I was, in many ways, a reflection of the larger public debate that had been occurring around that time in thousands of local school boards, dozens of state legislatures, and many national commissions. In particular, the debate focused on the terms and conditions under which state agencies, local school districts, and schools would be accountable for the academic learning of disadvantaged students, who were the intended beneficiaries of Title I's supplemental funding. The 1994 amendments substantially shifted the focus of Title I, away from treating Title I recipients as a separate class of beneficiaries with their own particular needs and toward an emphasis on bringing educationally disadvantaged students into the academic mainstream, judging their academic success in the same terms as those of all other students. The 1994 amendments also brought Title I into alignment with the growing movement toward standards-based reform at the state and local levels, which focuses on setting high and clear goals for student academic learning and judging schools on the basis of their contributions to students' progress toward those goals.
In spring 1995, just before the reauthorization of Title I was set to take
effect, the Board on Testing and Assessment of the National Research Council convened a workshop on the implications of Title I's new testing and assessment requirements for states and localities. This workshop, involving participants from federal, state, and local education agencies, as well as representatives of the research and testing community, surfaced a number of difficult technical and practical issues related to the implementation of the new requirements. As an outgrowth of this discussion, and with support from the U.S. Department of Education, the Pew Charitable Trusts, the Spencer Foundation, and the W.T. Grant Foundation, the National Research Council formed the Committee on Title I Testing and Assessment to look into these issues in greater depth. The committee began its work in November 1997.
The Committee on Title I Testing and Assessment was chartered for an explicitly practical purpose: to provide policy guidance to states and localities in using testing and assessment to improve the academic learning of students who are the intended beneficiaries of Title I. The committee was charged to assess research bearing on the use of testing and assessment for accountability purposes, to examine the experience of states and localities in this domain, and to develop a “decision framework that incorporates technical quality, effects on teaching and learning, costs and benefits, fairness and other criteria for evaluating assessment strategies.” Hence, the committee's primary concern has been to provide practical guidance to states and localities in the design and implementation of standards-based assessments and accountability mechanisms, consistent with both state and local policy and with the requirements of Title I.
Reflecting its orientation toward practical guidance, the committee's membership represents a cross-section of expertise on testing and assessment issues, from state and local practitioners to academic researchers, and the full range of practical and conceptual concerns related to Title I assessment.
As the committee's work progressed, we came to a common understanding of the daunting task confronting states and localities in their attempts to create new forms of standards-based improvement and accountability in Title I. We agreed, for example, to focus on broad policy guidance to states and localities, organized around specific problems that any standards, assessment, and accountability system would have to solve, allowing for substantial variation and creativity in crafting specific solutions appropriate to specific state and local contexts. So this report focuses on “mid-range” advice, specific enough to provide useful guidance for policy makers and practitioners, broad enough to accommodate a wide range of solutions adapted to specific contexts. We also broadened the initial charge slightly to include discussion of issues of instruction and professional development for teachers and administrators in addition to issues of assessment and accountability. It became clear to us, as we explored the practical implications of Title I assessment and accountability, that the construction of assessment and accountability systems cannot be isolated from their purposes, which are to improve the quality of instruction and ultimately the learning of
students. So we were inevitably drawn into the relationship between assessment and accountability issues and issues of large-scale improvement in teaching and learning.
In the five years or so since the reauthorization of Title I, progress on the assessment and accountability requirements of the law have been highly uneven. The 1994 law envisioned that by the year 2000 all states would have put in place content and performance standards, aligned with assessments of student performance, and coupled with systems for holding schools accountable for student learning. As the year 2000 and the next reauthorization of Title I approach, it is now clear that many states and localities are still struggling to meet the basic requirements of the law; some states and localities are meeting the requirements but having difficulties connecting assessments to a broad-scale strategy of instructional improvement; and some states have met the requirements of the law but discovered a new generation of problems related to the maintenance and improvement of their assessment and accountability systems.
The ambitious goals of the 1994 law are, in other words, still a work in progress in the field. This report is designed, to the extent possible, to speak to the entire range of states and districts, from the least to the most advanced. We also speak from the perspective that the struggle for increased focus and accountability in public education is a long-term project that will extend well beyond the present debate. We think our advice will be durable over the longer term, as public debate continues.
Because the implementation of Title I assessment is still a work in progress, the research available to the committee was limited. We have drawn on a broad body of research on testing and assessment issues generally, as well as the reports of previous NRC committees on specific questions of test development and utilization. But the practical nature of our charge and the limits of the evidence available to us have meant that we have also had to draw on the practical experience of committee members and outside experts in crafting our advice. Hence, this report relies heavily on expert advice from the field, in addition to scientific research.
Our hope is that state and local practitioners and policy makers will use this report as a guide to their continuing decisions in the development and improvement of new systems of assessment and accountability in Title I. It is not a simple template that prescribes a single approach or a single set of solutions. It is a framework, designed to lay out the major problems involved in the design of assessment and accountability systems, the knowledge that research and experience bring to bear on these problems, and the range of possible solutions to the problems. The framework also assumes that the purpose of assessment and accountability systems is to improve the quality of instruction in schools and school systems, rather than simply to measure and report school effectiveness.
RICHARD F. ELMORE, CHAIR
COMMITTEE ON TITLE I TESTING AND ASSESSMENT
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce M. Alberts is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. William A. Wulf is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Kenneth I. Shine is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy's purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce M. Alberts and Dr. William A. Wulf are chairman and vice chairman, respectively, of the National Research Council.
Acknowledgments
For all the reasons stated in the preface, this report could not have happened without support from a number of people, and the committee is grateful for their contributions. We want first of all to acknowledge our sponsors, who made the project possible and kept it going. At the U.S. Department of Education, Valena Plisko, Margaret McNeely, and Collette Roney showed a continuing interest in the project and kept us apprised of events and publications that would assist us in our work.
At the Pew Charitable Trusts, Robert B. Schwartz and C. Kent McGuire were instrumental in helping get the project off the ground. Their successor as education program officer, Edward F. Reidy, Jr., not only continued to support the project but also, during one meeting, donned his old Kentucky associate commissioner hat and helped the committee think through some of the nettlesome design issues involved in assessment and accountability at the state level. Sadly, Ed passed away shortly before this book went to press. We will miss his wisdom and his commitment to education reform.
At the Spencer Foundation, Mark Rigdon was an enthusiastic supporter of the work. At the William T. Grant Foundation, the former president, Beatrix Hamburg, helped nurture the project, and her successor, Karen Hein, maintained the support.
The committee was also aided greatly by individuals who participated in our meetings and helped us understand the complex issues involved in designing and implementing standards-based systems. Mary Jean LeTendre, director of compensatory education programs at the U.S. Department of Education, and Edward D. Roeber, then the director of the state education assessment center at the Council of Chief State School Officers, provided us with an overview of the state of play in Title I at the federal and state levels, respectively.
At our second meeting, a panel of educators from the school, district, and
state levels described for us how tests were used at their sites. These were: Peter Behuniak of the Connecticut State Department of Education, Susanne Murphy of the Norwich (CT) Public Schools, Gloria Woods and Mary Russo of the Boston Public Schools, Mitchell Chester of the School District of Philadelphia, and Brenda Steele of Community District 2 in New York City. Three testing programs also lent us materials to review: Harcourt Brace Educational Measurement, New Standards, and the Connecticut State Department of Education.
At our third meeting, the committee heard from a panel of researchers and practitioners on the design issues involved in establishing assessments and accountability mechanisms. These were: Joan L. Herman of the National Center for Research on Evaluation, Standards, and Student Testing at the University of California, Los Angeles; James P. Spillane of Northwestern University; Edward Chittenden of the Educational Testing Service, Edward Reidy of the Pew Charitable Trusts (and formerly of Kentucky), and Lynn Winters of the Long Beach (CA) Unified School District.
The committee also commissioned several papers to address some critical areas in the research literature. Karen K. Wixson of the University of Michigan conducted an analysis of the alignment between standards and assessment in elementary reading in four states. J. Douglas Willms of the University of New Brunswick provided a helpful review of data analysis and reporting issues. M. Elizabeth Graue of the University of Wisconsin-Madison conducted an extensive literature review of assessment issues, focusing on early childhood assessments. Mark D. Reckase of Michigan State University reviewed the measurement issues associated with the Title I statute.
The Board on Testing and Assessment, the division within the National Research Council that launched the study, also provided considerable support to the committee as it conducted its work. William Taylor, a member of the board, attended nearly all the committee's meetings and lent us his substantial knowledge about Title I and the implementation of standards-based reform. Robert L. Linn, the board's chair, and Carl Kaestle, the vice chair, were very helpful and supportive.
Within the National Research Council, a number of individuals supported the project and helped us keep it moving forward. Barbara Boyle Torrey, the executive director of the Commission on Behavioral and Social Sciences and Education, and Alexandra Wigdor, the director of the Division on Education, Labor, and Human Performance, enthusiastically backed the project and lent wisdom and advice at key stages. Michael J. Feuer, the director of the Board on Testing and Assessment, was the guiding force behind the project and provided substantive advice and moral support all the way through. Patricia Morison helped guide us through the end game of report completion, review, and publication. Viola Horek helped us manage the complex finances of the project and provided support in innumerable ways. Christine McShane's skillful editing kept our tenses straight and our metaphors from mixing.
Dorothy Majewski, the senior project assistant, handled the logistics of our work with incredible dexterity and even more incredible good humor. Her ability to plan and manage complex arrangements and respond to last-minute requests and changes made our work much easier and much more enjoyable.
This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the NRC's Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making the published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process.
We wish to thank the following individuals for their participation in the review of this report: Stephen B. Dunbar, Iowa Testing Programs, University of Iowa; William Firestone, Graduate School of Education, Rutgers University; John F. Jennings, Center on Education Policy, Washington, D.C.; Margaret J. McLaughlin, Institute for the Study of Exceptional Children and Youth, University of Maryland; Daniel J. Reschly, Peabody College, Vanderbilt University; Alan Sheinker, Wyoming Department of Education, Cheyenne; and Richard Wagner, Department of Psychology, Florida State University. Although these individuals provided many constructive comments and suggestions, responsibility for the final content of this report rests solely with the authoring committee and the institution.
RICHARD F. ELMORE, CHAIR
ROBERT ROTHMAN, STUDY DIRECTOR
COMMITTEE ON TITLE I TESTING AND ASSESSMENT
This page in the original is blank. |
This page in the original is blank. |
This page in the original is blank. |