NATIONAL ACADEMY PRESS
2101 Constitution Avenue, NW Washington, DC 20418
NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.
This report has been reviewed by a group other than the authors according to procedures approved by a Report Review Committee consisting of members of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce M. Alberts and Dr. Robert M. White are chairman and vice chairman, respectively, of the National Research Council.
The Mathematical Sciences Education Board was established in 1985 to provide a continuing national capability to assess the status and quality of education in the mathematical sciences and is concerned with excellence in education for all students at all levels. The Board reports directly to the Governing Board of the National Research Council.
Library of Congress Catalog Card Number 93-85917
International Standard Book Number 0-309-04981-4
Copyright 1993 by the National Academy of Sciences. All rights reserved.
Permission for limited reproduction of portions of this book for educational purposes but not for sale may be granted upon receipt of a written request to the National Academy Press, 2101 Constitution Avenue, NW, Washington, DC 20418.
Copies of the report may be purchased from the
National Academy Press,
2101 Constitution Avenue, NW, Box 285, Washington, DC 20055. Call 800-624-6242 or 202-334-3313 (in the Washington Metropolitan Area).
Printed in the United States of America
Calls for standards in education have been echoing across the nation for several years, especially since political leaders of both parties decided to adopt bipartisan national goals for education. Standards without appropriate means of measuring progress, however, amount to little more than empty rhetoric. To stay the course and achieve the national goals for education, we must measure the things that really count.
Standards take many forms and appear under many guises. Curriculum (or content) standards tell what students should learn. Teaching (or pedagogical) standards tell how students learn and how teachers should teach. Delivery (or opportunity-to-learn) standards tell what is necessary of schools so that students can learn and teachers can teach. Assessment (or performance) standards tell what students should know and be able to do as well as how evaluators can judge levels of performance.
Since 1989 mathematics has led the national movement towards standards with Everybody Counts (National Research Council, 1989), Curriculum and Evaluation Standards for School Mathematics (National Council of Teachers of Mathematics (NCTM), 1989), and Professional Standards for Teaching Mathematics (NCTM, 1991). In April 1991, the Mathematical Sciences Education Board (MSEB) convened a national summit on assessment, which led to For Good Measure (1991), a concise statement of goals and objectives for mathematics assessment. To move the national discussion from generalities to specifics, the MSEB then published Measuring Up (1993), which provided prototype assessment tasks for fourth grade mathematics that illustrate in concrete terms the goals of the NCTM Standards.
Measuring What Counts further advances this national discussion by establishing crucial research-based connections between standards and assessment. It demonstrates the importance of three key principles—on content, learning, and equity—for any program of assessment that is intended to support the national educational goals. The message of Measuring What Counts is quite simple, but its implications are profound: Assessment in support of standards must not only measure results, but must also contribute to the educational process itself.
The analyses and recommendations in this report were developed by a study group formed by the NRC in 1991 to develop conceptual guidelines that would make content the driving force in the reform of mathematics assessment and that would explore a variety of related measurement and policy issues. The intent of Measuring What Counts is not to offer immediate practical advice, but to lay out a conceptual framework that will help those who are struggling with the urgent need to develop new assessment that align properly with the national standards for mathematics education.
The three principles on content, learning, and equity articulated in Measuring What Counts are necessary but not sufficient criteria for effective assessment. They set forth fundamental conditions that form a foundation on which to build new approaches to traditional technical testing issues such as reliability and validity. Assessment should foremost reflect important mathematics, support good instructional practice, and enhance every student's opportunity to learn.
Although these principles are rooted in both informed practice and extensive research, it is fair to say that there remain many open questions. Research shows clearly that the task of assessing mathematical learning is far more subtle than previously believed; experience reveals enormous gaps between current assessment practice and new goals for mathematics education. It is clear from the recent history of failed reform that when assessment is separated from curriculum and instruction, teaching becomes distorted, thus diminishing learning.
Experts agree that for education to be effective, curriculum, instruction, and assessment must harmonize for their mutual support. Both internal (teacher-based) and external (district- or state-based) assessment must support improved learning. However, the
path from general agreement to specific assessments is far from clear. We are embarking on a new venture, guided by the principles of content, learning, and equity. Exploration of this new world of alternative assessment will take years of work from thousands of practitioners working with mathematics education specialists and measurement experts to achieve a more effective balance of assessment in practice.
Although many of the issues raised in this report apply to all disciplines, it is mathematics education that provides the primary motivation for the study, the background of the authors, the source of examples, the domain of research, and the field of practice on which the conclusions and recommendations are based. Measuring What Counts seeks to address issues in assessment that are important to the discipline of mathematics and about which the expertise of mathematics educators can make a singular contribution. Content, learning, and equity emerged as fundamental principles for assessment because they are fundamental concerns of mathematics education.
One consequence of Measuring What Counts should be a new wave of research on assessment, on learning, and on instruction. Since much that is in this report is based on expert conjecture rather than firm evidence, it opens scores of potential areas for further research. Indeed, the changing practice of mathematics itself—the increased focus on computer-enhanced work, on group problem solving, on modeling complex problems—challenges researchers in assessment and learning with issues rarely before considered. The resulting iteration of practice and research will provide an effective guidance system to keep assessment reform aligned with curricular objectives and principles of learning.
All reform is evolutionary. As society changes, the targets and goals for education change. Assessment is our primary tool for monitoring progress and making midcourse corrections. The principles of assessment set forth in Measuring What Counts provide a solid conceptual basis for current efforts to improve assessment and lay the groundwork for more detailed assessment standards to be published by the NCTM.
When the stakes for improved education are so high, when our children's futures are at stake, we must ensure that assessment supports standards-based education by adhering to fundamental principles of content, learning, and equity.
Measuring What Counts was prepared by the National Research Council's Study Group on Mathematics Assessment, which was chaired by Jeremy Kilpatrick, Regents Professor of Mathematics Education at the University of Georgia. The group met over a period of two years to develop drafts of this report. Members of the study group, in addition to the chair, were Janice Arceneaux, Magnet Specialist in the Houston Independent School District; Lloyd Bond, Professor of Educational Research Methodology at the University of North Carolina-Greensboro; Felix Browder, Professor of Mathematics at Rutgers University; Philip C. Curtis, Jr., Professor Mathematics at the University of California at Los Angeles; Jane D. Gawronski, Superintendent of the Escondido Union High School District; Robert L. Linn, Professor of Education at the University of Colorado-Boulder; Sue Ann McGraw, Mathematics Teacher at Lake Oswego High School; Robert J. Mislevy, Principal Research Scientist at Educational Testing Service; Alice Morgan-Brown, Statewide Director for Academic Champions of Excellence Program at Morgan State University; Andrew Porter, Director, Wisconsin Center for Education Research at the University of Wisconsin-Madison; Marilyn Rindfuss, National Mathematics Consultant at The Psychological Corporation; Edward Roeber, Director, Student Assessment Programs at the Council of Chief State School Officers; Maria Santos, Mathematics and Science Supervisor in the San Francisco Unified School District; Cathy Seeley, Director of Pre-college Programs, Charles A. Dana Center for Mathematics and Science Education at the University of Texas-Austin; and Edward A. Silver, Professor and Senior Scientist, University of Pittsburgh.
Members of the study group faced many significant hurdles posed by their differing professional perspectives, by the rapidly changing context of educational assessment, and by the challenges posed by the new Standards for school mathematics. We owe each of them a special thanks for persisting in this formidable task to reach consensus on the key principles enunciated in this report. Particular thanks are due Jeremy Kilpatrick not only for his able leadership as chairman of the Study Group, but also for the substantial contribution he made in writing and editing the various parts of the report.
The work of the study group was enriched by three commissioned papers that appear at the end of this volume. These
papers provided useful background for deliberations of the study group and constituted valuable additions to the research literature on assessment. We are particularly grateful to their authors—Lynn Hancock and Jeremy Kilpatrick, Stephen B. Dunbar and Elizabeth A. Witt, and Diana C. Pullin—for their contributions to this endeavor. We wish also to thank Linda Dager Wilson and Lynn Hancock for supplying other background information of value to this study.
Like all reports of the National Research Council, Measuring What Counts has been extensively reviewed—first by outside experts in early draft form, then by the MSEB Committee on Policy Studies at several key stages, and, at the final stage, under the careful protocol of the NRC's Report Review Committee. We thank these many reviewers for their insightful and knowledgeable comments. Special thanks are due Nancy Cole, Chair of the MSEB Committee on Policy Studies, for providing consistent and wise counsel as the report worked its way through various drafts. Measuring What Counts is much stronger as a result of the input and advice of these outside reviewers.
Financial support for work of the study group and preparation of Measuring What Counts was provided by the U.S. Department of Education and the National Science Foundation. We gratefully acknowledge the support of these organizations.
Staff work in support of the study group was provided by the MSEB Office of Policy Studies directed by Linda Peller Rosen, Associate Executive Director of the MSEB. Edward T. Esty and Patricia A. Butler deserve special thanks for managing this complex project with unfailing perseverance and tireless energy. Thanks are due also to Anuradha Sapru Kohls, Ramona Robertson, and Altoria Bell, who undertook diverse tasks without which this report would never have been completed. The staff of the National Academy Press deserves special mention for their efforts on our behalf.
Hyman Bass, Chairman
Mathematical Sciences Education Board