The work of the Committee on the Evaluation of National and State Assessments of Educational Progress benefited tremendously from the contributions and good will of many people.

Staff from the National Assessment Governing Board (NAGB), under the leadership of Roy Truby, executive director, and their subcontractor, American College Testing, Inc. (ACT), and staff from the National Center for Education Statistics (NCES), under the leadership of Pascal Forgione, commissioner of education statistics, and their subcontractors, Educational Testing Service (ETS) and Westat, Inc. were a valuable source of information and data on the National Assessment of Educational Progress (NAEP) throughout the project. Susan Loomis of ACT, Nancy Caldwell of Westat, and James Carlson, Stephen Lazer, John Mazzeo, and Christine O'Sullivan of ETS provided the committee with important information on occasions that are too numerous to mention. The committee especially extends thanks to Peggy Carr and Patricia Dabbs of NCES and Mary Lyn Bourque and Raymond Fields of the NAGB staff. In their roles as NCES and NAGB liaisons to the committee, they provided important information and perspectives throughout the course of its work.

Committee members and project staff benefited tremendously by attending and learning from discussions at meetings of the National Assessment Governing Board and its committees, the Technical Advisory Committee on Standard-Setting, the NAEP Design and Analysis Committee, the NAEP Subject Area Standing Committees, and the Advisory Council on Education Statistics. We thank all

of the committee members and staff for opening their meetings to us and for sharing their knowledge and perspectives.

The Office of Planning and Evaluation Services, U.S. Department of Education, administered the contract for this evaluation. Director Allen Ginsburg provided assistance in planning the evaluation, and Elois Scott, Collette Roney, and Audrey Pendleton each served as the contracting office's technical representative during various phases of the evaluation. The committee thanks them for their advice and assistance as they monitored its work.

Between March 1996 and April 1998, the committee met nine times. At its December 1996 meeting, the committee held a workshop on standard-setting models and their applications to the NAEP achievement-level-setting process. Each of the following individuals made a helpful and insightful presentation at the workshop and prepared a written paper, the collection of which was published in a recent issue of Applied Measurement in Education (volume 11, number 1, 1998): Jeanne Goldberg (Tufts University), Lawrence Hanser (RAND), Sheila Jasanoff (Cornell University), Robert Linn (University of Colorado), Robert Mislevy (ETS), Barbara Plake (University of Nebraska), and Mark Reckase (ACT, Inc., now of Michigan State University).

At its May 1997 meeting, the committee held a workshop to examine NAEP's mission, measurement objectives, and possible reconceptualization. The following individuals made presentations and provided valuable insights to inform the committee's deliberations on these complex issues: George Bohrnstedt (American Institutes for Research), Robert Boruch (University of Pennsylvania), Christopher Cross (Council for Basic Education), John Dossey (Illinois State University), Emerson Elliott (National Council for Accreditation of Teacher Education), Raymond Fields (NAGB), Robert Glaser (University of Pittsburgh), Herbert Ginsburg (Columbia University), James Greeno (Stanford University), Eugene Johnson (ETS), Daniel Koretz (RAND, now of Boston College), Alan Lesgold (University of Pittsburgh), James McBride (Human Resources Research Organization, Inc.), Robert Meyer (University of Chicago), Lawrence Mikulecky (Indiana University), Robert Mislevy (ETS), William Morrill (Mathtech, Inc.), Lois Peak (U.S. Department of Education), Andrew Porter (University of Wisconsin), Lauren Resnick (University of Pittsburgh), Lawrence Rudner (Educational Resources Information Center, University of Maryland), Richard Snow (Stanford University), David Thissen (University of North Carolina), Margaret Vickers (TERC), and Wendy Yen (CTB/McGraw Hill).

Early in the committee's work, Stephen Sireci (University of Massachusetts) wrote a paper for the committee synthesizing issues regarding the dimensionality of the NAEP assessments, and NRC consultant Joanne Capner provided analyses of NAEP's current assessment development and reporting strategies. The committee also commissioned new research and syntheses on several key topics to assist them in their evaluation. The following individuals contributed their time, energy, enthusiasm, and intellect to these efforts, and their work is published in

the volume of research papers that accompanies this report: Sheila Barron (RAND); Robert Boruch (University of Pennsylvania) and George Terhanian (Harris Black International); Patricia Kenney (University of Pittsburgh); Michael Kolen (University of Iowa); James Minstrell (Assessment, Curriculum, and Teaching Systems for Education); Stephen Sireci, Kevin Meara, Frederic Robin, and Hariharan Swaminathan (University of Massachusetts) and H. Jane Rogers (Columbia University); James Stigler (UCLA) and Michelle Perry (University of Illinois); and Jennifer Zieleskiewicz (Illinois Institute of Technology).

In November 1997, the committee convened a group of experts to discuss and explore the applications of contemporary cognitive and curricular research and theory and instructional practice to assessment development in NAEP. Consultants who shared their perspectives at this meeting included: David Pearson (Michigan State University), James Minstrell, Paul Nichols (University of Wisconsin), Leona Schauble (University of Wisconsin), Alan Schoenfeld (University of California, Berkeley), Patricia Kenney (University of Pittsburgh), Brenda Sugrue (University of Iowa), and Karen Wixson (University of Michigan). The committee's consideration of this topic was greatly enriched by the stimulating intellectual exchange at this meeting and in subsequent interactions with the attendees.

The Board on Testing and Assessment (BOTA) provided especially valuable guidance and feedback at critical stages of the committee's deliberations. The chair of BOTA, Robert Linn, assisted the committee by participating in discussions at several committee meetings and by reviewing and commenting on a draft of the final report. BOTA's intellectual contributions to the committee's work are much appreciated.

Many individuals at the National Research Council (NRC) provided guidance and assistance at many stages of the evaluation and during the preparation of the report. Executive director of the Commission on Behavioral and Social Sciences and Education (CBASSE) Barbara Torrey provided overall administration of the evaluation and shared her contagious enthusiasm for the work of the NRC with the committee. Alexandra Wigdor, director of the Division of Education, Labor, and Human Performance, provided continuing guidance and was especially central to the preparation of the committee's interim letter report on NAGB's proposed redesign of NAEP. Michael Feuer, director of the Board on Testing and Assessment, provided oversight of the committee's work and made frequent important contributions to committee discussions, providing unique and insightful perspectives and helping the committee maintain focus and achieve consensus. We also thank Eugenia Grohman, associate director for reports (CBASSE), for her advice on structuring the content of the report and for guiding the report through the NRC review process, and Christine McShane for her expert editing of the report manuscript and advice on the exposition of the report's main messages.

The committee especially expresses gratitude to the NRC project staff for

their intellectual and organizational skills throughout this evaluation. Jacques Normand and Susan McCutchen served as the study director and project assistant during the early phases of the committee's work. They were succeeded by senior program officer Karen Mitchell, study director Lee Jones, and project assistant Holly Wells. Karen and Lee tirelessly assisted the committee in many ways—serving as valuable sources of information about NAEP, organizing and synthesizing the committee's work, keeping the committee moving forward through its deliberations and the report drafting process, and providing energy, enthusiasm, and exceptional good humor along the way. Holly Wells capably and admirably managed the operational aspects of the evaluation—arranging meeting and workshop logistics, producing multiple iterations of drafts of committee writings and report text, and being available at all times to assist with committee requests, however large or small. The committee is deeply indebted to Holly for her commitment to the committee's work, her dedication to meeting the committee's many needs for information and service, her problem solving skills, and her affability in all circumstances.

This report has been reviewed by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the NRC's Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the authors and the NRC in making the published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The content of the review comments and draft manuscript remain confidential to protect the integrity of the deliberative process.

We wish to thank the following individuals, who are neither officials nor employees of the NRC, for their participation in the review of this report: Lizanne DeStefano, School of Education, University of Illinois, Champaign-Urbana; Emerson J. Elliott, National Council for the Accreditation for Teacher Education, Washington, D.C.; Susan Fuhrman, Graduate School of Education, University of Pennsylvania; Eric Hanushek, Wallis Institute of Political Economy, University of Rochester; Lyle V. Jones, L.L. Thurstone Psychometric Laboratory, University of North Carolina, Chapel Hill; Carl K. Kaestle, Department of Education, Brown University; P. David Pearson, Department of Education, Michigan State University; Gloria M. Rogers, Office of Institutional Research and Assessment, Rose-Hulman Institute of Technology; Bruce D. Spencer, Department of Statistics, Northwestern University; David M. Thissen, Department of Psychology, University of North Carolina, Chapel Hill; and Linda F. Wightman, Educational Research Methodology, University of North Carolina, Greensboro. Although the individuals listed above have provided many constructive comments and suggestions, responsibility for the final content of this report rests solely with the authoring committee and the NRC.

Finally, as chair, I would like to sincerely thank all of my fellow committee members, who generously contributed their time and intellect to this evaluation.

Our work covered an exceedingly broad array of complex topics and issues, and committee members exhibited a remarkable commitment to learning from each other's expertise, examining NAEP from new and varied perspectives, continuing the dialogue on some very tough issues, and producing a final report that clearly reflected a consensus among all members. All of this occurred in an atmosphere of substantial and ongoing collegiality and cordiality. It has been a professionally stimulating and personally gratifying experience to work with the members of this committee and the NRC project staff. It is my hope that their high standards and expectations have been fulfilled on this evaluation project and by our final report.

James W. Pellegrino, Chair

Committee on the Evaluation of National and State Assessments of Educational Progress

The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Bruce M. Alberts is president of the National Academy of Sciences.

The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. William A. Wulf is president of the National Academy of Engineering.

The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Kenneth I. Shine is president of the Institute of Medicine.

The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy's purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Bruce M. Alberts and Dr. William A. Wulf are chairman and vice chairman, respectively, of the National Research Council.

GRADING THENATION'SREPORTCARD