Designing a Market Basket for NAEP: Summary of a Workshop (2000)

A basic component of the NAEP market basket is the release of a representative set of test items. While NAEP has always released samples of items, under plans for the market basket, many more items would be released with the goal of representing the content domain for a specific subject. For example, the collection of items for fourth grade mathematics would consist of the appropriate number and mix of test items needed to represent the domain of fourth grade mathematics, as defined by NAEP's frameworks. The Committee on NAEP Reporting Practices was interested in the purposes that might be served by such a release of items and requested that workshop discussants consider who might use this information and how it might be used. The discussion below attempts to summarize the major points made by the speakers.

Workshop discussants remarked that an aura of mystery surrounds testing. In their interactions with the public, they have found that people question why so much time is devoted to testing and are unsure of how to interpret the results. The public is not fully aware of the material that is covered on achievement tests, the skills that students are expected to demonstrate, and the inferences about student achievement that can be drawn from test results. Moreover, the public does not always see the link between assessment programs and school reform efforts.

Marilyn McConachie, Illinois State Board of Education member, summarized these perceptions most succinctly, saying:

Analysis of public understanding of test results in Illinois parallels national commentary on NAEP and other large-scale testing. Put simply, the public believes too much time is spent on testing and doesn' t really understand what students know and are able to do or whether performance is good enough. These beliefs appear to erode support for state testing (and for NAEP).

According to McConachie, only when tests are “demystified” will the public understand what is being tested and why, and only then will the public support the continued gathering of this important information.

Other workshop discussants commented that public release of test items, scoring rubrics, and student work samples could serve to further public understanding of what NAEP tests. Many felt that the public is not generally aware of the difficulty level of the material covered on achievement tests today. For most people, the basis of comparison is their own school experiences, but curricula and expectations for students have changed. Joseph O'Reilly, director of assessment for Mesa, Arizona, public schools offered an example that illustrates this perception:

The public does not seem to understand the difficulty of the concepts taught today compared to when they were in school, especially in mathematics. For example, basic trigonometric angles are commonplace in algebra today but were not covered until much later in the curriculum, if at all, thirty years ago.

O'Reilly believes that the release of a large collection of items would be very useful in communicating the higher levels of expectations of tests like NAEP. Additionally, because many state testing programs cover content similar to that tested by NAEP, the information learned from NAEP's release of items could also increase understanding of state and local assessments.

Workshop speakers observed that release of a large number of representative items could be used to stimulate discussion among teachers regarding the format and content of test questions. In addition, review of the released items could facilitate discussions about ways to align local curricula and instructional practices with the material covered on the national assessment. Discussants explained that it is often difficult to draw conclusions about their students' NAEP performance because it is not clear

whether the material tested on NAEP is covered by their curricula or when it is covered.

The superintendents representing the First in the World Consortium provided examples of how they utilized information from their students ' participation in the Third International Mathematics and Science Study (TIMSS) to guide local curricular and instructional changes. The consortium participants did not have access to actual test items but, instead, had information on the specific topics covered and the content areas assessed. Teams of teachers from consortium schools examined the topics covered on the TIMSS assessment. They also considered when and how these topics were presented in their curricula, discussing such issues as: at what grade level the topic is first introduced in their programs; at what grade level mastery of the topic is expected; and how the topic is reinforced over the grades. Analysis of their TIMSS results, particularly of students' strengths and weaknesses, in comparison to teaching and instructional practices, allowed the participating school systems to identify needed changes in their curricula.

This use of TIMSS materials exemplifies one potential use of released NAEP materials. While the First in the World Consortium school systems did not see actual items, they had the benefit of receiving information on the topics and content areas covered by TIMSS items. For school systems and others to realize similar benefits from the released NAEP materials, items would need to be categorized into frameworks and content areas. Otherwise, teachers and other users might categorize items themselves, perhaps incorrectly, in order to make inferences about the relationships between material tested and content covered by their curricula.

Workshop discussants also suggested that it would be useful to associate the released items with the NAEP achievement level category of students expected to answer the question correctly. This matching of items with achievement levels would demonstrate the content and skills students should have mastered at each level, which would facilitate understanding of the assessment. For example, if teachers were able to view items that illustrated what students scoring at the proficient level in fourth-grade mathematics should be able to do, they would be able to adjust their teaching accordingly. Used diagnostically, this information could help students progress from below basic to basic, from basic to proficient, and from proficient to advanced.

NAEP often serves as a role model for the development of state and local assessments and the policy governing those assessments. During the committee's earlier workshop on reporting district-level NAEP results, participants commented that NAEP's frameworks, its innovative item design, and its use of achievement-level reporting have greatly influenced assessments around the country (National Research Council, 1999c.) Similar observations were made at the market-basket workshop.

Participants in the market-basket workshop thought that a large-scale release of NAEP items and related test materials could potentially improve state and local assessment programs. NAEP produces high-quality items and test materials. Allowing test developers to view large amounts of NAEP test materials (test questions as well as rubrics for scoring constructed response items) could therefore have a positive effect on the quality of item design for state and local assessments. The release of high-quality NAEP materials could also help revamp classroom-based assessments. Furthermore, in their opinion, policymakers would be able to see the breadth and depth of the content and skills assessed and the grade levels at which students are expected to have mastered certain subject matter—information that could play an important role in redefining curricula. Participants emphasized, however, that for such objectives to be realized, item release would need to be both large and representative of the domain assessed.