Grading the Nation's Report Card Evaluating NAEP and Transforming the Assessment of Educational Progress (1999) / Chapter Skim
Currently Skimming:
4 Frameworks and the Assessment Development Process: Providing More Informative Portrayals of Student Performance
4 Frameworks and the Assessment Development Process: Providing More Informative Portrayals of Student Performance
Pages 114-161
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 114... ...
However, the frameworks and assessment materials do not capitalize on contemporary research, theory, and practice in ways that would support in-depth interpretations of student knowledge and understanding. Large-scale survey instruments alone cannot reflect the scope of current frameworks or of more comprehensive goals for schooling.
|
From page 115... ...
Our evaluation of NAEP's frameworks and assessment development process follows; in this discussion we make arguments for: 1. determining the kinds of inferences and conclusions about student performances that are desired in reports of NAEP results, and then using this vision of student achievement to guide the entire assessment development process, 2.
|
From page 116... ...
When combined with the broader-scale interpretive information that emerges from the coordinated system of indicators described in Chapter 1, qualitative and quantitative summaries of student achievement can help educators and policy makers begin to answer the key question that is asked when achievement results are released: "What should we do in response to these results? " OVERVIEW OF NAEP'S CURRENT ASSESSMENT DEVELOPMENT PROCESS When this committee began its evaluation in spring 1996, the 1996 main NAEP science assessment was the focus, largely because the science achievement-level-setting process was undertaken concurrently with the term of this evaluation and because the science assessment included an unprecedented number and variety of constructed-response items and hands-on tasks.
|
From page 117... ...
(Analysis of Fig Id Test Results) (Selection of Assessment Materials for Final Assessment)
|
From page 118... ...
Items are developed to include a mix of multiple-choice and a variety of constructed-response items and performance tasks as specified in the framework and specifications. ETS staff and assessment development committee members review and edit all assessment materials, which are also reviewed for potential sources of bias.
|
From page 119... ...
. ETS staff and development committee members participate in the selection of items for the final version of the assessment and the revision of scoring rubrics based on the initial wave of incoming student responses.
|
From page 120... ...
120 ._ ._ it: o In Q In .
|
From page 121... ...
NAGB's current plans call for NAEP final assessments to be readministered periodically (at 4-year intervals for reading, writing, mathematics, and science; see Table I-1~. Because some assessment materials are released to the public after each administration of a final assessment, a new round of item development and field testing is conducted to replace those materials.
|
From page 122... ...
Framework Consistency with Disciplinary Goals In general, the NAE panel found the NAEP frameworks for the 1990 and 1992 mathematics assessments and the 1992 and 1994 reading assessments to be reasonably well balanced with respect to current disciplinary reform efforts and common classroom practices in reading and mathematics. In reading, the panel concluded that the framework and the assessments were consistent with current reading research and practice, incorporating innovations in assessment technology such as interesting and authentic reading passages, longer testing time per passage, and a high proportion of constructed-response items (National Academy of Education, 1996:9~.
|
From page 123... ...
Continuity Across Framework and Assessment Development Activities Recognizing the complex, multistep nature of the NAEP assessment development process, the NAE panel recommended that mechanisms be implemented to ensure continuity throughout the process. The panel suggested that the mechanism could be a set of subject-specific oversight committees that monitor all steps of the process, from framework development to reporting, in order to ensure that the intentions of the framework developers were reflected in the assessment materials and in reports of NAEP results (National Academy of Education, 1992:30~.
|
From page 124... ...
an argument for a broader conceptualization of student achievement in future NAEP frameworks and assessments, (3) a recommendation for the use of a multiplemethods strategy in the design of future NAEP assessments, and (4)
|
From page 125... ...
In order for desired inferences about student achievement to guide the assessment development process, there must be a high degree of continuity from one step to another in the process, from the conceptualization of the framework, to the development of assessment materials and scoring rubrics, through the reporting of results. Too often the intentions of the developers of the framework can be diluted, and even unrealized, if there is not sufficient attention to carrying out the inferential goals described in the framework throughout the entire assessment development process.
|
From page 126... ...
15~. Thus, the framework for the 1996 NAEP science assessment includes both broad and detailed content coverage and the process skills that are accorded importance in national science curriculum standards.
|
From page 127... ...
As described by NAGB (Reading Frameworkfor the National Assessment of Educational Progress:1992-1998; National Assessment Governing Board, no date, b:910) , the framework acknowledges a number of different aspects of effective reading and a number of variables that are likely to influence students' reading performance (see Figure 4-3~.
|
From page 128... ...
Maintaining broad coverage of subject-area knowledge and skills is still a major focus of the frameworks, particularly in science and mathematics. Although breadth of coverage supports traditional assessment methodologies that result in summary scores as indicators of student achievement, it provides little insight about the level and depth of student understanding that is valued in many current views of student learning.
|
From page 129... ...
A detailed description of methods and results is presented in the volume of research papers that accompanies this report (Sireci et al., 1999~. In one study, 10 eighth-grade science teachers who were familiar with science education reform and the goals of the NAEP science framework were asked to study and discuss the framework and then classify items from the 1996 eighthgrade science assessment by content area and process area.
|
From page 130... ...
Although it may truly be the case that these dimensions thread through all parts of the science assessment (and in ways that are perceived differently from one subject-matter expert to the next) , it is clear that these dimensions must be more clearly and narrowly defined in the framework.
|
From page 131... ...
Improved Assessment of Knowledge and Skills Described in the Frameworks As stated earlier, the science, mathematics, and reading frameworks have incorporated many aspects of the standards-based goals of the disciplinary com
|
From page 132... ...
They analyzed scoring rubrics and student responses for several extended constructed-response items from the 1996 main NAEP mathematics assessment and concluded that varying levels of sophistication in the reasoning used by students to respond to the items were not reflected in the rubrics they examined. When the NAEP science framework was developed in 1990-1991, the NAGB-appointed steering and planning committees believed that it was imperative that the assessment include measurement of student achievement via handson tasks.
|
From page 133... ...
In this appendix, we provide a detailed analysis of a grade 8 reading passage and set of related items and scoring rubrics that were administered as part of the 1994 main NAEP reading assessment. In doing so, we illustrate how there is still much to be gained through improvements to the current large-scale assessment materials.
|
From page 134... ...
These experts have been given major decision-making roles, and this effort appears to have helped improve continuity. For example, during the development process for the 1996 NAEP science assessment, there was notable continuity of personnel involved in various stages of the process: · the 2 leaders of the framework development effort also oversaw the development of the assessment and exercise specifications; · 5 of 11 members of the NAGB-sponsored committee that developed the preliminary achievement-level descriptions had served on the committees that developed the framework, and 5 were also serving on the assessment development committee; · 5 of 13 members of the assessment development committee had also served on the committees that developed the frameworks; · many members of the assessment development committee played a large role in developing and refining scoring rubrics and rater training protocols; · members of the assessment development committee were involved as leaders at various stages of the achievement-level-setting process; · 3 members of the original committees that developed the framework in 1991 continued to participate as members of the assessment development committee through the 1996 assessment scoring sessions and were leaders in the 1996 and 1997 achievement-level-setting sessions.
|
From page 135... ...
As stated earlier, current assessment items and tasks often are not well designed to measure complex aspects of student achievement described in the frameworks. Also, when items and tasks are well designed, the scoring rubrics are not consistently designed to attend to key differences in students' levels and types of understanding of the knowledge and skills specified in the framework.
|
From page 136... ...
For the four new main NAEP large-scale survey assessments that have been developed since that time (the 1994 U.S. history and geography assessments; the 1996 science assessment; and the 1998 civics assessment)
|
From page 137... ...
Field-testing could then occur in the following year, followed by the administration of the operational assessment in the year after that. Broader Conceptualization of Student Achievement In addition to improving the assessment of important cognitive skills presented in the current frameworks, we contend that NAEP frameworks should incorporate a broader conceptualization of achievement, and that there is considerable research on cognition, learning, and development that could inform the design, conduct, and interpretation of NAEP (see also Greeno et al., 1997; National Academy of Education, 1997; National Research Council, 1 999a)
|
From page 138... ...
. We contend, as did the NAE panel, that advances in the study of cognition provide valuable insights into problem solving, explanation, interpretation, and how complex understanding is achieved, and they can be used to inform the development of assessments that better measure these dimensions of achievement than can the current array of broadly used large-scale assessment technolog~es.
|
From page 139... ...
More important is the overall pattern of responses that students generate across a set of items or tasks. The pattern of responses reflects the connectedness of the knowledge structure that underlies conceptual understanding and skill in a domain of academic competence.
|
From page 140... ...
The implications of incorporating a cognitive perspective into NAEP on the types of results that can be reported is discussed in more depth in a later section of this chapter. The arguments for including a broader conceptualization of achievement in NAEP are strengthened further when one examines the degree to which these aspects of student achievement are consistent with the more comprehensive goals for schooling that have been put forth as required skills and abilities for an educated person in the next century (Resnick, 1987; SCANS Commission, 1991; Murnane and Levy, 1996~.
|
From page 141... ...
Re-creating these cognitively complex performances in assessment materials may not even be possible. However, extracting data from naturally occurring student performances by videotaping student activity and computer-based analysis of students' written work offers promise as alternative means of data gathering on these aspects of achievement.
|
From page 142... ...
The example provided, like the example shown in Appendix A for reading, starts from existing NAEP materials but significantly augments how items are structured individually and collectively, thereby enhancing what can be determined about levels of students' understanding in the domain. Later in this chapter, we discuss the relevance of this example in the context of recommendations for providing more informative portrayals of student achievement in NAEP reports.
|
From page 143... ...
that depicts the relative demands of the content knowledge and science process skills required for successful completion of a given science assessment task. In this space, task demands for content knowledge are conceptualized as falling on a continuum from rich to lean.
|
From page 144... ...
Hands-on science performance assessment tasks, as well as other innovative formats for science assessment (see Shavelson, 1997) , can involve many possible combinations of content knowledge and process skills.
|
From page 145... ...
The examples we have provided of the application of cognitive theory and research to the design of enhanced assessment materials are only illustrative. Accepting the reality that there are limits to how extensively cognitive theory and research can be applied to task and item development, in areas for which such knowledge exists, it should play a central role in framework and assessment development.
|
From page 146... ...
. If this set of questions is used as guidance, an assessment system designed to measure student achievement in the subject-area domains described in the frameworks as well as the broader conceptualizations of achievement would consist not only of the current large-scale survey assessments, but would also include a range of assessment methods a new paradigm NAEP.
|
From page 147... ...
Insofar as possible, data from multiplemethods NAEP and core NAEP' s large-scale surveys should be linked, and data from all methods administered across a subject area should be used to represent student achievement in NAEP's reports (i.e., summary scale score results from
|
From page 148... ...
. Our recommendation for the use of multiple assessment methods is in some ways similar to one proposed by the current testing subcontractor, Educational Testing Service, in its 1997 report, NAEP Redesigned, one of several papers submitted to NCES to inform planning for the current redesign of NAEP (Johnson et al., 1997~.
|
From page 149... ...
In our proposed design, some aspects of student achievement described in NAEP's frameworks would no longer be assessed via core NAEP and its large-scale survey instrumentation. The components of the current large-scale surveys that are intended to assess these aspects of achievement (extended-response questions, performance tasks)
|
From page 150... ...
Accelerated research regarding the use of naturally occurring student work as a basis for the assessment of student achievement is imperative. Successful development of multiple-methods NAEP also requires that new models for the development of assessment materials be implemented.
|
From page 151... ...
That alternative relegates NAEP to the role of an incomplete indicator of student achievement. Portraying Student Achievement in NAEP Reports Implementation of the committee's recommendations to improve the translation of the goals of current frameworks into assessment materials and to evolve the frameworks to encompass broader conceptualizations of student achievement would enable NAEP to produce broader and more meaningful descriptive information, both quantitative and qualitative.
|
From page 152... ...
However, summary scores should not be viewed as the only type of information needed to understand and interpret student achievement. In NAEP, we have argued that they represent performance on only a portion of the domain described in the frameworks, and thus they provide a somewhat simplistic view of educational
|
From page 153... ...
A single score tells very little about where students' strengths and weaknesses are, nor does it help improve student achievement, whereas a more descriptive analysis of student achievement could provide guidelines for curriculum decisions. How can NAEP provide the kinds of information about student achievement that is needed to help the public, decision makers, and education professionals understand strengths and weaknesses in student performance and make informed decisions about education?
|
From page 154... ...
Analysis of patterns of student responses across these items can reflect the knowledge structure that underlies students' conceptual understanding, providing a richer interpretive context for understanding overall achievement results. In such a scenario, families of items serve as the unit of analysis; that is, each item is not simply a discrete source of information unconnected to other items.
|
From page 155... ...
These reports, supported by funding external to NAEP, characterize student performance at different levels of detail appropriate for different audiences. For example, the most recent monograph, reporting on the sixth NAEP mathematics assessment, administered in 1992, includes an analysis of students' understanding of basic number concepts and properties, their computational skills, and their ability to apply number concepts and skills to solving problems, based on examinations of items that assess these skills and concepts (Kenney and Silver, 1997~.
|
From page 156... ...
. Although the NCTM interpretive teams have learned a great deal by analyzing student performance, the NAEP mathematics assessment is not specifically designed to support these kinds of within- and across-item analyses.
|
From page 157... ...
How would it differ from what exists now? If the recommendations presented in this chapter were implemented, NAEP would be characterized by: · an assessment development process that is guided by a vision of the kinds of inferences and conclusions about student achievement to be described in reports of NAEP results,
|
From page 158... ...
core NAEP subjects that are assessed using the current large-scale survey (for measurement of trends) and whatever multiple methods are best suited to assess aspects of the framework not well assessed through large-scale surveys, nontrend subjects assessed using whatever combination of surveys and alternative assessment methods is best suited to meet the goals described in the subject area's framework, · an array of alternative assessment methods to assess the broader conceptualizations of achievement that are included in future NAEP frameworks, and · subject-specific reports of achievement results that include in-depth portrayals of student achievement gleaned from the entire array of methods used to assess a subject area; in core subjects, such reports ideally would also include summary proficiency scores from large-scale assessments and results from achievement level setting.
|
From page 159... ...
Conclusion 4D. Measuring student achievement only through NAEP's current large-scale survey precludes adequate assessment of (1)
|
From page 160... ...
NAEP needs to include carefully designed targeted assessments to assess the kinds of student achievement that cannot be measured well by large-scale assessments or are not reflected in subject-area frameworks. Recommendation 4D.
|
From page 161... ...
FRAMEWORKS AND THE ASSESSMENT DEVELOPMENT PROCESS derstandable, minimize improper or incorrect inferences, and support the needs of users who seek information that assists them in determining what to do in response to NAEP results, and · development and implementation of sampling, scaling, and analysis models that accommodate the use of families of interdependent items in the large-scale survey assessment.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.