Grading the Nation's Report Card Evaluating NAEP and Transforming the Assessment of Educational Progress (1999) / Chapter Skim
Currently Skimming:
1 Creating a Coordinated System of Education Indicators
1 Creating a Coordinated System of Education Indicators
Pages 22-55
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 22... ...
The nation's educational progress should be portrayed by a broad array of education indicators that includes but goes beyond NAEP's achievement results.
|
From page 23... ...
Department of Education is providing an interpretive context for NAEP results helping policy makers, educators, and the public better understand student performance on NAEP and better investigate the policy implications of the results. · The nation needs a new definition of educational progress, one that goes beyond NAEP' s student achievement results and provides a more comprehensive picture of education in America.
|
From page 24... ...
, Congress mandated that NAEP should: provide a fair and accurate presentation of educational achievement in reading, writing, and other subjects included in the third National Education Goal, regarding student achievement and citizenship. To implement this charge, the National Assessment Governing Board (NAGB)
|
From page 25... ...
NAGB's three objectives call for the collection of data that support descriptions of student achievement, evaluation of student performance levels, and the use of NAEP results in educational improvement. These policy goals presage our discussion of the diverse needs of NAEP's users.
|
From page 26... ...
When scores were released, the governor' s education adviser declared (National Academy of Education, 1996:120~: California made a horrendous mistake in taking out the phonics and the basic decoding skills from our reading programs, and when you do that, kids aren't going to learn to read anywhere well enough, if at all. South Carolina's poor showing was attributed to low expectations for student performance; local newspapers reported (National Academy of Education, 1996: 122~: South Carolina students ranked near the bottom in a national test of reading skills, as more than half failed to achieve even basic reading levels....
|
From page 27... ...
Our analysis of the large body of reports on the 1996 mathematics and science assessments revealed that NAEP data were used by varied audiences to make descriptive statements, to serve evaluative purposes, and to meet interpretive ends. Our observations about these numerous uses of NAEP results parallel those of McDonnell (1994)
|
From page 29... ...
29 5 ~ ., ~ , _ _ ~ =~ ~ ~ U ' ~ ~· ~ .= us ~ .
|
From page 31... ...
During the May 1998 meeting of the National Assessment Governing Board, board members discussed the inadequacy of some of the current data presentations. NAGB asked NAEP's technical staff from the National Center for Education Statistics (NCES)
|
From page 32... ...
Policy makers and educators need an interpretive context for NAEP to support in-depth understanding of student achievement and to intelligently investigate the policy implications of NAEP results particularly if performance is disappointing. In fact, as shown in Table 1-1 and elsewhere, in the absence of contextual data,
|
From page 33... ...
· Interpretive information about the system-, school-, and student-level factors that relate to student achievement can be provided by including NAEP in a broader, well-integrated system of education data collections. Within the context of NCES' data collections, there appears to be considerable need for improvement in data coordination to support this level of interpretive activity.
|
From page 34... ...
The feasibility of the effort we propose relies on the department's ability to capitalize on potentially powerful synergies among current efforts in ways that enhance the usefulness of NAEP results and contribute to the knowledge base about American educational progress. Several of the current data collections could serve as important sources of contextual information about student achievement and signify educational progress in their own right.
|
From page 35... ...
We return to the discussion of these specific data collections later in the chapter. PURPOSE AND USE OF INDICATOR SYSTEMS It is difficult to conceive of a system of education indicators that does not assign a key role to measures of student achievement in informing the public about how well schools are fulfilling their role in a democratic society.
|
From page 36... ...
36 Cal 1 ¢ V, z V, V, ¢ V, V, V, V V, V, En V, V V, o .= o ~ ~ z VO ¢ z c o 3 ca .t ~ O ~ .
|
From page 37... ...
37 ca ;^ ~ En, ~ V ~z · V (: ¢ a, .
|
From page 38... ...
At the culmination of its work, the group issued a report entitled Education Counts (Special Study Panel on Education Indicators, l991~. This report documents the panel's thinking about the development of an education indicator system and makes recommendations for improved federal collection of education data.
|
From page 39... ...
The earlier-mentioned Special Study Panel on Education Indicators (1991: 13) observed that "indicators cannot, by themselves, identify causes or solutions and should not be used to draw conclusions without other evidence." That panel and others contend that indicator systems can help identify school outcomes, student groups, and relationships among achievement and other variables that deserve closer attention and stimulate initial discussions about possible solutions (Bryk and Hermanson, 1993; Burstein, 1980; National Research Council, 1993~.
|
From page 40... ...
, RAND/UCLA's Validating National Curriculum Indicators project (Burstein et al., 1995) , and the Council of Chief State School Officers' State Collaboratives on Assessment and Standards Project, as well as from the earlier-discussed Special Study Panel on Education Indicators (1991~.
|
From page 41... ...
Currently, within NCES itself as part of the Schools and Staffing Survey Program, researchers propose to track what is happening in the nation's schools around issues of school reform by collecting information on teacher capacity, school capacity, and system supports (National Center for Education Statistics, 1997~. They will examine teacher capacity by documenting teacher quality, teacher career paths, teacher professional development, and teacher instructional practices.
|
From page 42... ...
NOTE: TIMSS = Third International Mathematics and Science Study; NELS = National Education Longitudinal Study; ECLS = Early Childhood Longitudinal Study. POTENTIAL VALUE OF A COORDINATED SYSTEM OF INDICATORS Two studies illustrate the value of embedding measures of student achievement within a broader range of educational measures: the Third International Mathematics and Science Study (TIMSS; Peak, 1996)
|
From page 43... ...
TIMSS researchers collected and analyzed data from student tests, student and teacher questionnaires, curriculum and textbook analyses, videotapes of classroom instruction, and case studies on policy topics. TIMSS researchers posed a series of questions and then designed data collections to obtain the information needed to help answer those questions.
|
From page 44... ...
. Grissmer and Flanagan investigated potential sources of improved performance by combining NAEP information with census data, information from the National Longitudinal Survey of Youth, and from the National Education Longitudinal Study.
|
From page 45... ...
In fact, these data collections could serve as important sources of contextual and associative data for NAEP and provide more telling information about educational performance. CSEI, the coordinated system of education indicators we envision, should: .
|
From page 46... ...
Examples of current NCES efforts include the High School and Beyond Survey and the National Education Longitudinal Study. NCES recently explored links between NAEP and the longitudinal studies, but links are not yet in place.
|
From page 47... ...
The rationale and more detailed descriptions of new paradigm NAEP both core NAEP and multiple-methods NAEP are addressed in subsequent chapters of this report. Relying on Mixed Methods of Data Collection Student achievement measures should rely on survey and other measurement methods; the same is true for schooling variables.
|
From page 48... ...
FIGURE 1-3 Measures of student achievement, including new paradigm NAEP. NOTE: TIMSS = Third International Mathematics and Science Study; NELS = National Education Longitudinal Study; ECLS = Early Childhood Longitudinal Study.
|
From page 49... ...
Some were previously mentioned, including the appointment of the NCES Special Study Panel on Education Indicators (1991) , work on the Third International Mathematics and Science Study (1995-1998)
|
From page 50... ...
Boruch and Terhanian prompt designers to consider respondents from the primary data collection to be the primary sample and to consider samples from the same population, but from other data collections, as augmentations to the main sample. Similarly, they suggest that variables from secondary datasets should augment those from the primary data collection; they cite information derived from students' high school transcripts as an example of augmentation data.
|
From page 51... ...
Domain and construct definitions, sampling constraints, data collection designs, instrumentation, accommodations, analysis procedures, and reporting models also should improve as information from the field and from large-scale survey data collections suggest questions of interest for small-scale observational or experimental studies; knowledge derived from the smaller studies should be funneled back into the large-scale data collections. PLANNING AND MANAGING THE SYSTEM The development and implementation of CSEI calls for careful consideration of alternatives and careful planning.
|
From page 52... ...
identification of data elements not collected by NCES or the U.S. Department of Education and determination of other data sources (other government statistical agencies or new data collections)
|
From page 53... ...
Unfortunately, despite the number of researchers, panels, and conferences that have issued calls for development of a coordinated system of education indicators, none provided cost and time estimates for developing, managing, and refining the system, nor did we uncover sufficiently parallel work in other statistical agencies to support estimation. Thus, the cost of implications of changing from NCES's current structure to the proposed coordinated system are not well understood, and they would undoubtedly be highly dependent on the design of the new system.
|
From page 54... ...
Other indicators of educational inputs, practices, and outcomes should be included in a coordinated system of education indicators. Data on curriculum and instructional practice, academic standards, technology use, financial allocations, and other important variables should be gathered using mixed methods of data collection.
|
From page 55... ...
CREATING A COORDINATED SYSTEM OF EDUCATION INDICATORS 55 improved by experience, public debate over results, and the system's findings over time. The subsequent chapters of this report demonstrate how the descriptive, evaluative, and interpretive purposes of NAEP's users would be met under our proposal for new paradigm NAEP and its inclusion in a coordinated system of education indicators.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.