Grading the Nation's Report Card Evaluating NAEP and Transforming the Assessment of Educational Progress (1999) / Chapter Skim
Currently Skimming:
2 Streamlining the Design of NAEP
2 Streamlining the Design of NAEP
Pages 56-86
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 56... ...
However, the proliferation of multiple independent data collections national NAEP, state NAEP, and trend NAEP is confusing, burdensome, and inefficient, and it sometimes produces conflicting results. Summary Recommendation 2.
|
From page 57... ...
We discuss and provide proposals for: · Combining the trend NAEP and main NAEP designs in core subjects to preserve measurement of trends and allow updating of frameworks; Using more efficient sampling procedures for national NAEP and state NAEP in order to reduce the burden on states and schools, decrease costs, and potentially improve participation rates; .
|
From page 58... ...
However, as noted previously, the short-term trend lines of national NAEP and state NAEP reflect different assessment materials and student samples than does trend NAEP. NAEP' s multiple assessment programs evolved to preserve trend lines, at the same time allowing for updating of NAEP frameworks, and to obtain state-level NAEP estimates in main NAEP.
|
From page 59... ...
s9 be be a' .
|
From page 62... ...
. The sampling design for state NAEP has only two stages of selectionschools and students within schools since clustering of the schools within states is not necessary for economic efficiency (Allen et al., 1998b)
|
From page 63... ...
. The school and student sampling plan for trend NAEP is similar to the design for national NAEP.
|
From page 64... ...
Statistics of interest, such as proficiency distributions for the current NAEP samples and for subgroups defined by demographic characteristics, can be regarded as functions of aggregates of predicted latent proficiencies and student characteristics g(Oj,yj) for each student j.
|
From page 65... ...
NAEP's policy board, the National Assessment Governing Board (NAGB) , has expressed concern over the inefficiency of maintaining main NAEP and trend NAEP; they recently announced plans to investigate more efficient design options.
|
From page 66... ...
The panel cautioned that diminished participation in the state program might have deleterious effects on national NAEP. They and others have reviewed school and student sampling for national NAEP and concluded that the national samples are drawn by experienced staff using well-established scientific, multistage stratified probability sampling designs (KPMG Peat Marwick LLP and Mathtech, 1996~.
|
From page 67... ...
The NAEP program has been strongly encouraged to press for more timely reporting. Other reviewers join the NAE panel in making suggestions for the improvement of NAEP reports (Hambleton, 1997; Hambleton and Slater, 1996; Jaeger, 1992, 1996, 1997; Wainer, 1997; Silver and Kenney, 1997; Barron, 1999; Widmeyer Group, 1993~.
|
From page 68... ...
Sampling Designs for Current NAEP The role of probability sampling is crucial for current NAEP, since it minimizes selection biases in making inferences from the sample to the population. As with any sample survey, NAEP is equipped to provide estimates at high levels of aggregation (national, state, gender)
|
From page 69... ...
Although the estimates based on improper multiple imputation are valid, standard errors tend to be underestimated, particularly when the fraction of missing information is large. One possible fix is to include the entire process of fitting the item response models and creating the plausible values in the jackknife standard error calculation.
|
From page 70... ...
The distributions of performance in each of these narrow areas of proficiency could then be easily computed as empirical score distributions, with standard errors computed using jackknife repeated replication. Summary measures such as means of proficiencies aggregated over the narrow areas would also be easy to derive, but distributions of the aggregate summaries would not be available, since there would be no information on how each student performs on areas other than the one tested.
|
From page 71... ...
Specifically, our analysis of reporting for current NAEP and its timeliness, clarity, and utility suggests the following. Timeliness As we have noted, a major effort of the NAEP program has been to produce reports, especially the Report Cards, in a more timely manner.
|
From page 72... ...
In general, policy makers, teachers, administrators, and parents said that achievement data are important but that NAEP results and reports do not point them to potential implications for policy and practice. We contend that the NAEP program should report descriptive (scale score/proficiency)
|
From page 73... ...
Rationale for Combining Designs Inconsistent Findings The existence of multiple assessments is potentially confusing to NAEP's constituencies; for example, it can and has led to situations in which the trend in results in two successive national NAEPs are in the opposite direction from trends in successive trend NAEPs over the same time period. Figures 2-1 through 2-6 show NAEP results for reading and mathematics by grade for the national NAEP and the trend NAEP designs.
|
From page 74... ...
.. -'-I -- it 240 1984 1986 1988 1990 1992 1994 1996 Year Tested FIGURE 2-2 Mean NAEP reading scores (eighth grade and age 13~.
|
From page 75... ...
. -- i -- -- -- -- -- -- -~ -- -- -- -- -- -- -I'~ 1984 1986 1988 1990 1992 1994 1996 Year Tested FIGURE 2-3 Mean NAEP reading scores (twelfth grade and age 17~.
|
From page 76... ...
-- -- - ~ . all , ~ A ,.T ~ National - - ~ - - Trend 270 1984 1986 1988 1990 1992 1994 1996 Year Tested FIGURE 2-6 Mean NAEP mathematics scores (twelfth grade and age 17)
|
From page 77... ...
The 1Two of the inconsistencies in national NAEP and trend NAEP data may be attributable to anomalies in the 1994 reading results for grades 4 and 12.
|
From page 78... ...
Additional inefficiencies are associated with ongoing administration of assessments for every trend line the NAEP program supports. As currently configured, every cycle of trend NAEP administration, analysis, and reporting adds $4,000,000 to NAEP program costs.
|
From page 79... ...
Research to examine the effects of content differences and differences in item types for trend NAEP, main NAEP, and successive assessments would also needed. Linking and scaling research would be needed initially to place main NAEP results on the trend scale or trend results on the main scale and, again, to continue the trend line as NAEP evolves.
|
From page 80... ...
First, test administrators observe that some high school examiners do not make a serious effort to answer NAEP questions, rendering their scores of questionable value. The administrators' observations are corroborated by the high omit and noncompletion rates of 17-year-olds on trend NAEP and seniors on national NAEP.
|
From page 81... ...
Declining participation rates and earlier mentioned arguments about burden and inefficiency suggest the need to coordinate designs for national and state NAEP. In 1996 the NAE panel recommended that the scope and function of the state assessment program be reviewed in the context of an overall reevaluation of the NAEP program; at the same time, they noted that state NAEP is an important component of the NAEP program and recommended that it move beyond a developmental status.
|
From page 82... ...
Separate state and national testing is costly, since it requires that national NAEP and state NAEP are essentially two different data collection programs. Recall that the state program costs as much as the national assessment for testing at fewer grades.
|
From page 83... ...
It is unclear whether calibration samples would be necessary in every state in which main NAEP data would be derived from state NAEP administrations, or if calibration samples would be necessary in every state in which national NAEP data would be derived from state NAEP administrations. The need for calibration samples would reduce cost savings and sampling efficiencies from combining state and national NAEP.
|
From page 84... ...
SUMMARY OF PROPOSED DESIGN FEATURES NAEP: . A number of characteristics distinguish our proposal for a new paradigm Trends in reading, writing, mathematics, science, and other subjects for which there are sufficient resources would be estimated by core NAEP using large-scale assessment methods (separate testing for trend NAEP and main NAEP would be discontinued)
|
From page 85... ...
Recommendations Recommendation 2A. For reading, writing, mathematics, and science, combine main NAEP and trend NAEP into a single design that preserves the measurement of trends and allows periodic updating of frameworks.
|
From page 86... ...
Recommendation 2F. In order to accomplish the recommendations listed above, NAEP's research and development agenda should emphasize the following: · Estimation of the effects of differences in sample definition, content, task types, and administration procedures for trend NAEP and main NAEP with subsequent derivation of links to support the use of a single trend line in each discipline, · Estimation of the effects of the differences in participation rates, administration procedures, and bias for state and national NAEP with subsequent development of more efficient sampling procedures, · Exploration of alternatives for obtaining meaningful data from high school students, and · Development of clear, comprehensible reports and reporting metrics that provide descriptive, evaluative, and interpretive information in a carefully articulated and described report series.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.