Skip to main content

Currently Skimming:

7 Conclusions and Recommendations
Pages 185-205

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.


From page 185...
... An answer to this question should help us learn how to respond to these debates. Curricular implementation involves the coordination of a variety of factors at differing levels of a system.
From page 186...
... The committee held fast to a single commitment, namely, that our greatest contribution would be to clarify the proper elements of an array of evaluation studies designed to judge the effectiveness of mathematics curricula, and to clarify what standards of evidence would need to be met to draw conclusions on effectiveness. The committee does not believe any single study determines effectiveness; however, drawing from what could be learned from previously conducted evaluations, we sought to uncover and present practical, sound, and rigorous evaluation designs that could produce the necessary evidence to resolve the debates, bring to the surface variations in values, and propel us toward better serving the nation's youth.
From page 187...
... To complement this work, the committee argued for approaches that draw from multiple methodologies, involve multidisciplinary roots, recognize the importance of ethics and volition, and acknowledge the dependency of the work on building and maintaining mutually respectful relationships with practitioners. Furthermore, evaluation, like research, benefits from the careful accumulation and synthesis of such work.
From page 188...
... The contribution of the committee is intended to help us ascertain these distinctive outcomes. THE QUALITY OF THE EVALUATIONS The charge to the committee was "to assess the quality of studies about the effectiveness of 13 sets of mathematics curriculum materials developed through NSF support and six sets of commercially generated curriculum materials." Based on our activities, the final product of our work was to present "the criteria and framework for reviewing the evidence, and indicating whether the currently available data are sufficient for evaluating the efficacy of these materials." Finally, if these data were not sufficiently
From page 189...
... The committee emphasizes that we did not directly evaluate the materials. We present no analysis of results aggregated across studies by naming individual curricular programs because we did not consider the magnitude or rigor of the database for individual programs substantial enough to do so.
From page 190...
... This permitted us to define a scientifically valid study as follows: For a single curricular evaluation to be scientifically valid, it should address the components identified in the "Framework for Evaluating Curricular Effectiveness." In addition, it should conform to the methodological expectations of the appropriate category of evaluation as discussed in the report (content analysis, comparative study, or case study)
From page 191...
... Furthermore, neither content analyses nor comparative studies typically provide enough detailed information about the quality of the implementation of a particular curriculum. A case study provides deep insight into issues of implementation; however, by itself, it cannot establish representativeness or causality.
From page 192...
... We view this as an optimal goal to which the field should strive in the attempt to make confident decisions about the effectiveness of any particular curriculum. The committee recommends that the following standard be used by agencies, curriculum developers, and evaluators: For a curricular program to be designated scientifically established as effective, a collection of scientifically valid evaluation studies address ing its effectiveness should (1)
From page 193...
... . The case studies should provide documentation that the implementation and outcomes of the program are closely aligned and consistent with the curricular program components and add to the trustworthiness of implementation and the comprehensiveness and va lidity of the outcome measures.
From page 194...
... and of the extent and type of professional development provided. The committee recognized the importance of even more specific information and encourages evaluators to seek methods to gather data on additional implementation components.
From page 195...
... These problems include: · A large and quite varied set of tests for the measurement of achievement, without a sensible and methodologically sound means to compare them; · Too many tests that rely exclusively on multiple-choice format, limiting the assessment of the cognitive levels of performance and neglecting the long-term development of student knowledge; · When tests are administered independent of the regular assessment activities, few means to gauge the level of student motivation to perform; · Lack of clear delineation of whether the measures of prior performance assess different content and skills, prerequisite skills, or the extent to which the current curricular material is already known, or other nonspecific factors of less obvious relevance to curricular effectiveness; · Reliance on a total test score of mathematics to make judgments, when such tests tend to be less sensitive to curriculum effects than subtest scores focused around very specific content such as fractions; · For longitudinal studies, lack of methodology to determine if variation in performance by subtopics, across school years, can be validly compared in relation to the psychometrics of the whole test-equating process; and · Lack of methodology on how to draw conclusions concerning the distribution of results across student groups, including by prior performance levels, to examine not only gains between subgroups or between comparative curricula, but to examine gains within subgroups using a particular curriculum. The committee could not solve this myriad of problems concerning the outcome measures used to assess curricular effectiveness.
From page 196...
... In our review of the evaluations of the curricula, our deliberations were hampered by the absence of adequately demonstrated curricular validity in outcome measures. The committee also recognized the importance of the demonstration of evidence of curricular alignment necessary for school decision makers, and that these may be dependent on local contexts.
From page 197...
... In order to assist the field in stabilizing this methodology, we outlined dimensions of content analysis and identified some of the key sources of debate. In relation to content analyses, the committee recommends that: Content analyses should be recognized as a form of connoisseurial assessments, and thus should be conducted by a variety of scholars, including mathematical scientists, mathematics educators, and math ematics teachers and well-qualified individuals, who should identify their qualifications, values concerning mathematical priorities, and potential sources of bias regarding their execution of content analyses.
From page 198...
... In addition, a study must have included at least one of the following additional design elements: · A report of implementation fidelity or professional development activity; · Results disaggregated by content strands or by performance by student subgroups; and/or · Multiple outcome measures or precise theoretical analysis of a measured construct, such as number sense, proof, or proportional reasoning. A set of 63 studies met these criteria and were closely examined for the lessons they could offer on the conduct of future comparative studies of curricular effectiveness.
From page 199...
... and by relation of testing to everyday practice (formative, summative, high stakes) , and ensure curricular validity of measures and assess curricular alignment with systemic factors.
From page 200...
... The committee recognizes the value of diverse curricular options and finds continuing experimentation in curriculum development to be essential, especially in light of changes in the conduct and use of mathematics and technology. However, it should be accompanied by rigorous efforts to improve our conduct of comparative studies, strengthening the results by learning from previous efforts.
From page 201...
... Thus evaluation studies would become far more valuable to the educational field. Moreover, the inclusion of explanatory variables would give program adopters more precise information about whether the conditions for effectiveness demanded by a particular curriculum coincide with their own local conditions, commitments, and resources.
From page 202...
... At the federal level, the committee recommends that: Calls for proposals by federal agencies should include more explicit expectations for evaluation of curricular initiatives and increasing so phistication in methodological choices and quality. No federal agency should provide continued funding for major curricular programs that fail to present evaluation data from well-designed, scientifically valid studies.
From page 203...
... differentiate market research from scientifically valid evaluation studies and (2) make such evaluation data available to potential clients who use federal funds to purchase curriculum materials.
From page 204...
... We have called for increased attention to program theory and implementation measures in program evaluation, for improvement in the curricular validity of outcomes measures, for improved use of experimental and quasi-experimental research design and coordination of multiple methodologies, and for a coalition of the federal government, districts and schools, and the
From page 205...
... Although the corpus of evaluations is not sufficient to directly resolve the debates on curricular effectiveness, we believe that in the controversy surrounding mathematics curriculum evaluation, an opportunity exists to forge solutions through negotiation of perspective, to base our arguments on empirical data informed by theoretical clarity, and to build in a critical degree of coherence that is often missing from curricular choice, that is, feedback from careful, valid, and rigorous study. Our intention in presenting this report is to help the nation to take advantage of this opportunity.


This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More information on Chapter Skim is available.