On Evaluating Curricular Effectiveness Judging the Quality of K-12 Mathematics Evaluations (2004) / Chapter Skim
Currently Skimming:
3 Framework for Evaluating Curricular Effectiveness
3 Framework for Evaluating Curricular Effectiveness
Pages 36-64
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 36... ...
The framework design was formed by the testimony of participants in the two workshops held by the committee, and by a first reading of numerous examples of studies. The framework's purpose is also to provide various readers with a consistent and standard frame of reference for defining what is meant by a scientifically valid evaluation study for reviewing mathematics curriculum effectiveness.
|
From page 37... ...
. This permitted us to consider the content issues of particular concern to mathematicians and mathematics educators, the implementation challenges requiring significant changes in practice associated with reform curricula, the role of professional development and teaching capacity, and the need for rigorous and precise measurement and research design.
|
From page 38... ...
In contrast, at the classroom level, teachers may select curricular programs and materials from a variety of sources that address these topics and call the result the curriculum. When a publisher or a government organization supports the development of a set of materials, they often use the term "curriculum" to refer to the physical set of materials developed across grade levels.
|
From page 39... ...
The primary components are presented in Figure 3-1: program components, implementation components, and student outcomes. Secondary components of the framework include systemic factors, intervention strategies, and unanticipated influences.
|
From page 40... ...
, we articulated a set of subtopics likely to need consideration. Program Components Examining the evaluation studies for their treatments of design elements was a way to consider explicitly the importance, quality, and sequencing of the mathematics content.
|
From page 41... ...
It is important for evaluators to determine these differences and to design evaluations to assess the advantages and disadvantages of these decisions in relation to student learning. At the heart of evaluating the quality of mathematics curriculum materials is the analysis of the mathematical content that makes up these mate
|
From page 42... ...
A second source of controversy evolves from a debate over the value of conducting content analysis in isolation from practice. Some claim that until one sees a topic taught, it is not really possible to specify what is BOX 3-1 Factors to Consider in Content Analysis of Mathematics Curriculum Materials Listing of topics Sequence of topics Clarity, accuracy, and appropriateness of topic presentation Frequency, duration, pace, depth, and emphasis of topics Grade level of introduction Overall structure: integrated, interdisciplinary, or sequential Types of tasks and activities, purposes, and level of engagement Use of prior knowledge, attention to (mis)
|
From page 43... ...
Alternatively, a curriculum might be designed to appeal to a particular subgroup, such as gifted and talented students, or focus on preparation for different subsequent courses, such as physics or chemistry. Implementation Components Curricular programs are enacted in a variety of school settings.
|
From page 44... ...
Appropriate Assignment of Students Decisions concerning student placement in courses often have strong implications for the success of implementation efforts and the distribution of effects across various student groups. Evaluations must carefully document and monitor the range of student preparation levels that teachers must serve, the advice and guidance provided to students and parents as to what curricular choices are offered, and the levels of attrition or growth of student learning experienced over a curricular evaluation study period by students or student subpopulations.
|
From page 45... ...
FRAMEWORK 45 TABLE 3-1 Categories and Examples of Implementation Component Variables Community/Cultural Resource Variables Process Variables Influences Teacher supply, Teacher organization Teacher beliefs concerning qualifications, and rate and professional learning, teaching, and of turnover community assessment Professional development Curricular decision Expectations of schooling and teacher knowledge making and future educational and career aspirations Length of class Course requirements Homework time Class size and number Course placements, Stability, language of hours of preparation guidance, and proficiency, and mobility of per teacher scheduling student populations Cost and access to Administrative or Combinations of ethnic, materials, manipulatives, governance of school racial, or socioeconomic and technology decision making status among students, teachers, and community Frequency and type of Forms and frequency Community interest and formative and summative of assessment and use responses to publicly assessment practices of data announced results on school performance Extent and type of Student beliefs and student needs and expectations support services Parental involvement Parental beliefs and expectations of new technologies, taking a function-based approach to algebra, using dynamic software in teaching geometry, contextual problems, and group methods. In addition, school districts are facing increasing shortages of mathematics teachers, so teachers frequently are uncertified or lack a major in mathematics or a mathematics-related field (National Center for Education Statistics [NCES]
|
From page 46... ...
Documenting the duration, timing, and type of professional development needed and implemented is essential in the process of examining the effectiveness of curricular programs. Because many of the programs require that teachers develop new understandings, there is a need for adequate amounts of professional development prior to implementation, continued support during implementation, and reflective time both during and after implementation.
|
From page 47... ...
Evaluation studies that consider this variable typically include classroom observation. Assessment Formative or embedded assessment refers to the system of classroom assessment that occurs during the course of normal classroom teaching and is designed in part to inform subsequent instruction.
|
From page 48... ...
It is also important that such testing provides evidence on the variation among students in learning and creates a longitudinal progression on individual student learning. More and more, schools and school districts are working to coordinate formative assessment with high-stakes testing programs, and when this is done, one can gain an important window on the relationship between curricula and one standardized form of student outcomes.
|
From page 49... ...
We believe that effectiveness must be judged relative to curricular validity of measures as a standard of scientific rigor.1 In contrast, local decision makers may wish to gauge how well a curricular program supports positive outcomes on measures that facilitate student progress in the system, such as state tests, college entrance exams, and future course taking. We refer to this as "curricular alignment with systemic factors."2 One should not conflate curricular validity of measures with curricular alignment with systemic factors in evaluation studies.
|
From page 50... ...
. In analyzing evaluations of curricular effectiveness, we were particularly interested in measures that could produce disaggregation of results at the level of common content strands because this is the most likely means of providing specific information on student success on certain curricular objectives.
|
From page 51... ...
Finally, in drawing conclusions about effectiveness, we considered whether the evaluations of curricula employed the use of multiple types of measures of student performance other than test scores (e.g., grades, coursetaking patterns, attitudes and perceptions, performance on subsequent courses and postsecondary institutions, and involvement in extracurricular activities relevant to mathematics)
|
From page 52... ...
Textbook adoption processes and cycles also would be considered systemic factors. As reported to the committee by publishers, the timing and requirements of these adoption processes, especially in large states such as Texas, California, and Florida, exert significant pressure on the scope, sequence, timing, and orientation of the curricula development process and may include restrictions on field testing or dates of publication (as reported by Frank Wang, Saxon Publishers Inc., in testimony to the committee at the September 2002 workshop)
|
From page 53... ...
In this case, evaluators might not seek broad systemic impact, but concentrate on data supplied by the adopters only, and perhaps only those adopters who embraced the fullest use of the program across a variety of school settings. Catalytic programs might concentrate on documenting a curriculum's potential and then use dissemination strategies to create further interest in the approach.3 A third strategy often considered in commercially generated curriculum materials might be to gain market share.
|
From page 54... ...
EVALUATION DESIGN, MEASUREMENT, AND EVIDENCE After delineating the primary and secondary components of the curriculum evaluation framework, we focused on decisions concerning evaluation and evidence gathering. We identified three elements of that process: articulation of program theory, selection of research design and methodology, and other considerations.
|
From page 55... ...
In terms of our framework, program theory requires the precise specification of relationships among the primary components (program components, implementation components, and student outcomes) and the secondary components (systemic factors, intervention strategies, and unanticipated influences)
|
From page 56... ...
Only by clearly articulating program theory and then testing competing hypotheses can evaluators disentangle these complex issues and help decision makers select curricula on the basis of informed judgment. 4Larger concepts are broken down into small subconcepts that are covered in individual daily lessons that are spread out throughout the year.
|
From page 57... ...
Typically, content analyses concentrate on program components while case studies tend to elaborate on issues connected to implementation. Comparative studies involve all three major components -- program, implementation, and student outcomes -- and tend to link them to compare their effects.
|
From page 58... ...
The selection of variables was often critical in determining if a comparative study was able to provide explanatory information to accompany its conjectures about causal inference. Many of the subsequent sections of this chapter apply directly to comparative study, but can inform the selection, conduct, and review of case studies or content analyses.
|
From page 59... ...
. The goal of a comparative research design in establishing the effectiveness of a particular curriculum is to describe the net effect of a curricular program by estimating the gross outcome for an intervention group and subtracting the outcome for the comparable control group, while considering the design effects (contributed by the research methods)
|
From page 60... ...
It stresses that a slow, and sometimes agonizing, process of analyzing cases provides the detailed structure of argument often necessary to understand and evaluate complex phenomena. In addition to documenting implementation, this methodology can also include pre- and post outcome measures and the use of logic models, which, like program theory, produces an explicit statement of the presumed causal sequence of events in the cause and effect of the intervention.
|
From page 61... ...
However, as one reaches the stage of summative evaluation, there are clear concerns about bias when an evaluator is too closely affiliated with the design team. Time Elements The conduct of summative evaluations for examining curricular effectiveness must take into account the timeline for development, pilot testing, field testing, and subsequent implementation.
|
From page 62... ...
In addition, the extent of the variation in the types and quality of the outcome measures used in these studies of evaluating curricula makes effect sizes a poor method of comparison across studies. Nonetheless, by more informally considering effect size, statistical significance and the distribution of results across content strands, and the effects on various subgroups, one can identify consistent trends, evaluate the quality of the methodologies, and point to irregularities and targets for closer scrutiny through future research or evaluation studies.
|
From page 63... ...
acknowledged the difficulties associated with experimental methods as he wrote, "Interpreting [RCTs] results depends on many other things, an unbiased assignment process, adequate statistical power, a consent process that does not distort the populations to which results can be generalized, and the absence of treatment-correlated attrition, resentful demoralization, treatment seepage and other unintended products of comparing treatments.
|
From page 64... ...
must be used to inform decisions at a higher level. How one uses scientifically collected knowledge at one level to understand the dynamics at higher levels is still a key methodological and philosophical issue in many scientific fields of inquiry.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.