Skip to main content

Currently Skimming:

8. The Measurement of Opportunity to Learn
Pages 229-266

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.


From page 229...
... Part ITI Making Inferences 229
From page 231...
... The weight of evidence in those studies has shown positive association between OTL and student achievement, adding to interest in ways to use OTL data to deepen understanding of the relationships between schooling and student learning. In the broader realm of education research, its use has been extended to frame questions about the learning opportunities for others in education systems, including teachers, administrators, *
From page 232...
... This chapter will address these several areas by looking at the role of OTL in international comparative studies and at its use in selected U.S. studies of teaching, learning, and education policy.
From page 233...
... That shift supports the interest in opportunity to learn as a potentially modifiable characteristic of school that could significantly affect student learning. Carroll posits that the degree of student learning is a function of five factors: 1.
From page 234...
... For each level, common sense and, in some cases, empirical evidence suggest that OTL will be related to whether or how well students learn the content. International comparative studies, and the International Association for the Evaluation of Educational Achievement (IEA)
From page 235...
... They found that student achievement was most highly associated with high success rate.3 Therefore, the version of opportunity to learn that they found to be tied most closely to student learning is what they dubbed "Academic Learning Time" (ALT) , defined as "the amount of time a stu
From page 236...
... Questionnaires have been used in international studies to gather information on allocated time, tied to specific content. Such questionnaires give information on time allocation and the nature of the task, but shed little light on student engagement in the tasks, or on students' success rate.
From page 237...
... Policy makers in a country might, for example, be interested to see that some countries have included algebra content for all students in middle school, contrary to a belief in their own country that such content is appropriate for only a select group, or only for older students. The language of the FIMS reports suggests that the reason for asking whether students had an opportunity to learn content was to determine whether the tests used would be "appropriate" for the students: "Teachers assisting in the IEA investigation were asked to indicate to what extent the test items were appropriate for their students.
From page 238...
... In those three systems, the study of geometry constitutes a significant portion of the mathematics curriculum, and these results are an indication of the lack-of-fit between the geometry curriculum in those systems and geometry as defined by the set of items used in this study. These findings underscore the importance of interpreting these achievement results cautiously.
From page 239...
... HOW HAS OTL BEEN MEASURED IN INTERNATIONAL COMPARISONS? I will focus on FIMS, SIMS, and TIMSS, the three international comparative studies in which OTL has played the most significant role.
From page 240...
... Thus, teachers were asked both about whether the mathematics related to the test item has been taught or reviewed (the OTL question) and about the percentage of students in the class who would get the problem correct.
From page 241...
... That omission reduces information about the country's mathematics curriculum. It also might be important, for the appropriateness of the item for the test, to know whether the content was studied recently or some time further in the past.6 For SIMS, teachers also were asked OTL questions for each item.
From page 242...
... domestic research has included changing approaches to measurement, which can be drawn on in planning for future measurement of OTL in international comparisons. Using Teacher Logs to Gather Information on Instruction One line of OTL work in the United States was initiated by Porter and his colleagues at Michigan State University in the Content Determinants Project (Porter, Floden, Freeman, Schmidt, & Schwille, 1988~.
From page 243...
... This research team used teacher questionnaires to gather information on the content coverage in high school mathematics courses intended to provide a bridge between elementary and college preparatory mathematics. The questionnaires focused specifically on the content on which students would be tested in this study.
From page 244...
... Because the test covered only 19 of the 558 possible mathematics topics, values of this index are small, with averages across different tvues of mathematics classes ranging from 0.046 to 0.086. J 1 Gamoran labels the match of distribution to the pattern of topics on the test "configuration of coverage." The configuration is a measure of the match between the relative time spent on each tested topic and the number of test items on that topic.
From page 245...
... If the pattern of topic coverage on the test represents agreement about relative value, then it seems sensible to adjust aggregate indices of OTL according to the distribution of test items. If the pattern of coverage does not represent such agreement, that creates a problem for interpreting any aggregate score.9 Web-Based Approaches to Teacher Logs Ball and her colleagues have been pilot testing the use of Web-based technology to gather detailed information from teachers about their classroom instruction.
From page 246...
... The options included "students read trade books," "worked in textbooks," "worksheets," and "teacher-led activity." Agreement between researchers and teacher reports was about 75 percent for both reading and mathematics. Ball indicated that the pilot test results were encouraging for the ability of the Web-based instrument to capture number and duration of lessons, but that more work was needed to get valid information about topic, instructional task, student and teacher activity, and student engagement.
From page 247...
... of instruments used to study secondary school mathematics. Smithson and Porter asked teachers to keep logs over the course of a school year, recording information both on content and on instructional practices.
From page 248...
... The categories of practice for the study were chosen to permit a contrast between "traditional" teaching practices (e.g., lecturing) and the types of practices advocated in the NCTM teaching standards (National Council of Teachers of Mathematics, 1991~.
From page 249...
... The measures Mayer studied asked teachers to describe their practices, rather than asking them to report on the topics they covered. Mayer's study does, however, stand as a reminder that the reliability of teacher reports, whether in logs or in surveys, is an issue that deserves attention when developing OTL measures.
From page 250...
... Asking teachers to report about opportunities to learn at other grades would reduce the number of teachers who would have to give information, but the information would likely be less accurate. What about the timing of OTL measures when the purpose is to adjust for differences in OTL in interpreting achievement results or in understanding relationships within the education system?
From page 251...
... HOW HAS OTL BEEN USED? In the series of international comparative studies of mathematics and science, the OTL data have been used in both the intended ways described at the beginning of this chapter: as a basis for "appropriately" interpreting the achievement results and as a description of the curricula actually implemented within and across the participating countries.
From page 252...
... but substantial correlations between countries. Within countries the correlations between achievement scores and teachers' perceptions of students' opportunities to learn the mathematics involved in the test items were always positive and usually substantial.
From page 253...
... In interpretation of the SIMS results, however, OTL began to be seen as a policy relevant curriculum variable. The OTL data continued to be used as a basis for interpreting achievement data, giving a basis for deciding whether test items were appropriate for particular countries, but commentators also began to argue that a country's pattern of OTL was an appropriate topic for policy discussion, rather than a given part of the context.
From page 254...
... In his rejoinder, Westbury contends that he was trying to ground the comparison in "a notion of intentional curricula," which includes a wider range of factors, for which looking at courses as a whole was more appropriate. The debate illustrates some of the detailed analytic choices that can be made about how the OTL data are used in interpreting achievement data.
From page 255...
... and that more than one strategy for analysis may be used, and may yield substantially different interpretations. Analyses like these should be pursued in future studies, as OTL data are used as part of the process of modeling the determinants of student achievement.
From page 256...
... 22~.~2 Such family background variables are like OTL in that they may provide an explanation for achievement differences that otherwise might be attributed to differences in the education system. To understand the connections between schooling and achievement, some of the effects of family background can be statistically "controlled," either by including measures of the background variables in statistical models used to estimate links between schooling and achievement or by including measures of prior achievement, which would themselves be highly associated with family background.
From page 257...
... , they argue that the associations of the gains with curriculum content at the corresponding grade levels indicate that curriculum, that is, OTL within schools, is an important factor in learning the content of these items. The general purport of all specific items discussed above is that, while developmental and life experience factors may be involved in accounting for achievement changes, curricular factors undoubtedly are....
From page 258...
... , which used a combination of teacher logs and classroom observations to record how much time a sample of elementary school teachers "allocated to reading and mathematics curriculum content categories (e.g., decoding consonant blends, inferential comprehension, addition and subtraction with no regrouping, mathematics speed tests, etc." (Fisher et al., 1980~. (Classroom observations also were used to estimate the fraction of allocated time that students actually engaged in the learning opportunities.)
From page 259...
... The withincountry analysis used a hierarchical model with predictive variables that included student gender, language of the home, family help, hours of mathematics homework, proportion of class in top one-third nationally, class size, school size, and teacher's age. The model was estimated for each country by achievement topic area (i.e., arithmetic, algebra, geometry, measurement, statistics, total)
From page 260...
... OTL's link to curricular intentions has become a major spur for discussions about an individual country's curriculum, at least in the United States. The positive association between OTL and student learning has been documented in a number of studies, although the measured association between OTL and achievement or achievement gains has been quite varied.
From page 261...
... Moreover, methodological studies suggest that teachers are less reliable in giving retrospective information about time spent using different methods of instruction than they are in giving information about time spent on specific topics. · Information about opportunities to learn in grades other than those tested is important for understanding a system's curriculum and for understanding connections between school practices and student learning.
From page 262...
... Further work could be done to investigate the construct validity of OTL measures. In short, researchers and policy makers have come to agree that OTL should be an important part of international comparative studies of achievement, in large part because OTL has been shown to have a link to student learning.
From page 263...
... 7. Porter and his colleagues have combined an instrument for measuring instructional practices with one measuring instructional content in mathematics and science into a package they call the Surveys of Enacted Curriculum (SEC)
From page 264...
... , The IEA Study of Mathematics III: Student growth and classroom processes. New York: Pergamon Press.
From page 265...
... (1988~. Content determinants in elementary school mathematics.
From page 266...
... (1989~. The IEA Study of Mathematics I: Analysis of mathematics curriculum.


This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More information on Chapter Skim is available.