Methodological Advances in Cross-National Surveys of Educational Achievement (2002) / Chapter Skim
Currently Skimming:
11. Large-Scale, Cross-National Surceys of Educational Achievement: Promises, Pitfalls, and Possibilities
11. Large-Scale, Cross-National Surceys of Educational Achievement: Promises, Pitfalls, and Possibilities
Pages 319-350
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 319... ...
Conclusion
|
From page 321... ...
The United States already has participated in six cross-national surveys of mathematics and science achievement, four surveys of reading/literacy/language achievement, two surveys of civics education, and several surveys of student achievement in other domains (for a list of studies, see Table 4-1 of Chromy [this volumes. What is striking about this corpus of work, besides its growing size, is that cross-national surveys of achievement have been fielded at a more rapid pace each decade since the 1960s.
|
From page 322... ...
· How important is it to have international surveys of student achievement on a regular basis and with participation of a constant set of countries? In the following pages, I do not address these questions directly.
|
From page 323... ...
INTERNATIONAL COMPARISONS AS BENCHMARKS As Smith (this volume) shows, large-scale, cross-national surveys of student achievement have figured centrally in debates about educational standards in the United States since the 1980s, when findings about the performance of U.S.
|
From page 324... ...
As we shall see, there is room for improvement in the ways that international comparisons of student achievement are reported to audiences in the United States. But a case can be made that even the poorly crafted, early reports on international studies of educational achievement performed an important service in American education.
|
From page 325... ...
From this perspective, it appears that sound efforts have been made to ensure "fairness" in testing by allowing analysts from particular nations not only to have detailed knowledge about the alignment of all test items to national curricular goals, but also by allowing analysts from different nations to analyze achievement results using only items that meet an alignment standard for their own country. Linn also details important progress in constructing and scaling achievement tests over time.
|
From page 326... ...
All of this further enhances the fairness of cross-national comparisons by allowing researchers to examine differences in student achievement among groups of students and/ or across curricular domains that are more or less aligned to national standards. In combination with Linn's chapter, Hambleton's chapter shows that progress also has been made in adapting achievement tests and other data collection protocols for use in the many different nations involved in the cross-national studies.
|
From page 327... ...
Issues Related to Achievement Tests One problem I want to address is the extent to which the achievement tests used in cross-national surveys supply the kinds of "benchmarks" for student achievement that we want in the United States. It is well known that the average test scores for American students in cross-national surveys rarely lie at the top of the cross-national performance distribution and that our students frequently perform more in the middle of the pack (or below)
|
From page 328... ...
Nevertheless, as Linn points out, the use of newer item formats is inherently limited by restrictions on testing time in cross-national surveys and by the need to increase the sheer number of test items in particular content domains to enhance test reliability. This inherent tradeoff, Linn argues, explains why cross-national achievement tests still have a preponderance of conventional, multiple-choice test items pitched at lower levels of cognitive demand.
|
From page 329... ...
From this perspective, comparisons of average student performance in the United States to average student performance in other nations are not in and of themselvesan especially good yardstick for judging progress toward our most ambitious vision of educational standards. Instead, an appropriate international benchmarking process would more thoroughly investigate national curricula outside the United States and develop more challenging achievement tests.2 From this perspective, the goal of bringing the average performance of American students on current tests up to the national averages found in "higher performing" countries serves only as a useful starting point in achieving higher educational standards in American schools, for even if we achieved this goal, we would still not know whether we had met our most ambitious goals for student learning.
|
From page 330... ...
It is my view that the focus on country means is largely a legacy of the relatively primitive scaling procedures used in the earliest cross-national surveys, for as Linn (this volume) reports, it was not until the use of more sophisticated scaling techniques that the possibility of detailed reporting of within-country variation in student achievement scores emerged.
|
From page 331... ...
The problem of whether or not to adjust country means for sample composition cuts to the very heart of setting standards for student learning in American society. For example, many critics of the cross-national surveys have argued that focusing on mean differences across nations obscures many of the unique challenges faced by the American educational system.
|
From page 332... ...
The United States, in contrast, maintains an education system that is very open offering many "second chances" for slow starters all of which allows for parenting strategies that emphasize early investments in human, social, and cultural capital that are only loosely related to the narrow goal of acquiring school knowledge. To the extent that these observations are true, and they have been a stable feature of arguments about the American education system for decades in comparative sociology, cross-national comparisons of achievement among school-aged populations may not adequately reflect patterns of ultimate achievement in the United States.
|
From page 333... ...
Currently, we get some sense of how human capital is distributed at various stages of the life course within the United States in the longitudinal studies program of the National Center for Education Statistics. But I know of no systematic program of cross-national research dealing with this critical question.
|
From page 334... ...
For one, we might ask what we have learned from three decades of investment in largescale, cross-national surveys of student achievement, especially in comparison to the three-decade-old program of longitudinal studies supported by the National Center for Education Statistics (NCES) .5 But more importantly, we might ask how survey research can be used to study issues of educational improvement by probing more deeply into the idea that the world's varied education systems provide us with a "natural laboratory" allowing examination of the effects of alternative educational arrangements on student achievement and thereby informing education policy in the United States.
|
From page 335... ...
My personal view on this question, which arises from interest and experience in conducting research on schooling in the United States, is that the cross-national surveys of achievement have led to a number of important insights that have relevance, not only to researchers in the field of comparative education, but also to those interested in educational processes in the United States. In fact, like the BICSE members who contributed to the 1993 monograph, I see an important "cross-walk" occurring between studies of educational achievement conducted solely in a U.S.
|
From page 336... ...
have developed fascinating and important ideas about the fragmentation of the U.S. mathematics and science curricula and used these ideas to great effect as explanations for the performance of American students on TIMSS achievement tests.
|
From page 337... ...
The TIMSS video studies also are a methodological advance, allowing teaching events to be studied and restudied by observers who are not physically present, using different coding schemes for understanding the events that transpire in a given setting. Already, the use of video studies of teaching is attracting the attention of NCES and the U.S.
|
From page 338... ...
With these steps, cross-national surveys at least will be able to yield credible education production functions, even if they cannot produce the kinds of sound causal inferences gained from randomized experiments.7 For our purposes, an even more telling discussion of how cross-national research can be used to inform issues of school improvement is provided by Raudenbush and Kim (this volume)
|
From page 339... ...
For example, what would happen in the example above if no school system in the United States had a curriculum that even approximated the kind of curricular coherence, rigor, and focus characteristic of "high-performing" nations? 8 Under these conditions, we might be forced to actually create school conditions that approximate arrangements in other countries before testing their effects on student achievement in the United States, and from this perspective, the work of groups like the New Standards Project and the First in the World Consortium seem to be necessary first steps in translating at least some cross-national findings into practice in the United States, with sound causal inferences awaiting true randomized experiments.
|
From page 340... ...
. In this view, processes occurring within societies often depend less on unique circumstances within societies than on a given society's location in a worldwide system of international relationships, where national societies hold unequal statuses in a dense network of international relationships and participate in an increasingly uniform, worldwide culture.
|
From page 341... ...
As LeTendre discusses in this volume, well-conducted case studies can contribute in important ways to cross-national surveys by capturing the unique, culturally embedded nature of educational practices in nations. But LeTendre's discussion also shows that a great deal of ambiguity remains within the research community about how to use the information derived from case studies in relation to surveys, as well as the extent to which insights from case studies should drive issues of survey design, and how conclusions from case studies can be reported so that various members of the research and policy communities find them "valid." In fact, the simple contrast between LeTendre's discussion of the uncertain
|
From page 342... ...
In this regard, it is interesting to note that the practitioner community in American education seems to be doing just this, carefully recreating practices imported from other nations and testing them in their own educational settings.9 But this real progress in applying cross-national findings to problems of educational improvement is not much reflected in the current volume, except perhaps in Raudenbush and Kim's advice that hypotheses derived from cross-national comparisons should be carefully tested within the United States and in Smith's cautions about making inferences from cross-national studies to guide education policy. One hopes, therefore, that BICSE will pay more attention to this problem in its future discussions of the validity of large-scale crossnational research and articulate more clearly how cross-national findings can be used to stimulate school improvement in the United States.
|
From page 343... ...
However, they do not guarantee that the tests of achievement used in the cross-national surveys are "valid" indicators of the educational standards to which we aspire, or that the cross-national comparisons based on these studies give us a valid picture of where the United States stands in terms of meeting these standards. As I argued in the body of this paper, despite advances, the cross-national achievement tests used most recently still do not reflect our most ambitious learning goals for students, and the ages at which students are tested in the crossnational surveys might not reflect the goals we actually hold for our education system.
|
From page 344... ...
This is a difficult question to answer in the absence of information about levels of funding for future cross-national research. Certainly, advances in the development of computerized adaptive testing are worth exploring as means of improving cross-national achievement tests, especially if this approach to achievement testing can be used to improve the information yielded per item in measures of achievement, thereby reducing the number of items required in testing.
|
From page 345... ...
How important is it to have international surveys of student achievement on a regular basis and with participation of a constant set of countries? The evidence on this point seems fairly clear.
|
From page 346... ...
The cross-national surveys have helped stimulate a national and public debate about educational standards in the United States, and have done so on a regular basis. They have also pointed the way to some very interesting designs for educational improvement in the United States that have given rise to some very interesting efforts at school improvement.
|
From page 347... ...
raise an additional set of concerns about the various strategies used by comparative educationists to make causal inferences about system-level change using cross-national surveys, including cautions about missing data problems and how these problems affect cross-national analyses of cohort differences in achievement and system-level analyses of changes in achievement over time. I do not discuss these forms of analysis or Raudenbush and Kim's critique of them in this chapter, except to note that these kinds of analyses, and the problems discussed by Raudenbush and Kim, are central to the important goal of cross-national work discussed by BICSE (National Research Council, 1993)
|
From page 348... ...
(1995~. International comparative studies in education: Descriptions of selected large-scale assessments and case studies.
|
From page 349... ...
Educational Researcher, 22~3)
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.