Understanding Others, Educating Ourselves: Getting More from International Comparative Studies in Education (2003)

Since the end of World War II, the U.S. government and private funders have supported a wide range of international education activities, including student and scholarly exchange activities in various disciplines, and the development of K-12 curricula to expand awareness and understanding of the rest of the world. By contrast, U.S. government funding for comparative social science research focusing on education in other countries—what we will refer to as international comparative studies in education—is a much more recent phenomenon, focusing to date on a relatively narrow range of studies.

This report redirects attention to the full range of international and comparative education studies. This range includes studies that involve one or many countries; collect large and small samples; employ a variety of quantitative and qualitative methodologies¹; aim both to generate new hypotheses and test existing ones; and cover topics relevant to many levels, ranging from early childhood education to employment-related training for adults, and from governance issues for public and private schools to more practical issues, such as parent involvement in schools.

The breadth of these international comparative education studies and the way they spur interest in other research are illustrated by the summaries in the boxes scattered throughout this chapter. For the most part, these boxes highlight individual studies, rather than research programs

or reviews of specific topics. For example, books and studies by researchers associated with predecessors of TIMSS 1995 fed grassroots practitioner interest in Japanese lesson study, a teacher-driven process of professional development. Lesson study experiments are now being conducted in the United States as action research, with small collaboratives of researchers and teachers from both Japan and the United States using ethnographic methods in a few schools and districts. These studies are partly hypothesis generating, partly hypothesis testing studies; at present they do not aim to make generalizations beyond the cases they are studying, although at some later point they may. In contrast, the data from Liping Ma’s single-researcher study included interviews with only about 72 Chinese teachers, but Ma eventually used those data to articulate a kind of knowledge base for elementary mathematics teachers. Her work is now spurring domestic research in the United States on what Ma refers to as “the profound understanding of fundamental mathematics.”

In this report, we distinguish three types of international comparative education studies according to their initial, primary purpose. Type I studies typically include large-scale surveys that aim to compare educational outcomes at various levels in two or more countries. Type II studies are designed to inform one or more particular U.S. education policies by studying specific topics relevant to those policies and their implementation in other countries. Type III studies are not designed to make direct comparisons between the United States and other countries in terms of specific policies or educational outcomes. Rather, they aim to further understanding of educational processes in different cultural and national contexts.

This typology avoids false dichotomies in educational research— large-scale versus small scale, qualitative versus quantitative—but it is not without difficulties. First, the domains of different study types are not mutually exclusive. TIMSS 1995, for example, was comprised of several somewhat independent studies, each using different methodologies. Some of these methodologies—such as case studies and videotaped classroom observations—are generally associated with more open-ended goals. However, all TIMSS components were undertaken to inform the overall goal of comparing achievement cross-nationally. The large-scale survey component of TIMSS is therefore an unambiguous Type I study. To the extent that the video study component of TIMSS was undertaken primarily to inform the survey, the video component is also a Type I study, but as a stand-alone study focusing on classroom interactions, it would be considered a Type III study.

Second, Type III includes the vast majority of international comparative studies in education, but Type I studies receive the vast majority of funding. Although cost is certainly not the most important factor distinguishing the three types of studies, it is certainly one of the most contentious and therefore figures prominently in the discussion below.

To summarize, the typology used here is based upon the initial, primary purpose, rather than the ultimate use of a study. The typology creates overlapping rather than mutually exclusive domains. Furthermore, the types do not neatly divide the corpus of international comparative education research into three equal parts, either in terms of number of studies or funding. This typology would be inadequate for the study of international comparative education research as a discipline per se. Rather, it is intended to expand the discussion begun in earlier reports (1990 Framework and 1993 Agenda) to a broader range of studies and to a broader policy audience.

Type I studies typically use survey data to compare outcomes—such as performance on standardized tests—in two or more countries. They also include rates-of-return analyses, social mobility studies, and many cross-national studies of comparative education development. Box 2-1 provides three examples of recent Type I studies. Many Type I studies to date have been large-scale quantitative studies with one of two primary goals. One goal is to monitor the status of a target population subgroup (such as 15-year-olds) in each country over time. These we refer to as indicator studies; the Programme for International Assessment of Student Assessment (PISA) of the Organisation for Economic Co-operation and Development is one example. The other goal is to understand the relations between education variables and to generate hypotheses about causal relations. Studies of this kind are referred to as research studies; of these, TIMSS is the best-known example and the Civic Education Study of the International Association for the Evaluation of Educational Achievement (IEA) is another (see Box 2-1). Although research studies of this kind may be very useful in testing hypotheses about correlations and generating hypotheses about the causes of these correlations, they are not well suited to evaluating causal hypotheses (National Research Council, 2002a).

Trend studies covering a stable group of countries can strengthen the grounds for speculating on causes but cannot establish them. As noted previously, the quality of both indicator and research studies has improved dramatically over the past decade. Nonetheless, areas for improvement remain. For example, these studies require continued efforts

to improve measurement of important school and student background variables pertaining to socioeconomic status and to experiment with cross-national comparisons using jurisdictions closest to the level of education decision making—such as school districts or states in the United States. Note that this last area for improvement is likely to increase cost by increasing the number of samples and observations. For example, in PISA, national samples of around 5,000 were required, but to make interstate comparisons, Canada tested around 35,000 and Germany around 80,000. Finally, those who conduct and sponsor these studies are still struggling to find ways to communicate the difference between correlations and causality in releasing their findings to the public, and to make clear that surveys are more often the beginning, not the end, of a research process that must involve many different types of studies.

Type I studies that collect primary data tend to be the most expensive, in terms of both direct costs, such as the diplomatic capital required to secure agreement at the national level for primary data collection and the cost of conducting surveys and employing experts to organize and direct them, and indirect costs, such as the time demands placed on student and

administrator participants. But even within this subset of Type I studies, costs vary dramatically. For example, the estimated direct costs to the U.S. government of TIMSS 1995 are almost $20 million over five years. Other direct costs, however, including support for the Board on International Comparative Studies in Education (BICSE),² curriculum studies, and secondary analysis grants, totaled more than $3.5 million. In contrast, the U.S. contribution to the IEA Civic Education Study, much of it from private sources, was less than $1.7 million over three years.

The high cost of Type I studies must be weighed against the particular benefits. These studies enable us to establish benchmarks for comparing the performance of students in the United States to that of students in other countries. They also stimulate hypotheses about the reasons for differences in achievement levels in the United States and those elsewhere. The scale of Type I studies justifies spending significant sums on publishing and disseminating a variety of reports aimed at many levels of the education system. With help from public relations firms, these reports can attract the attention of a broad spectrum of the public and practitioners to specific issues in education, such as the depth of the curriculum. This is a double-edged sword: results issued with much public fanfare may dominate public debate long after smaller studies with much smaller budgets call them into question.

The relatively high cost of Type I studies increases the likelihood that attempts will be made to expand their potential uses by increasing the number of background variables selected, subgroups sampled, and topics studied, increasing cost at the design and data collection stages. As a result, two tightly focused Type I studies, each with a single purpose, may be less expensive than one expansive study, designed to serve many purposes.

Funding for the largest Type I studies derives principally from national governments and dwarfs funding for all other types. This focus on surveys is consistent with the pattern of development for federally funded domestic education research. The collection and dissemination of domestic education statistics was mandated almost 90 years before Congress authorized the U.S. Office of Education to fund broader education research (National Research Council, 1992).³ Demand for Type I studies is often generated by policy makers. For the past 10 years, the U.S. Department of Education has worked from a draft multiyear strategy for U.S.

participation in these studies (U.S. Department of Education, National Center for Education Statistics, 1994).

Nonetheless, the choices of timing, topics, frequency, and target group for these types of studies are not entirely within the control of U.S. policy makers, since the studies can be undertaken only with close cooperation from other participating countries. Sometimes U.S. participation in a study of relatively low priority for the United States may be necessary in order to secure broader cross-national participation in a study of higher priority. Every study that is rigorously conducted improves the capacity of the participating countries to produce better quality statistics, and in turn those can provide better, more consistent comparisons with U.S. education statistics. Conversely, by bearing more than its share of the cost, the United States sometimes persuades other countries to participate in studies of particular interest to the United States. For example, U.S. costs for TIMSS 1995 were very high, in part because the United States wanted more rigorous technical standards and shouldered much of the international costs of the study necessary to achieve those standards. The U.S. share of funding of both TIMSS 1999 and TIMSS 2003 were expected to decrease over time, as more countries adopt those standards and participate in cost sharing.

Type II studies are designed to inform U.S. education policy in a direct way, by examining specific policies and their implementation in other countries. Type II studies also include evaluations of attempts in the United States to implement similar policies and practices originating in other countries. Many issues—for example, innovation and change, teacher involvement in change, and teacher development—simply cannot be addressed by Type I studies. Such studies demand a mix of quantitative and qualitative, descriptive and interpretive studies. A study of the experience of U.S. schools that have implemented the Singapore mathematics curriculum falls into this category, as do school choice and high school tracking studies (see Box 2-2) and many of the studies produced by the Organisation for Economic Co-operation and Development, such as the 13-nation study of educational innovation and change in science and mathematics education (Black and Atkin, 1996; Raizen and Britton, 1997). These studies may be initiated by policy makers or by researchers, either anticipating or responding to U.S. interest in specific policies or practices in other countries.

Type II studies tend to be smaller in scale and therefore less expensive than those Type I studies that collect primary data, although they may be more costly than Type III studies (described below). Studies that aim to

make inferences relative to policy at decentralized levels may require larger sample sizes for primary data collection than broader, more generalized studies. Type II studies, however, do not necessarily require primary data collection. Meta-analyses of existing datasets, or syntheses of existing studies across countries, are also valuable; both the 1990 Framework and the 1993 Agenda reports specifically encouraged the United States to participate in more of this type of research, in order to get the most out of existing studies. Like Type I studies, Type II studies must draw on a limited pool of experts; they also require modest diplomatic capital.

Neither the federal government nor states have established formal priorities or budgets for this type of study, although they offer the most direct means to explore causal links between public policies and the performance of the education system, often for a fraction of the cost of Type I studies.

Type III studies are designed to increase general understanding about education systems and processes. Such studies are not designed to have immediate policy relevance, though many are of relevance to policy makers. Rather these studies are designed to bring to light new concepts, to stimulate interest in educational issues, to generally deepen understanding of education as a practice and as a social phenomenon and, most generally, to establish the foundation on which all other comparative education research is based.

These studies, like Type II studies, may be qualitative or quantitative, large- or small-scale, incorporating positivist, interpretive, or critical perspectives (Heck and Hallinger, 1999:143). However, unlike the other two types, Type III studies do not fall neatly under one purpose. Donmoyer (1999) identifies at least five potential purposes for qualitative Type III studies. These include efforts to answer the questions “What is the correct answer (assuming a particularly clear question)?” or “How do the people studied interpret the phenomenon?” or “How does an organization, individual, or group of individuals change over time?” or “How can the researchers simultaneously learn about and change educators and/or educational organizations?” Some quantitative Type III studies also relate to Donmoyer’s purposes and others’. To designate all these studies as one type does not do them justice, yet to disaggregate them systematically is a task larger than the charge of this report.

As described in Box 2-3, the work of Catherine Lewis, in her book Educating Hearts and Minds, and the work of Robin Alexander, using videotapes to explore classroom culture and pedagogy in five cultures, are examples of this type of study. Although relatively large in scale, the Alexander study qualifies as Type III since it was not driven by the desire

either to compare achievement or to inform a specific policy. Such studies may be conducted by U.S. scholars in foreign countries or by foreign scholars in the United States and provide new ways of looking at familiar practices.

Private foundations pay for many of these studies, the cost of which is generally measured in tens or hundreds of thousands of dollars. In terms of indirect costs, Type III studies are undertaken only as expertise is available; they are often arranged between individual scholars or research groups and require minimal diplomatic capital. Because they deal with many fewer schools and smaller samples, they have relatively little impact on the time of students, teachers, and administrators. Type III studies that involve in-depth analysis of a single aspect of an education system can be relatively inexpensive in terms of both direct and indirect costs; unlike more expensive Type I and II studies, they often do not produce either generalizable findings or research methodologies capable of being precisely replicated in other jurisdictions. Instead, many offer rich descriptions of context in particular settings; these rich descriptions can provide clues to the variables influencing the phenomenon.

As a result, although they vastly outnumber Type I and Type II studies, Type III studies often do not come to the attention of policy makers or

the public. This is a loss, since many are rich in narrative detail and paint a more engaging and provocative portrait of education in other countries than do the summary bar charts and graphs typical of many larger studies. Ethnographic and case studies, in particular, can explore cultural context in depth and, in turn, help elucidate the way education is organized and understood in different cultures. The detail these studies offer, however, renders them difficult to summarize, and their complexity often leads researchers to focus on just one or two countries, making their findings difficult to compare cross-nationally. The U.S. government currently has no formal priorities or budget for this type of study.

Establishing funding priorities for such a wide range of studies poses a challenge to all funders of international comparative education research. A balanced research agenda involves some support for basic statistics and indicators, other investments to address short- to medium-term policy concerns, and also significant investment in studies that explore broader issues in education over a longer period of time. Here the board suggests several criteria for developing a balanced, coherent research agenda. These criteria include investing in international studies that can help to address high-priority issues on the domestic research agenda; areas neglected to date by Type I studies; areas in which international research has a comparative advantage over domestic research; studies that respond to questions from a variety of positions and levels in the education system; and studies of the relative effectiveness of different methodologies in answering different types of questions and in communicating with different constituencies in the U.S. education system.

A part of the international comparative agenda should address some of the high-priority issues for domestic education research. Federally funded studies to date do this, to a limited extent, by focusing on Type I studies of achievement, in core curriculum areas, in formal K-12 schools, in countries perceived to be economic competitors of the United States. To the extent that these issues remain priorities for domestic education research and these international studies indeed are structured and timed in such a way that they are able to inform domestic research and policy, this current de facto international agenda is on target.

The domestic research agenda, however, is broader than the issues and methodologies evident in the de facto international research agenda. The list of high-priority domestic issues that could be illuminated by thoughtful research with international dimensions includes alternative assessment methods, finance, governance, teacher education, education at either end of the K-12 continuum, aspects of school outcomes other

than achievement, closing the achievement gap between students from differing family backgrounds, and the school-to-work transition.

To take one example: the National Educational Research Policy and Priorities Board in the U.S. Department of Education identified as a priority finding ways to better serve the growing number of English language learners in U.S. schools (U.S. Department of Education, National Educational Research Policy and Priorities Board, 2000). This particular domestic research priority should be informed by international studies comparing other countries’ experiences with second-language students and understanding of education systems in such geographic regions as Africa, Central and South America, and South Asia, which constitute the points of origin for large immigrant communities and many English language learners in U.S. schools. The weak state of education statistics in many of the countries in these regions means that many cannot participate in Type I studies in a meaningful way; fortunately, there is much to be learned about the children and parents who emigrate from these countries by means of other research methodologies.

Informing the domestic research agenda, however, should not be the only concern driving the international and comparative education research agenda. We should also reflect on how international comparative work can make unique contributions to basic understanding of the relationship between education and society. Because international research is so well positioned to reveal variation across societies, international comparative studies in education offer rich promise in deepening understanding of school-society relations and links between culture and schooling, religion and schooling, and the home and the school. In addition to those areas unevenly addressed by research (described above), these additional areas point to questions worth pursuing in future involvement in international studies.

Recommendation 1: Funding for international comparative education research should reflect a balance among the three types of international comparative education studies and should encompass a broad array of methodologies, scale, purposes, and topics. Specifically, the United States should increase investments in studies that focus on understanding the education experiences of other countries in their own context (Type II and Type III) to provide a broader context for U.S. experiences and efforts to innovate.

Yet another criterion for any research agenda for U.S. involvement in international comparative education studies is that it needs to include a range of questions generated from different originating points: address-

ing policy concerns, growing out of discipline-based theoretical concerns, accommodating and responsive to issues raised by other countries, and flowing from the experiences of practitioners. As elaborated in the following chapters, serious research is needed to uncover new ways of understanding education and good ideas for improving education should come from a variety of sources and flow in all directions among researchers, practitioners, parents, and students.

One way to ensure that international comparative education studies support and inform the domestic education research agenda is to encourage major domestic education research efforts to include an international component when that would add value to the domestic research findings. As do multimethod studies, international components work best when they are not simply contemporaneous add-ons to relatively complete domestic studies, but rather are executed in time to provide input at specific stages of the design, data collection, and analysis of domestic studies.

No single type of international comparative education study can answer all of the most pressing questions about how education works in other countries and how it might be improved in the United States. The most fruitful studies, however, are often those that combine multiple methodologies. For example, in the IEA Civic Education Study, case studies informed the design of frameworks for large surveys, which in turn generated correlations; these can now be explored by a range of other Type II and Type III studies tailored to specific geographic locations and contexts. At present, however, this sequencing of different types of related studies seems to be the exception rather than the rule. LeTendre (2002) argues that when multiple methods are used, an overarching framework for analysis that integrates results of the different research compo-

nents strengthens the overall analytic power of any study. (We return to this topic in Chapter 5.)

Many scholarly debates about the validity, relevance, and generalizability of findings of international comparative education studies focus on the extent to which context has been recognized and properly taken into account in various aspects of the study. While randomized field trials and quasi-experimental designs are sometimes held up as the gold standard in domestic education research on effects of interventions, such methods can work only when in-depth knowledge of specific contexts allows researchers to model and control independent variables.

In many areas of international comparative studies, those who wish to pursue randomized field trials will need to invest in much exploratory, open-ended work in order to identify salient variables, comparable populations, and critical differences in environment. Can we understand the function of school exit exams or university entrance exams in different countries if we do not understand the income distribution and the returns to school attainment in those countries? Can we understand the early school performance of children without understanding the child-care and pre-school policies of a country? Patterns of income distribution and returns to school attainment must first be identified and the means to measure them must be developed before researchers can “control” them sufficiently to apply experimental methods. More support is needed for qualitative and historical studies, which, in addition to being valuable in and of themselves, can build the necessary foundation for further quantitative work.

Although there is no single method of analysis that is agreed on by all international comparative education researchers, there is room for improvement in the methodologies of all three types of studies, and some component of any international comparative education research agenda should include studies to compare and improve research methodologies. Specific goals for improvement include organizing studies that use multiple methodologies in more effective ways. Designers and secondary data users could collaborate more effectively throughout the life of large-scale research studies to ensure greater use of the data. The use of video-

taped studies of classrooms in other countries could be better understood. Findings from international comparative education studies could be used to stimulate better-informed discussions about education among U.S. policy makers and the public.

In summary, because policy makers and educators in the United States cannot know in advance which studies will be critical, a prudent approach to developing a research agenda for international comparative studies in education is to support and encourage a broad range of study types and topics, with increased attention to those methodologies, topics, and geographical regions that have received relatively less investment in recent years, incorporating more international components into domestic studies, with some formative evaluations to study how domestic and international components can complement each other.

Finally, the scope of international comparative education studies means that no single set of methodological criteria will be adequate to evaluate the quality and promise of proposals for international comparative research and finished studies. As noted above, there is a continuing need for more qualitative and historical work that, while conforming to fundamental principles of sound research, is subject to different canons of systematic inquiry and different warrants for generalization than are large-scale surveys.