Methodological Advances in Cross-National Surveys of Educational Achievement (2002) / Chapter Skim
Currently Skimming:
7. Advancements in Conceptualizing and Analyzing Cultural Effects in Cross-National Studies of Educational Achievement
7. Advancements in Conceptualizing and Analyzing Cultural Effects in Cross-National Studies of Educational Achievement
Pages 198-228
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 198... ...
Scholars interested in comparative studies of education can still find themselves in a quandary as there is, to date, no single definition of culture or method of cultural analysis that is agreed on by all researchers. Researchers in the field still debate questions such as "How can an understanding of differences in cultures help us to better understand international differences in student achievement?
|
From page 199... ...
I will summarize the most important methodological advances in modeling cultural effects on student achievement in two large recent IEA studies, and propose new models of multimethod research design that can integrate cultural analyses and quantitative analyses in the same study. Finally, I will discuss the problems inherent in trying to create large, qualitative, public databases: the kind of databases that further the integration of cultural analysis in studying many aspects of educational achievement.
|
From page 200... ...
This kind of epistemological impasse has long kept qualitative and quantitative researchers from uniting in a common study of the cross-national causes and correlates of educational achievement. Invested in one mode of investigation, both quantitative and qualitative researchers find themselves bogged down in fruitless epistemological debates, missing a way to bridge qualitative and quantitative approaches to research: incorporating both methods in a larger research project designed to identify and test patterns of causality.
|
From page 201... ...
, a host of variables measured a wide range of teacher behaviors, beliefs, and backgrounds. Table 7-1 provides a partial list of variables related to teacher quality in the TIMSS Population 2 teacher questionnaire that many qualitative analysts would consider to measure cultural effects.4 Faced with such an alarming number of variables, researchers have turned to qualitative studies to help orient research and guide analysis and interpretation.
|
From page 202... ...
As early as 1976, IEA scholars called for increased attention to alternative ways to model cultural effects: The most interesting, and perhaps the most useful, approach to crossnational research proceeds not in terms of existing country-wide units, but on the basis of sub-national units. This means that it may be more interesting (for comparative work)
|
From page 203... ...
Researchers conducting comparative studies that use a cultural dynamic approach to study educational achievement, for example, first document the range and variation in respondent's definitions and knowledge of key concepts (e.g. achievement)
|
From page 204... ...
HOW CULTURAL ANALYSES IMPROVE OUR UNDERSTANDING OF ACHIEVEMENT How can an understanding of differences in national cultural dynamics help us to better understand international differences in student achievement? How can an analysis of culture help us to understand what are and are not possible lessons to be learned for the United States in terms of improving student achievement?
|
From page 205... ...
More accurate data on daily life form the basis for more accurate comparisons of national cultural dynamics and improve our knowledge of the implications and problems that need to be faced if programs or reforms are to be transferred from one nation to another. Capturing Daily Life Researchers in a variety of fields are drawn to qualitative methods as a way of more accurately documenting and portraying the social experiences of groups of interest.
|
From page 206... ...
That is, holistic analysis of the dynamic system can highlight pertinent cultural features or subsystems that can be targeted for more intense study. Qualitative data allow researchers to document the extent to which behaviors vary and to which disagreement is raised, and the kinds of behaviors about which people argue.
|
From page 207... ...
, like researchers in other fields, believed that some form of qualitative data was needed to accurately interpret survey results. Modeling Dynamic Systems Attempts to model cultural effects by using an increasingly large array of variables in cross-national studies failed because the potential number of variables that can be considered cultural is so large.
|
From page 208... ...
The problem remains in determining what the best way is to integrate an analysis of cultural effects or qualitative studies with quantitative studies in order to improve cross-national studies of achievement. Because the basic research designs and rationales of qualitative and quantitative studies "ask very different questions of the data," an overarching strategy of combined analysis seems most appropriate (see Tashakkori & Teddlie, 1998~.
|
From page 209... ...
COMPARISON OF TIMSS AND THE IEA CIVIC EDUCATION STUDY All IEA studies follow a rigorous research design process that includes extensive planning and review phases. TIMSS and the IEA Civic Education Study are of particular interest in that the designers explicitly tried to incorporate qualitative components into the research design in order to investigate cultural effects.
|
From page 210... ...
In this section, I will compare how cultural effects were modeled and studied, contrast the research designs in terms of component models, and identify the research design strengths that improve the integration of qualitative and quantitative data. In discussing TIMSS and the Civic Education Study, I will limit my discussion to the qualitative components and their role in providing an analysis of cultural effects.
|
From page 211... ...
Several of the components, such as the achievement tests, surveys, curriculum study, and case studies, measured multiple age ranges, a fact that further complicated the analysis needed to adequately assess the results of any one component. Initial planning and review National case study Review of case studies Analysis of preliminary data { Develop assessment measures FIGURE 7-2 Schematic of IEA Civic Education Study research design (mixed iterative)
|
From page 212... ...
rather than focus on integrating the case study data with other TIMSS components. A second design problem was the placement of the case studies in the overall TIMSS research design and the overall timeline of TIMSS.
|
From page 213... ...
To use such analysis to inform quantitative research designs requires early integration. The kind of analysis generated in the TIMSS case studies is most useful in the early stages of survey or instrument design, when important questions about meaning in specific linguistic or cultural contexts are being assessed.
|
From page 214... ...
The organizing committee for the Civic Education Study began by calling on the expertise of national experts and the IEA general assembly to support a study of civic education. Although the study, like previous IEA studies, used an implicit model of nation as the "case" or unit of analysis, the committee members evinced a belief that "civics" as a topic was likely to be strongly linked with national cultures and values in a way that studies of previous IEA topics were not.
|
From page 215... ...
The overall research design of the IEA Civic Education study is summarized in Figure 7-2. This design follows the integrated process suggested by Morgan (1998)
|
From page 216... ...
In dealing with national systems of education, issues of regional variation, the presence of subcultures, and variation in teacher practices over the course of schooling and a host of other issues mean that to achieve an integrated use of different forms of analysis, the study must have an extended time frame. INTEGRATING A CULTURAL ANALYSIS: AN ITERATIVE, MULTIMETHOD APPROACH The Civic Education Study demonstrates that an integration of cultural analysis and quantitative analysis can be achieved in cross-national studies through careful research design.
|
From page 217... ...
The IEA Civic Education Study employed an iterative model in which qualitative studies were used to generate ideas for further quantification, but several variations of such a model are feasible. The analytic power of the research design to identify and explain the impact of cultural effects can be maximized by paying attention to the analytic strengths of different kinds of data collected in the component studies, by controlling the temporal relationship of components to each other, by controlling the relative emphasis given to each component, and by using an overall integrated data analysis and instrument generation strategy.
|
From page 218... ...
Each method will record the instructional practice in slightly different ways, but these measures can be related to one another to provide a more accurate assessment of homework assignment. Triangulation also provides researchers with the opportunity to revise their instruments and fine tune their overall research design (see U.S.
|
From page 219... ...
However, if survey questions had been derived with input from analysis of the case study data, it is likely that such questions would have provided more insight into how intranational variation in belief structures compared with international variation (for a preliminary attempt to achieve this comparison, see LeTendre, Baker, Akiba, Goesling, and Wiseman, 2001~. To maximize the complementarily within the overall research design, however, more interaction between the exploratory components (case studies or other field studies)
|
From page 220... ...
Relative Emphasis of Components Tashakkori and Teddlie (1998) , for example, note that most multimethod research designs assign more or less emphasis to the component studies.
|
From page 221... ...
This material then could be used to generate a survey or an observation instrument that would be pilot tested. This pilot study would be likely to raise further questions or reveal other areas of interest that then would be incorporated into the final research design, where revised quantitative instruments and more focused qualitative study (perhaps a case study)
|
From page 222... ...
CONCLUSION: PROBLEMS THAT REQUIRE FURTHER ATTENTION AND EXPERIMENTATION The major challenge in incorporating cultural analyses in studies of student achievement is that most research in either qualitative sociology or cultural anthropology is exploratory and is thus designed specifically to raise questions, not test hypotheses formally derived from an existing body of theory. For some researchers trained in quantitative methods, the basic techniques of qualitative research violate what they perceive as the basic requirements of "good science." For example, using snowball sampling, where informants lead the research on to other informants, is a classic technique in qualitative research, especially among "hidden" (e.g.,
|
From page 223... ...
Documenting cultural change is crucial to understanding how current beliefs or values may or may not hold for the future and thus may or may not be relevant in subsequent assessments or policy recommendations. Large cross-national studies of educational achievement that attempt to incorporate significant qualitative components in order to analyze cultural effects on learning are a relatively new type of study.
|
From page 224... ...
The 1998 theme of the Comparative and International Education Society (CIES) was "Bringing Culture Back In." Clearly, many scholars who would identify themselves as comparativists are interested in issues of culture, yet the fact that the CIES organizers agreed that culture was something that needed to be "brought back in" suggests that significant problems remain in integrating studies that focus on a cultural analysis with more traditional "national case studies" of education and educational achievement.
|
From page 225... ...
NUD* IST is one of several qualitative data analysis packages used by qualitative researchers.
|
From page 226... ...
Comparative Education Review, 43~4)
|
From page 227... ...
Comparative Education Review 43~1)
|
From page 228... ...
(1999~. Civic education across countries: Twentyfour national case studies from the IEA Civic Education Project.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.