Skip to main content

Currently Skimming:

4. Sampling Issues in Design, Conduct, and Interpretation of International Comparative Studies of School Achievement
Pages 80-114

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.


From page 80...
... A more recent text by Lessler and Kalsbeek (1992) is devoted entirely to nonsampling errors in surveys, classified as frame error, nonresponse error, and measurement error.
From page 81...
... may be less likely to participate in any of them or may have to select among them. Consequently, school response rates will suffer.
From page 82...
... · Advance arrangements with school officials should be arranged to ensure high participation rates. A maximum acceptable nonresponse rate should be specified for inclusion of a country's data in the international analyses.
From page 83...
... Minimum acceptable response rates are not specified. Weighting guidelines require use of base weights based on the selection probability and adjustments for nonresponse.
From page 84...
... With development of computing power and specialized software, direct estimation of survey sampling errors and the ability to routinely monitor other quality measures, including response rates, became the norm in survey practice. The availability of computers also fostered the execution of more complex sampling plans and the development of comparable sampling approaches through the use of a common set of procedures and sample selection software.
From page 85...
... Sponsor Description Countries Conducted IEA First International Mathematics 12 countries 1964 Study (FIMS) IEA Six Subjects Study: 1970-71 Science 19 systems Reading comprehension 15 countries Literature 10 countries French as a foreign language 8 countries English as a foreign language 10 countries Civic Education 10 countries IEA First International Science Study 19 systems 1970-71 (FISS)
From page 86...
... . SAMPLE ISSUES IN DESIGN, CONDUCT, AND INTERPRETATION · First International Assessment of Educational Progress (IAEP-I)
From page 87...
... The use of subsamples to generate a simple measure of sampling error also is discussed. Peaker presents data on achieved sample sizes by population studied, but does not present data on school or student response rates.
From page 88...
... For the age 14 sample, 18 systems reported school response rates ranging from 34 to 100 percent and student response rates ranging from 22 to 98 percent. Ten of the 18 had school response rates exceeding 85 percent; only six of 18 had student response rates exceeding 85 percent.
From page 89...
... SIMS had an international sampling manual (copy appended to the Garden report) and an international sampling committee.
From page 90...
... Although Garden's summary shows that 12 of 20 systems achieved response rates exceeding 90 percent and only two systems were below a 70-percent response rate, his examination of the country reports leaves some doubt about whether these reports account for the overall response rate when considering both school and student nonresponse or whether the executed student sample size could be determined. In some cases, the achieved sample exceeds the designed sample and no information is provided on the executed student sample size.
From page 91...
... and computed intraclass correlation coefficients, r, at the school level based on equating the design effect to 1 + rim - 1~. These estimates of the intraclass correlation coefficient were quite high, ranging from about 0.15 to in excess of 0.50.2 Second International Science Study SISS was conducted in 1983-84 in 17 countries.
From page 92...
... Students in the last year of secondary education had additional subpopulations defined as: Population 3B: Students studying biology for examination purposes. poses.
From page 93...
... identified the need for better definitions of target populations, for more thorough specification of sampling and other procedures, for consistent measurement of response rates using accepted definitions, and for improved monitoring procedures. First International Assessment of Educational Progress IAEP-I was conducted in February 1988 in six countries (12 educational systems)
From page 94...
... For the age 13 sample, the school response rate ranged from 77 to 100 percent, with 16 out of 20 countries exceeding 85 percent. Student response rates ranged from 92 to 99 percent for 19 countries reporting.
From page 95...
... Three target populations and two optional subpopulations were defined for TIMSS: · Population 1: All students enrolled in the two adjacent grades that contain the largest proportion of 9-year-olds at the time of testing. · Population 2: All students enrolled in the two adjacent grades that contain the largest proportion of 13-year-olds at the time of testing.
From page 96...
... less allowable exclusions. A criterion of limiting exclusions to 10 percent or less of the defined target populations was specified.
From page 97...
... Because Population 3 involved a general population of all students in the final year of secondary education and two subpopulations based on enrollment in advanced courses in mathematics and science, it was sometimes necessary to partition the eligible students into as many as four groups: · Those enrolled in both advanced science and advanced mathematlCS courses. · Those enrolled in advanced mathematics only.
From page 98...
... With regard to within-school sample execution, 24 countries provided complete information and all provided some information. Some comments noted included: · Unapproved school sampling procedures.
From page 99...
... For the upper grade schools in Population 2, before-school replacement rates ranged from 24 to 100 percent; after replacement they ranged from 46 to 100 percent. Most countries were able to increase their participating school sample by using replacements, particularly those with low initial school response rates; one country with a low initial school response rate only increased school participation by one school after replacement sampling.
From page 100...
... Designation in Reports 1 Acceptable sampling participation rates without replacement schools Acceptable sampling participation rates only with replacement schools 3 Before replacement of schools: School and student response rates each exceed 85 percent, or the combined rate exceeds 75 percent Not in Category 1 before replacement, but weighted school response rate before replacement exceeds 50 percent and after replacement the response rates meet the Category 1 requirements Unacceptable sampling participation rates even with replacement schools Not in Category 1 or 2 Appear without notation and may be ordered by achievement as appropriate Annotated with a dagger in tables and figures and may be ordered by achievement as appropriate Appear in separate section of reports ordered alphabetically
From page 101...
... · Determining school and student sample sizes consistent with PISA requirements. · Selecting the school sample (or providing the sampling frame to Westat, which will select the sample)
From page 102...
... Later in this chapter, I present some of my own conclusions, with a focus on current (2000) status of sample design and execution in international comparative studies in education.
From page 103...
... includes several statements relating to sampling, survey design, and, particularly, response rates. On coordination of survey design, measurement design, and analysis, it states: It was recognized that progress toward better statistical standards for international assessments will necessarily involve a more thorough understanding of the interrelationships between the educational measurement aspects of instrument design and testing and of survey sampling design and implementation issues, together with a recognition of the need to make explicit the analytic framework within which the data from the assessments are ultimately to be used.
From page 104...
... He also sees age definition as a possible solution to defining a target population near the final year of secondary education among students attending different types of educational institutions. Although sampling procedures for these studies involved standard applications of sample survey methodology, he notes the difficulty of ensuring uniformity across countries.
From page 105...
... data showing 13year-olds spread across grades 2 through 11, with most in grade eight and nearly 89 percent of enrolled students in grade eight or grade nine; this has serious implications for complete population coverage in the sampling frame when selecting samples defined by age. The grade by age distribution creates a more serious problem for test developers.
From page 106...
... This is not a sample design decision (as noted by Postlethwaite, 1999) , but it has serious implications for defining the sampling frame and selecting the student sample.
From page 107...
... Sampling Frame Completeness Sampling frame completeness can only be evaluated relative to the target population definition. Exclusion of schools for inappropriate reasons could be viewed as either a population definition issue or a sampling frame incompleteness issue, depending on the intended population of inference for the analysis.
From page 108...
... definition; generally, the sample is then drawn as a sample of students using simple random sampling. For populations defined by grade, the sampling frame often is developed based on a "complete" list of classrooms.
From page 109...
... Substitution for nonresponding schools has been a practice allowed in most of the international assessments. Although substitution does not eliminate bias due to nonresponse, it does maintain the sample size required to control sampling error.
From page 110...
... Are these criteria generally achievable? School response rates appear to be the more serious problem, particularly in the United States.
From page 111...
... The flagging of results depending on participation rates employed by TIMSS is a good example of a way to warn users about data quality based on something
From page 112...
... This applies particularly to the conditions in the educational system and how they should influence the definition of the desired target populations. The concept of final year of secondary education remains vague, especially in countries with alternative academic tracks; procedures designed to avoid double counting in the final year of secondary education population may be creating undercoverage of populations defined by subject matter specialization.
From page 113...
... . Prepared for the Board on International Comparative Studies in Education.
From page 114...
... Board on International Comparative Studies in Education, Norman M Bradburn & Dorothy M


This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More information on Chapter Skim is available.