3
The NIOSH-Sponsored Anthrotech Study’s Sampling Strategy
The validity of the National Institute for Occupational Safety and Health (NIOSH)-sponsored Anthrotech report depends largely on the sample population that it assembled; therefore, the committee examined the details of that sampling in this chapter.
THE ANTHROTECH SAMPLING PLAN
The goal of the NIOSH-sponsored Anthrotech study was to survey a group of individuals reflecting the racial, ethnic, age, and gender diversity of the respirator wearing workforce in the United States. The sampling plan used in the study (Appendix C) called for approximately 4,000 subjects that would be divided evenly into men and women, three age categories (18-29, 30-44, and 45-65), and four racial/ethnic categories (White, African American, Hispanic, and “Other”) (Table 3-1) (Zhuang, 2001). It was unclear to the committee why the NIOSH protocol directed Anthrotech to sample by “age, race, and gender,” and not use ethnic and racial categories from the United States Census, which uses White, African American, Hispanic, Asian/Pacific, and American Indian (U.S. Census Bureau, 2000; Anthrotech, 2004). The committee recognizes that issues of race and ethnicity are complex, and the terms are often inconsistently applied. However, to remain consistent with the designations used in the NIOSH-sponsored Anthrotech report the committee uses the terms as they are used in the original Anthrotech report.
TABLE 3-1 Proposed Demographic Sample Distribution
Race |
Male Age Group |
Female Age Group |
||||||
18-29 |
30-44 |
45-65 |
Total |
18-29 |
30-44 |
45-65 |
Total |
|
White |
166 |
166 |
166 |
498 |
166 |
166 |
166 |
498 |
African |
166 |
166 |
166 |
498 |
166 |
166 |
166 |
498 |
American |
||||||||
Hispanic |
166 |
166 |
166 |
498 |
166 |
166 |
166 |
498 |
Other |
166 |
166 |
166 |
498 |
166 |
166 |
166 |
498 |
Total |
664 |
664 |
664 |
1,992 |
664 |
664 |
664 |
1,992 |
SOURCE: Anthrotech, 2004. |
NIOSH requested that Anthrotech obtain measurements from 166 subjects in each age, gender, and racial/ethnic category. The proposed number of 166 subjects per category was based on a standard developed by the International Organization for Standardization (ISO). Standard 15535 recommends n = (1.96 × CV/a)2 × 1.534, where n is the sample size per cell, CV is the coefficient of variation for a specific measurement (standard deviation divided by mean times 100), and a is the level of precision desired. The investigators chose to define CV based on the menton-sellion length (CV=6.5/121.9=5.3%), as it was one of the more biologically variable measurements. Thus, if the CV is met for this measurement, it would likely also be met for the other measurements (Zhuang, 2001). The 95 percent confidence limits (reflected in the term 1.96), the choice of a 1 percent level of precision (1 percent of the mean), and the value 1.534 (a constant computed using a Bonferroni type of argument to adjust for the multiple comparison) were based on procedures outlined in the ISO 15535—General Requirements for Establishing an Anthropometic Database. However, the committee is unsure about the accuracy of the calculation for this task, since there was no discussion in the NIOSH-sponsored Anthrotech report and only limited discussion in the protocol, if the 95 percent confidence limits were appropriate for this specific task (Zhuang, 2001; Anthrotech, 2004).
To ensure that the sample adequately captured the variation of measurements within each of the racial groups, Anthrotech oversampled the African American, Hispanic, and “Other” groups—that is, the sample included a larger percentage of these minorities than is found in the general population. During the analysis, each subcategory was then weighted to accurately represent its proportion within the total population. The advantages of this approach are that it provides the needed level of preci-
sion for each group separately, and it provides the flexibility to reweight the data in the future as population demographics change.
Unfortunately, the NIOSH-sponsored Anthrotech study did not appropriately subdivide the sample population based on race and ethnicity. The “Other” category was overly diverse, including Asian, American Indian, and Alaskan Native subjects, while not including other groups that might be designated “Other.” Further, the groups included in “Other” have vastly different facial characteristics, so it is possible that most or all of those groups may have been undersampled. Given the diversity of the respirator-wearing workforce, it is important that the “Other” group be included in analyses of the entire target population. However, its small sample size and great diversity may make it difficult, if not impossible, to perform useful analysis of the “Other” category as a distinct subgroup.
SAMPLING PROTOCOL
Anthrotech’s sampling protocol involved three steps to obtain a characteristic sample population: (1) identify a subset number of representative geographical locations, (2) choose representative worksites, and (3) identify individuals willing to have their facial dimensions measured.
Because of the need to have investigators present at the worksites, as well as the difficulty and expense of moving raters from one site to another, the NIOSH-sponsored Anthrotech study limited the sample geographically to eight states: California, Illinois, Kentucky, New York, Ohio, Pennsylvania, Texas, and Virginia (Anthrotech, 2004).
Within each state, Anthrotech investigators selected worksites that were deemed representative. The NIOSH-sponsored Anthrotech report indicates that each potential study site was telephoned to ascertain the willingness to participate in the study. “More often than not,” the report says, “the organization or company invited chose not to participate.” When an organization did agree to participate, Anthrotech provided the company with a packet of information explaining the study protocol and purposes.
The report provides no further details on either the criteria Anthrotech used to select the various establishments, response rate from these establishments, or any comparison of participating versus nonparticipating sites. This committee found that the 41 worksites listed in the final report were more often larger establishments. It is not clear whether this
was due to a decision by Anthrotech to approach larger companies or whether larger employers were more likely to agree to participate. Anthrotech did note some difficulty identifying companies that had the time and energy to participate (Anthrotech, 2004). The committee is unsure, and there is no discussion in the report, on how this selection bias may have skewed the final datasets.
Once a work site was identified, Anthrotech investigators worked with a single individual who provided access to test subjects. This point of contact assisted with both scheduling and recruitment. Unfortunately, the criteria for selection of the test subjects are ill defined in the report, so it is difficult to ascertain if the selection process was random, or how potential issues that arose during the selection of the subjects may have biased the sample.
Anthrotech may have introduced an additional source of bias by providing monetary incentives to participate at some worksites. This is difficult to determine as the number and size of incentives is not reported. However, monetary incentives are well-known to create biases in survey data, partly because these monetary incentives appeal more to some kinds of potential participants than others and partly by increasing bias in responses. The first of these is not likely to be great here, and the second should not apply to facial measurements.
SAMPLE DISTRIBUTION
In some instances Anthrotech was not able to meet the sampling quotas, while in others the quotas were surpassed (e.g., the final male-female ratio was closer to 60:40 rather than the desired 50:50). Due to the weighting of the subpopulations, the NIOSH-sponsored Anthrotech report regards the final sample as accurate, although the report does not define exactly what is meant by “accurate.”
Researchers initially measured 4,026 subjects. The final sample was 3,998, after eliminating 27 subjects who were younger than 18 or older than 66, and an additional subject that failed to provide an age (Table 3-2). This final total was very close to the original target number of 24 × 166 = 3,984. The final distribution was also closer to the actual demographic makeup of the U.S. workforce than were the planned equal numbers per category. This has certain advantages, as the imbalance among
TABLE 3-2 Number of Individuals Measured in Each Demographic Category
categories adds to the variance of comparison among groups. Both objectives may be important, but the committee notes that little has yet been published comparing groups.
Geographic Distribution
As with demographic variation, Anthrotech intended to have an even distribution across geographic regions. However, the final distribution was more heavily weighted (~60 percent) toward Texas, Ohio, and Illinois (the midsection of the United States) (Table 3-3). While the NIOSH-sponsored Anthrotech study found no evidence that geographic regions affected face size and shape, the 2000 U.S. Census shows that most residents from Illinois and Ohio come from white European ancestors, whereas states in the southeast and southwest have more residents with ancestry other than Caucasian (U.S. Census Bureau, 2006). Additionally, many immigrants tend to concentrate in distinct areas of the United States (e.g., Asian immigrants in California, and Hmong and Somali immigrants in Minnesota) (Camarota and McArdle, 2003; Ronningen, 2004). Thus the limited geographic distribution may have unintentionally introduced a racial and ethnic bias.
TABLE 3-3 Geographic Distribution
State |
n |
% |
California |
229 |
5.69 |
Illinois |
1,564 |
39.12 |
Kentucky |
93 |
2.31 |
New York |
120 |
2.98 |
Ohio |
751 |
18.78 |
Pennsylvania |
29 |
0.72 |
Texas |
857 |
21.44 |
Virginia |
355 |
8.82 |
Total |
3,998 |
100.00 |
SOURCE: Anthrotech, 2004. |
Occupational Distribution
The test sample was achieved by surveying workers from a variety of different occupations, including construction, firefighting, health care, law enforcement, manufacturing, and others (Table 3-4) (Anthrotech, 2004). However, the study protocol did not explain why these industries were chosen as the representative sample population. Further, an analysis of the results indicates that in some of the categories, such as law enforcement and health care, the representation of men and women is out of the range of what might be expected. Consequently, the committee questions if the tallies were recorded correctly or if there was an unintended selection bias.
These data presented in the NIOSH-sponsored Anthrotech report (Table 3-4) differ from those published that are reproduced below in Table 3-5 (Zhuang and Bradtmiller, 2005). NIOSH explained that the discrepancy resulted from a data sorting error in the Excel spreadsheet. The data presented in Table 3-5 is consistent with collected data from the survey (Personal communication, Z. Zhuang, National Personal Protective Technology Laboratory, October 27, 2006).
LIMITATIONS OF THE SAMPLE POPULATION
Overall the committee found limited analysis in the NIOSH-sponsored Anthrotech study comparing the categorized subgroups, so it is not known whether age, race, and gender affect specific measurements, composite measure, or ultimately the overall utility of anthro-
TABLE 3-4 Occupational Distribution
Occupation |
Men |
Women |
Total |
|||
n |
% |
n |
% |
n |
% |
|
Construction |
594 |
23.35 |
47 |
3.23 |
641 |
16.03 |
Firefighting |
429 |
16.86 |
60 |
4.13 |
489 |
12.23 |
Health care |
776 |
30.50 |
75 |
5.16 |
851 |
21.29 |
Law enforcement |
381 |
14.98 |
1,100 |
75.65 |
1,481 |
37.04 |
Manufacturing |
121 |
4.76 |
7 |
0.48 |
128 |
3.20 |
Others |
243 |
9.55 |
165 |
11.35 |
408 |
10.21 |
Total |
2,544 |
100.00 |
1,454 |
100.00 |
3,998 |
100.00 |
SOURCE: Anthrotech, 2004. |
TABLE 3-5 Occupational Distribution
|
Men |
Women |
Total |
|||
Occupation |
n |
% |
n |
% |
n |
% |
Construction |
594 |
23.4 |
47 |
3.2 |
641 |
16.0 |
Firefighting |
774 |
30.4 |
74 |
5.1 |
848 |
21.2 |
Health care |
381 |
15.0 |
1,100 |
75.7 |
1,481 |
37.1 |
Law enforcement |
121 |
4.8 |
7 |
0.5 |
128 |
3.2 |
Manufacturing |
429 |
16.9 |
60 |
4.1 |
489 |
12.2 |
Others |
244 |
9.6 |
166 |
11.4 |
410 |
10.3 |
Total |
2,543 |
100.0 |
1,454 |
100.0 |
3,997 |
100.0 |
SOURCE: Zhuang and Bradtmiller, 2005. |
pometric face panels. Further, Anthrotech was unable to find a reference reflecting the ethnic and geographic distribution of respirator users, so the data set were weighted against the U.S. workforce as a whole. Anthrotech’s plan calls for reweighting of this sample, should data characterizing the population of respirator users become available in the future.
If survey data are to serve as a sound foundation for conclusions, both the sample and the analysis of the sample must meet certain criteria. These include
-
precise definition of the target population (the population to be covered by inferences from the data);
-
precise operational definition of the sampling frame (the persons to be sampled, including any clustering variables such as employer);
-
some form of random sampling from the frame;
-
understanding the nature (and if possible the general size and direction) of biases likely to impair inferences from the data to the target population;
-
specific uses of the data, with careful consideration of the need for statistical power, particularly in light of the expected biases in the data;
-
to the extent feasible, study of the data to learn about important influences on the findings; and
-
understanding the nature, sources, and size of variations in measurement.
The committee found no evidence that these matters received adequate consideration in the design and analysis of the NIOSH-sponsored Anthrotech survey.
Defining the Target Population
A clear definition of the target population is critical to survey design, which was not done in the NIOSH-sponsored Anthrotech study. The survey intends to reflect “respirator users,” but that term is ambiguous: Does “respirator user” mean workers who typically use respirators? What about part-time workers, occasional users, workers who only wear respirators in emergency situations, and other workers who should be wearing respirators? These questions may impact the target sample population, because the demographics (and facial measurements) of workers in these additional categories may differ from those of regular, full-time users.
Identification of Subjects
Anthrotech identified establishments that were willing to participate as measuring sites a number of ways, including searching public databases, the Internet, and telephone directories. However, the report did not provide sufficient information about how workers who use respirators were recruited for participation in this study (Anthrotech, 2004). Were they self-selected, selected by management, or randomly selected? Does the sample include overrepresentation of workers whom the management knew had adapted well to the respirators they wore? If so, the hard-to-fit may be seriously underrepresented.
Other Potential Biases
An understanding of potential biases is critically important in the interpretation of survey data. For example, responders are likely to differ from non-responders, so non-response rates are important. However, the committee recognizes that it would be difficult to even define non-response with a sample such as that in the Anthrotech study.
Additionally, random sampling at each stage of a multilevel scheme is important. Equal probabilities are not needed, but each person in the sampling frame must have a known, nonzero probability of being selected, and correct calculation of variances requires that pairwise probabilities also be known (e.g., the probabilities of each possible pair of workers within and between establishments will differ if sampling is clustered by establishment). It is not clear from the survey protocol or report if NIOSH and Anthrotech considered using a random sampling approach (Zhuang, 2001; Anthrotech, 2004).
Conclusion
The proposal and NIOSH-sponsored Anthrotech Report did not adequately define or represent an appropriate target population.
Recommendation 3-1: Define Target Populations More Precisely.
Future anthropometric face panel studies undertaken by, or on behalf of, NIOSH should have a statistically rigorous and valid sampling plan and implementation strategy that precisely define the target population and also ensure that the samples of selected subjects included in the studies are representative of the predefined sample population (e.g., the current workforce that wears respirators, workers who should be using respirators, the general United States workforce).
REFERENCES
Anthrotech. 2004. A head-and-face anthropometric survey of U.S. respirator users: Final report. Prepared by B. Bradtmiller and M. Friess for NIOSH/NPPTL.
Camarota, S., and N. McArdle. 2003. Where immigrants live: An examination of state residency of the foreign born by country of origin in 1990 and 2000. Washington, DC: Center for Immigration Studies.
Ronningen, B. J. 2004. Estimates of selected immigrant populations: 2004. St. Paul, MN: Minnesota State Demographic Center.
U.S. Census Bureau. 2006. Ancestry: 2000. http://www.census.gov/prod/ 2004pubs/c2kbr-35.pdf (accessed December 15, 2006).
Zhuang, Z. 2001. Anthropometric survey of respirator users: Study protocol. Pittsburgh, PA: National Institute for Occupational Safety and Health.
Zhuang, Z., and B. Bradtmiller. 2005. Head-and-face anthropometric survey of U.S. respirator users. J Occup Environ Hyg 2(11):567-576.