Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Appendix B Examination of the Dimensionality of NALS CONFIRMATORY FACTOR ANALYSIS OF NALS The National Adult Literacy Survey (NALS) used a balanced incom- plete block (BIB) spiraling design for assigning items to test booklets and booklets to test takers, much like what is done for NAEP. There are 26 NALS test booklets. Each booklet contains 3 blocks of items, and test takers are given about 15 minutes per block. Each block of items appears in three different test booklets; each block appears with every other block at least once. In 1992, all test takers were given a set of six âcoreâ questions to familiarize them with the examination and testing procedures. The core questions were relatively easy and consisted of two questions per literacy area (prose, document, and quantitative). NALS included some questions/tasks that had been developed for the 1985 Young Adult Literacy Survey (n = 85) and some newly developed questions/tasks (n = 81). Table B-1 shows the distribution of the tasks across the three literacy areas. For this analysis, six booklets were selected, and the responses of the household survey participants (n = 24,944) were studied. This analysis replicated some of the procedures that had been used for prior dimension- ality analyses (e.g., Rock and Yamamoto, 2001) but selected different blocks of items. Table B-2 shows the booklets and blocks of items included in the analysis along with the distribution of the tasks across the three literacy areas and the number of test takers who received each of the tasks. Using LISREL, six confirmatory factor analyses were run in which a 214

EXAMINATION OF THE DIMENSIONALITY OF NALS 215 TABLE B-1 Distribution of NALS Tasks Across the Three Literacy Areas Task Scale Prose Doc. Quant. Total Blocks 1992 New Tasks 27 26 28 81 7 1985 Old Tasks 14 56 15 85 6 Total 1992 Tasks 41 82 43 166 13 TABLE B-2 Booklets and Blocks of Items Included in the Exploratory Factor Analysis Booklet Prose Doc. Quant. # of (Blocks) N Tasks Tasks Tasks Tasks 1 (1, 2, 13) 957 11 22 10 43 6 (9, 7, 10) 895 12 16 12 40 8 (8, 6, 12) 920 11 25 10 46 12 (12, 5, 3) 855 11 23 8 42 15 (2, 4, 6) 925 8 32 11 51 17 (4, 9, 11) 929 11 11 17 39 three-factor model was specified. Analyses were run separately for each test booklet. Because the six core items evidenced limited variability, the analy- ses were repeated with these six items removed. Tables B-3 and B-4 present the results with the core items included and excluded, respectively. Correlations between the literacy scales were quite high. When the core items were included in the analyses, correlations between the prose and document scales ranged from .89 to .94 for the six booklets, from .77 to .97 for the document and quantitative scales, and from .80 to .97 for the prose and quantitative scales. When the core items were removed from the analy- ses, correlations between the prose and document scales ranged from .86 to .94 for the five booklets (data matrix for Booklet 1 was not positive defi- nite), from .75 to .95 for the document and quantitative scales, and from .79 to .97 for the prose and quantitative scales. Model fit was evaluated using the root mean square error of approxi- mation (RMSEA). Fit tended to decrease slightly when the core items were

216 APPENDIX B TABLE B-3 LISREL Results for a Three-Factor Model When the Six Core Tasks Were Included in the Analyses Intercorrelations Booklet RMSEA* Prose/Doc. Doc./Quant. Prose/Quant. 1 .06 .92 .85 .88 6 .07 .92 .98 .91 8 .08 .89 .85 .97 12 .05 .91 .97 .95 15 .08 .94 .77 .80 17 .08 .94 .89 .88 *The RMSEA provides an estimate of the fit of the model to the data. TABLE B-4 LISREL Results for a Three-Factor Model When the Six Core Tasks Were Excluded from the Analyses Intercorrelations Booklet RMSEA* Prose/Doc. Doc./Quant. Prose/Quant. 1 Did not converge 6 .08 .91 .95 .90 8 .10 .87 .81 .97 12 .05 .86 .94 .97 15 .10 .94 .75 .79 17 .08 .92 .89 .87 *The RMSEA provides an estimate of the fit of the model to the data. removed. These results suggest that a three-factor model provided accept- able fit to the data. EXAMINATION OF THE RELATIONSHIPS BETWEEN LITERACY SCORES AND SOCIAL AND ECONOMIC CHARACTERISTICS Another set of statistical analyses addressed questions about the rela- tionships between the prose, document, and quantitative scores and an array of literacy outcomes. First, were the dimensions of literacy associated differentially with social and economic characteristics? For example, was prose more highly associated with outcome x than with outcome y, while

EXAMINATION OF THE DIMENSIONALITY OF NALS 217 quantitative was more highly associated with outcome y than with outcome x? If so, there would be empirical support for use of each separate dimen- sion to guide adult education policy and the activities of adult educators. If not, either the assessments do not measure the dimensions independently, or there is little practical significance to the distinctions among them. Second, were some dimensions of literacy more highly related to the social and economic characteristics than others? For example, is prose the most important type of literacy, or are document and quantitative equally important? The answer to the second question is both simpler and more important if the answer to the first is that the dimensions of literacy are not associated differentially with their correlates. That is, if one weighted com- bination of the prose, document, and quantitative scores adequately de- scribes the relationship of measured literacy to the several possible corre- lates, the weights of prose, document, and quantitative become more instructive. These analyses were based upon the national and state household samples from the 1992 NALS.1 The total sample size was 25,987. The possible literacy correlates used in the analysis were similar to those used in the panelâs search for break points in the distributions of literacy scores (as described in Chapter 4): â¢ years of school completed, â¢ immigration within the last five years, â¢ reporting at least one health impairment, â¢ reporting a health problem that limits work, â¢ reporting not reading well, â¢ voting within the last five years, â¢ being in the labor force, â¢ weekly earnings (log), â¢ reporting never reading newspaper, â¢ reporting reading no books, â¢ working in an occupation with high formal training requirements (professional, technical, managerial, nonretail sales), â¢ working in an occupation with low formal training requirements (skilled worker, semi-skilled worker, labor, service work, farm work), â¢ using Food Stamps within the past year, â¢ having interest income in the past year, â¢ reporting use of reading on the job, â¢ reporting help needed with written material, and â¢ reporting use of math on the job. 1Thus, the sample of incarcerated persons was not included.

218 APPENDIX B The prose, document, and quantitative scores used were the first set of plausible values in the public release of the data (http://www.nces.ed.gov/ naal/analysis/resources.asp). The analyses were based on a multiple-indicator, multiple-cause (MIMIC) model (Hauser and Goldberger 1971; Joreskog and Goldberger 1975), estimated by maximum likelihood. Rather than estimating separate regressions, one for each of the correlates of prose, document, and quanti- tative, the model posits that there is one linear composite of prose, docu- ment, and quantitativeâmuch like the predicted values in a single regres- sion equationâand that the statistical effects of prose, document, and quantitative on the correlates are completely described by the relationships of the correlates with the composite variable. An equivalent way of describ- ing the model is that the statistical effects of prose, document, and quanti- tative on each of the correlates are in the same proportion. This model was estimated in the total household sample of NALS and in groups defined by race-ethnicity (black, Hispanic), gender, and age (16- 29, 20-44, 45-59, and 60 and older). The constrained model of the effects of prose, document, and quantitative on literacy correlates never fits statis- tically. This is to be expected because the sample is so large that any deviation from the model, no matter how trivial in substance, is statistically reliable. However, the actual deviations of the data from the constraints of the model are neither large nor numerous.2 Typical deviations from the model are that (a) using mathematics on the job is more highly correlated with quantitative literacy, (b) voting within the past five years is less highly correlated with document literacy, and (c) earnings are more highly corre- lated with quantitative literacy. Nevertheless the model provides a useful framework for assessing the relative importance of prose, document, and quantitative. As shown in summary in Table B-5, estimates from the constrained model are roughly similar across all of the groups. For fully constrained models, the left hand panel shows the effects of each of the dimensions of literacy. Prose, document, and quantitative are in the score metric, and the coefficients show effects on grouped levels of educational attainment.3 The right hand panel shows corresponding standardized coefficients. That is, 2For example, in the total sample, the model yields a likelihood ratio fit statistic of 2093.5 with 32 degrees of freedom, but the adjusted goodness of fit index (AGFI) is 0.949, a value that is commonly regarded as acceptable. 3The choice of educational attainment as the outcome variable is completely arbitrary. Any of the correlates could have been used because the effects of P, D, and Q on each outcome are in the same proportion in the constrained model.

TABLE B-5 Constrained Associations of Literacy Dimensions with Life Outcomes Test Score Coefficients Standardized Coefficients Population Prose Doc. Quant. Prose Doc. Quant. Total 6.907 1.378 4.766 0.532 0.107 0.394 (0.170) (0.179) (0.151) (0.013) (0.014) (0.012) Black 6.932 0.751 4.943 0.548 0.060 0.432 (0.167) (0.175) (0.139) (0.013) (0.014) (0.012) Hispanic 4.601 2.185 4.349 0.417 0.200 0.414 (0.117) (0.135) (0.116) (0.011) (0.012) (0.011) Female 7.404 1.386 4.131 0.574 0.110 0.347 (0.173) (0.176) (0.151) (0.013) (0.014) (0.013) Male 6.595 1.129 5.436 0.506 0.085 0.441 (0.171) (0.183) (0.153) (0.013) (0.014) (0.012) Ages 16-29 6.038 1.089 2.748 0.625 0.113 0.295 (0.148) (0.155) (0.132) (0.015) (0.016) (0.014) Ages 30-44 7.999 1.095 4.227 0.610 0.083 0.335 (0.172) (0.185) (0.156) (0.013) (0.014) (0.012) Ages 45-59 8.611 0.880 5.329 0.587 0.059 0.381 (0.187) (0.211) (0.173) (0.013) (0.014) (0.012) Ages 60-99 5.899 3.310 4.353 0.436 0.236 0.382 (0.151) (0.167) (0.128) (0.011) (0.012) (0.011) NOTE: Standard errors are in parentheses. 219

220 APPENDIX B the variables are all expressed in standard-deviation units. All three types of literacy have statistically significant associations with the correlates. The effect of document literacy is much less than that of prose or quantitative literacy and the effect of prose literacy is slightly larger than that of quanti- tative literacy. Model fit deteriorates markedly, however, if the effect of either document or quantitative literacy is ignored. Thus, while the panel notes the apparently prime importance of prose literacy, the other dimensions should not be ignored, and for some purposes it may be useful to construct a composite of the three literacy scores. It is not clear how to interpret the separate effects of the three literacy dimen- sions because they are so highly confounded by design in NALS and NAAL. That is, as long as the same task yields items scored on multiple dimensions, prose, document, and quantitative scores are intrinsically confounded. REFERENCES Hauser, R.M., and Goldberger, A.S, (1971). The treatment of unobservable variables in path analysis. In H.L. Costner (Ed.), Sociological methodology 1971 (pp. 81-117). San Fran- cisco: Jossey-Bass. Joreskog, KG., and Goldberger, A.S. (1975). Estimation of a model with multiple indicators and multiple causes of a single latent variable. Journal of the American Statistical Asso- ciation, 70, 631-639.