National Academies Press: OpenBook

A Database for a Changing Economy: Review of the Occupational Information Network (O*NET) (2010)

Chapter: Appendix A: Dissent--Juan I. Sanchez and David H. Autor

« Previous: Appendixes
Suggested Citation:"Appendix A: Dissent--Juan I. Sanchez and David H. Autor." National Research Council. 2010. A Database for a Changing Economy: Review of the Occupational Information Network (O*NET). Washington, DC: The National Academies Press. doi: 10.17226/12814.
×

Appendix A
Dissent

Juan I. Sanchez and David H. Autor

ISSUES OF CONTENTION

We believe the report could have provided more expedited solutions on three issues: (1) deletion of scales, (2) weaknesses of descriptors and scales, and (3) whether information on some content domains could be eliminated. In our opinion, none of the recommendations in the report acknowledges that these concerns can be at least partly addressed through immediate modification of the O*NET measurement model without diminishing the utility of the overall database; we believe that such modifications would be supported by the extant O*NET data and do not need to await further research.

ARGUMENTS IN SUPPORT OF THE MINORITY VIEW

We believe the latest O*NET 14.0 database provides empirical evidence of redundancy of descriptors between scales and also within and across domains. First, the data strongly suggest that the importance and level scales are highly redundant. Indeed, their bivariate correlations computed across the various items in each domain and across the 832 occupations included in the 14.0 database are as follows:

Domain

Importance × Level Pearson Correlation

Generalized Work Activities

.92**

Abilities

.97**

Skills

.95**

Interests

.97**

Knowledge

.97**

** = p < .01

Suggested Citation:"Appendix A: Dissent--Juan I. Sanchez and David H. Autor." National Research Council. 2010. A Database for a Changing Economy: Review of the Occupational Information Network (O*NET). Washington, DC: The National Academies Press. doi: 10.17226/12814.
×

These correlations suggest that the ranking of items does not change for practical purposes regardless of which scale, importance or level, is employed. Although correlations between the importance and the level scale might be smaller if individual-level data were used, such data are not made available to the public and, therefore, O*NET users rely solely on the aggregate ratings (i.e., average ratings across approximately 25 respondents) extracted from the O*NET 14.0 database for these analyses.

An analysis of variance components using the same database supported the finding that the type of scale, namely level or importance, has practically negligible effects (3 percent or less variance) on the ratings:

Descriptor

Percentage of Variance Accounted for by Scale (i.e., importance versus level)

Generalized Work Activities

.50

Abilities

3.00

Skills

1.54

Interests

1.32

Knowledge

1.31

Between the two scales, the questionable and often disconcerting behavioral anchors (see Chapter 4) placed at the various points of the level scale strengthen the case for its elimination. The elimination of the level scale will cut more than 150 items from the surveys, thereby cutting survey costs and possibly increasing response rates. Unlike the level scale, other scales (e.g., frequency or duration) may provide independent, valuable, and incremental occupational information above and beyond the information provided by the importance scale, and their potential inclusion warrants further cost-benefit analysis.

We believe there are compelling reasons for at least the temporary suspension of the procedure currently employed to measure the ability and the skill domains, which are rated by trained analysts on the basis of a methodically assembled yet paper-based description of the job. First, these analysts do not have a chance to interview or observe actual occupational incumbents to help them formulate their ratings. The evidence indicating adequate interrater reliability among analysts suggests that they consistently rate abilities and skills, but interrater agreement does not imply validity. Second, a factor analysis of the ability ratings in the 14.0 database confirms the presence of substantial data redundancy among the ratings of the 52 abilities included in this particular domain. That is, a single factor accounts for 43 percent of the variance in ability ratings. There is also quite a bit of empirical redundancy between the two domains currently populated by analyst ratings, namely the ability and skill domains, on one hand, and

Suggested Citation:"Appendix A: Dissent--Juan I. Sanchez and David H. Autor." National Research Council. 2010. A Database for a Changing Economy: Review of the Occupational Information Network (O*NET). Washington, DC: The National Academies Press. doi: 10.17226/12814.
×

the domains of generalized work activities and work context populated by incumbent ratings, on the other. That is, a statistical regression of any ability or any skill rating on the set of generalized work activities and work context ratings reveals statistically reliable multiple R correlations ranging from .65 to .98, p < .01, even after correcting for shrinkage. Therefore, analyst-based ability and skill ratings can be reliably predicted using simple linear combinations of incumbent-based ratings in other domains.

Eliminating analyst-based ratings of abilities and skills would cut almost 90 additional items from the surveys, hence lowering data collection costs and possibly increasing response rates. However, current uses of skill and ability ratings would not need to be disrupted until a better measurement procedure to estimate these domains is developed. Indeed, current users could resort to the mechanical estimates based on incumbent ratings from other domains, which provide practically equivalent values (as mentioned, Pearson R correlations between analyst ratings and mechanical estimates range from .65 to .98). In future waves of data collection, these estimates could be automatically computed and added to the O*NET database in lieu of the analyst-based ratings.

Suggested Citation:"Appendix A: Dissent--Juan I. Sanchez and David H. Autor." National Research Council. 2010. A Database for a Changing Economy: Review of the Occupational Information Network (O*NET). Washington, DC: The National Academies Press. doi: 10.17226/12814.
×

This page intentionally left blank.

Suggested Citation:"Appendix A: Dissent--Juan I. Sanchez and David H. Autor." National Research Council. 2010. A Database for a Changing Economy: Review of the Occupational Information Network (O*NET). Washington, DC: The National Academies Press. doi: 10.17226/12814.
×
Page 195
Suggested Citation:"Appendix A: Dissent--Juan I. Sanchez and David H. Autor." National Research Council. 2010. A Database for a Changing Economy: Review of the Occupational Information Network (O*NET). Washington, DC: The National Academies Press. doi: 10.17226/12814.
×
Page 196
Suggested Citation:"Appendix A: Dissent--Juan I. Sanchez and David H. Autor." National Research Council. 2010. A Database for a Changing Economy: Review of the Occupational Information Network (O*NET). Washington, DC: The National Academies Press. doi: 10.17226/12814.
×
Page 197
Suggested Citation:"Appendix A: Dissent--Juan I. Sanchez and David H. Autor." National Research Council. 2010. A Database for a Changing Economy: Review of the Occupational Information Network (O*NET). Washington, DC: The National Academies Press. doi: 10.17226/12814.
×
Page 198
Next: Appendix B: Descriptor Taxonomies Included in the Content Model »
A Database for a Changing Economy: Review of the Occupational Information Network (O*NET) Get This Book
×
Buy Paperback | $60.00 Buy Ebook | $47.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Information about the characteristics of jobs and the individuals who fill them is valuable for career guidance, reemployment counseling, workforce development, human resource management, and other purposes. To meet these needs, the U.S. Department of Labor (DOL) in 1998 launched the Occupational Information Network (O*NET), which consists of a content model--a framework for organizing occupational data--and an electronic database. The O*NET content model includes hundreds of descriptors of work and workers organized into domains, such as skills, knowledge, and work activities. Data are collected using a classification system that organizes job titles into 1,102 occupations.

The National Center for O*NET Development (the O*NET Center) continually collects data related to these occupations. In 2008, DOL requested the National Academies to review O*NET and consider its future directions. In response, the present volume inventories and evaluates the uses of O*NET; explores the linkage of O*NET with the Standard Occupational Classification System and other data sets; and identifies ways to improve O*NET, particularly in the areas of cost-effectiveness, efficiency, and currency.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!