A Database for a Changing Economy Review of the Occupational Information Network (O*NET) (2010) / Chapter Skim
Currently Skimming:
4 The Data Collection Program
4 The Data Collection Program
Pages 61-92
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 61... ...
NET content model was complex, including many hundreds of descriptors organized into 10 domains, also referred to as taxonomies. The final stage in the prototype development project was a field test of the content model using real-world occupational data.
|
From page 62... ...
. The current rating scales used by trained occupational analysts to assess the importance and level of Abilities descriptors are largely unchanged from those developed in the prototype development project.
|
From page 63... ...
Figure 4-1 Work Styles For the Work Styles domain, the O* NET prototype development team developed rating scales that were similar to those for the Abilities taxonomy, including a 5-point scale for importance and a 7-point rating scale for the level of the Work Style believed to be required for a particular occupation.
|
From page 64... ...
NET Development does not routinely collect data on Work Values as part of its main data collection program but does periodically develop data for this domain. knowledge As was the case for Abilities and Work Styles, the prototype rating scales were designed to help individuals rate each Knowledge descriptor in terms of its importance for performance in a particular job and the level of the descriptor needed "to perform the job." Again, the prototype development team calibrated the level scale with behavioral anchors.
|
From page 65... ...
Figure 4-2 Occupational Preparation The prototype development team did not develop rating scales to gather information on the level or importance of occupational preparation. Instead, the prototype questionnaire included seven questions related to education, training, licensure/certification, and experience requirements.
|
From page 66... ...
Organizational Context The prototype development team created a complex taxonomy of Organizational Context descriptors along with rating scales to assess levels of these descriptors required to perform various occupations. Currently, the O*
|
From page 67... ...
Prototype Data Collection: Applying the Rating Scales In the field test, the prototype development team used the rating scales to obtain information about the level and importance of each descriptor for each of the 1,122 occupational units from two groups of individuals -- trained occupational analysts and job incumbents. Before these field tests began, the project team believed that job incumbents, employed in the occupational unit, would be best positioned to rate the level and importance of each descriptor, based on their familiarity with the occupation (Peterson et al., 1999)
|
From page 68... ...
NET prototype field test. Expert Review Expert review is a quick and efficient appraisal process conducted by survey methodologists.
|
From page 69... ...
In cases in which an existing form or publication is being evaluated, an expert reviewer will identify key areas or features of the document that should be addressed in questioning participants. In the context of designing a larger populationbased survey, focus groups are used to elicit important areas that should be addressed by the survey.
|
From page 70... ...
PRETESTING OF THE O* NET PROTOTYPE SuRvEYS Following the disappointing response rates to the surveys of job incumbents used in the field test of the prototype content model, in 1998 the U.S.
|
From page 71... ...
; • The level scale was removed from the Work Styles questionnaire; • Some rating scales were eliminated on the basis of previous data (Peterson et al., 1999) showing them to be highly intercorrelated; • A small number of redundant descriptors were combined or ag gregated into a single descriptor; and • A very small number of descriptors was eliminated because no respondents could understand their meaning.
|
From page 72... ...
Finally, the working group decided to retain two different rating scales for four of the seven taxonomies of descriptors, one to measure the importance of the descriptor and another to measure the level required to perform the occupation. In the panel's view, the revisions made in response to the pilot test
|
From page 73... ...
NET Data Collection Program along any one of these dimensions increases the total cost of data collection; holding constant data collection costs, improvements on one dimension necessitate cutbacks along either or both of the remaining dimensions. As a result of the DOL decision to make few revisions to the content model following the field test, the O*
|
From page 74... ...
NET Data Collection Program Number of Total Number Scales per Data Source Questionnaire Descriptors Descriptor of Scales Skills 35 2 70 Analysts Knowledge 33 2 66 Job incumbents Work Stylesa 16 1 16 Job incumbents Education and Traininga 5 1 5 Job incumbents Generalized Work Activities 41 2 82 Job incumbents Work Context 57 1 57 Job incumbents Abilities 52 2 104 Analysts Tasksb Varies 2 Varies Job incumbents Total (not including Tasks) 239 NA 400 NA NOTES: Occupation experts use the same questionnaires as job incumbents for those occupations whose data collection is by the Occupation Expert Method.
|
From page 75... ...
found that some of the behavioral anchors do not represent a clear continuum of levels of difficulty of a descriptor. As in the example of reading comprehension above, Gustafson and Rose found that some rating scales are consistently biased, placing behaviors in professional domains viewed as high level (including medicine, law, science, and "corporate")
|
From page 76... ...
In addition, they found that some behavioral anchors are inconsistent with -- or only vaguely related to -- the definition of the descriptor being measured, and others conflate learned techniques or skills with high levels of physical abilities. Importance Scales It is clear that some of the behaviorally anchored rating scales used to measure the level of descriptors are problematic, and the rating scales for importance have also been subject to some criticism.
|
From page 77... ...
argued that job incumbents are the most knowledgeable source of information about their own jobs and suggested that the O* NET Center replace the current behaviorally anchored rating scales with different types of questionnaires, including specific questions that he viewed as more understandable to incumbents.
|
From page 78... ...
NET Data Collection Program, which examined various sources and methods for collecting occupational data. The later decision to use analysts to rate Skills was based in part on a study by Tsacoumis and Van Iddekinge (2006)
|
From page 79... ...
Other questions concern how analysts acquire occupational information. For example, how may the streamlining of rating stimulus materials affect the quality of analyst ratings?
|
From page 80... ...
. Although they do not rate the importance and level of the Skills and Abilities descriptors, job incumbents do provide data for 6 other domains (Knowledge, GWAs, Work Context, Education and Training, Work Styles, Tasks)
|
From page 81... ...
NET are that virtually all 5-point scale ratings have a 95 percent confidence interval no wider than +/–1 and that virtually all 7-point scale ratings have confidence intervals no wider than +/–1.5.
|
From page 82... ...
NET Development and RTI (2009a) , the occupational expert method is used when the establishment method would be problematic due to low rates of employment in some occupations, such as new and emerging occupations.
|
From page 83... ...
Steps in the development and analysis seem appropriate. The nonprobability approach for the occupational expert method is distinguished from the probability approach of the establishment method, and it is noted that these estimates should be viewed separately from the establishment method.
|
From page 84... ...
reports relatively high response rates for both the employees and the establishments contacted using the establishment method. However, it is unclear whether the rates reported are response rates or cooperation rates, as defined by the AAPOR.
|
From page 85... ...
A final stage of data cleaning for the O* NET data is defined as the "deviance analysis" for task questionnaire data.
|
From page 86... ...
This mixed-method approach results in the collection of occupational data from different types of respondents (occupational analysts, job incumbents, occupational experts) who may or may not represent the work performed in that occupation.
|
From page 87... ...
In addition, in the scales used to assess the required level of a descriptor, many of the behavioral anchors are taken from occupations that are not familiar to many job incumbent raters. Recommendation: The Department of Labor should, in coordination with research on the content model discussed in Chapter 2,3 and with advice and guidance from the technical advisory board recommended in Chapter 2, conduct a study of the behaviorally anchored rating scales and alternative rating scales.
|
From page 88... ...
The results of these studies should be released for public use. Because the representativeness of the establishment method sample depends partly on the response rates, the O*
|
From page 89... ...
. Effects of descriptor specificity and observability on incumbent work analyst ratings.
|
From page 90... ...
Paper prepared for the Panel to Review the Occupational Information Network (O*
|
From page 91... ...
O* NET data collection program, PDF questionnaires.
|
From page 92... ...
O* NET data collection program, Office of Management and Budget clearance package supporting statement, Volumes I and II.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.