The committee’s review of the Abt report is organized around the four objectives listed in the Statement of Task: (1) suitability of the database(s) utilized, (2) soundness of the methodology to connect occupational tasks to social interactive and adaptive functional capacities, (3) appropriateness of the expertise gathered to perform the analyses, and (4) overall confidence in the report. Each objective is restated in a box at the start of the corresponding section of the committee’s review.
In Phase 1 of its work, Abt conducted an environmental scan to determine the most appropriate extant data source to inform its ratings of social interactive and adaptive functional capacities in relationship to essential job tasks and specific vocational preparation (SVP) requirements of 134 occupations in the national economy. The data sources considered were nationally representative and ideally would include (1) measures of required tasks and functional capacity of occupations and (2) related Standard Occupational Classification (SOC) codes and SVP ratings for the 134 specific occupations specified by the Social Security Administration (SSA) (see Abt Associates, 2020, p. 7). Ultimately Abt decided to use the Department of Labor’s Occupational Information Network (O*NET) database because none of the other data sources included sufficient detail on required tasks or required functional capacity for a given occupation. O*NET is the nation’s contemporary and most comprehensive source of occupational information. The O*NET database, contains standardized and occupation-specific descriptors of nearly 1,000 occupations across the U.S. economy. The database1 is periodically updated from a broad range of workers in each occupation.
1.1–1.2 Was O*NET an appropriate data source to use to identify occupations’ core tasks and to inform the ratings?
Based on its familiarity with the national surveys reviewed by Abt (see Box 1), the committee agreed with the conclusion that O*NET was an appropriate database to use to identify core tasks of occupations and inform the SVP ratings because it is the only nationally representative, modern data source with sufficient detail to address the objectives articulated by SSA. In particular, it was the only data source reviewed that included task specificity and functional requirements of occupations coded by SOC.
However, the committee had questions about how Abt conducted the environmental scan and the criteria Abt used for the selection of data sources. The committee was not able to discern how Abt identified data sources for potential consideration, how it optimized inclusion of all possible viable data sources, and how it addressed potential selection bias. Aside from the criteria mentioned previously, there is little detail provided about why Abt selected the eight potential data sources listed in Exhibit 2-1 of the Abt report (Abt Associates, 2020, p. 7; see also Box 1), whether any potential data sources were excluded, and if so, the rationale for exclusion. A clearer and more comprehensive description of this process would have been helpful.
1.3 Were there other data sources that would be better to inform the ratings?
The O*NET does have limitations. Specifically, it does not map tasks to SVP levels, nor does it relate tasks to the functional requirements of those tasks. Furthermore, it uses generalized and specific work activities to summarize tasks performed across more than one occupation. Thus, in some cases, task descriptions may not be sufficiently specific to an organization. The O*NET also does not include ratings of adaptation and social interaction, the specific functional requirements of interest to SSA. However, the committee is not aware of any other dataset that would better inform the ratings.
In Phases 2–4, Abt worked with a panel of experts (working group) that it assembled to generate ratings of social interaction and adaptability requirements for occupations within a given SVP range (see Box 2). Abt developed and provided the expert work group with the following information:
- A list of tasks in each occupation, their importance, and the distribution of the required frequency for these tasks,
- SVPs for each occupation,
- Summary of the latent modeling results (see Box 3), and
- Guidance documents from SSA
The Abt analytic staff assigned tasks in each occupation to a level of SVP, and for each SVP it rated the frequency of required work interactions (e.g., basic, verbal, with general public). Tasks were then rated for functional capacity for adaptation and four areas of social interaction (as defined by SSA) by the expert working group (see Box 2). These ratings were then rolled up to occupational level. Abt carried out this work over the course of nine meetings during which it worked with the experts to ensure continuity of the ratings process and common understanding of SSA’s guidance on rating.
2.1 Did Abt provide the expert work group with sufficient guidance and information to complete its task? Was this an appropriate data source to use to inform the ratings?
It is clear that Abt’s data preparation was thorough, but there is a lack of detail about how the information provided to the experts by Abt may have influenced the expert group’s ratings (see Box 2). Appendix D of the Abt report focuses on the definitions of terms provided to the expert work group (e.g., social interaction and adaptability) and summarizes the guidance provided by SSA to Abt that was used to inform the deliberations of the expert panel. It would have been helpful, however, to include an example of one of the worksheets that the Abt analytic team prepared that provided the “main touch point to guide discussion” (Abt Associates, 2020, p. 13). Reported results focus on the ratings of the expert panel and the process used to generate these ratings. However, the report does not describe how the results of the alternative approach, i.e., the latent factor modeling (see Box 3), compared to the expert group’s ratings. Since considerable work was devoted to the latent factor model, it would have been informative to understand how the modeling results compared with those of the panel to provide support for the panel’s ratings or to highlight weaknesses in either of the approaches. In the end, the expert group did not appear to rely on the modeling results as part of its process, so the purpose that was served by the modeling is unclear. The report states that the experts did not rely on the latent factor modeling results, but rather used the descriptive information on tasks to form their own expert opinions.
Despite these concerns, the committee determined Abt’s data preparation process to be sound and thorough. The materials it created were useful to inform the expert panel, even if the expert work group did not rely on them heavily in some instances. Examples include core task identification (Abt Associates, 2020, p. 8, Exhibit 3-1), task frequency (Abt Associates, 2020, p. 9, Exhibit 3-2) and SVP/occupation grouping (Abt Associates, 2020, p. 10).
2.2 Did the expert work group use the O*NET data constructs appropriately and in a manner consistent with the limitations of the data?
The committee agreed that the expert work group used the O*NET data constructs appropriately and in a manner consistent with the limitations of the data. In particular, the O*NET does not map tasks to SVP levels, nor does it relate tasks to the associated functional requirements of those tasks (see Box 4). Therefore, Abt needed to create initial linkages to inform the remaining linkages which required judgment by content experts. For instance, Abt assembled information on essential tasks in each occupation, SVPs for each occupation, and ratings for each occupation for each of three SVP ranges on five categories of mental function. Then, the expert panelists rated required frequency of interactions across jobs, rated tasks by functional capacity, and determined the minimum functional capacity for jobs and occupations. The narrative description of this work, which begins on page 13 of the Abt report, is detailed and flows logically, although an appendix with additional information in the form of a flow chart would have been useful. The experts used O*NET for a list of tasks and their frequency distribution, and these were used to make ratings.
The committee had questions about why Abt chose to incorporate some aspects of O*NET and not others in its work. For example, the committee questioned whether the “abilities” descriptions from O*NET could have been used to characterize mental job requirements more thoroughly. A few of these descriptions are particularly relevant to the examination of social interaction and adaption. Thus, the committee questioned whether these descriptions could have further informed the expert panel beyond their use in the latent modeling, which ultimately was not used in the expert panel’s process.
2.3–2.4 Was Abt’s process for obtaining consensus ratings appropriate and was the meeting process effective to provide ratings?
The committee agreed that Abt’s process for obtaining consensus ratings, which was described in considerable detail in the report, was appropriate. A statistical model of individual ratings could have been employed to predict consensus ratings, and such evidence could have shed light on the extent to which, for instance, some members of the panel were more influential than others, or whether certain experts were more influential in specific judgments. Despite the potential limitations due to a lack of empirical checks on the consensus process, the iterative nature of the work group’s approach ensured consistency through the repetition of the rating process, both during the project and in the final meetings (see Box 5). It also implemented a process for dealing with conflicting ratings, should they occur. For these same reasons, the committee concluded that Abt’s overall meeting process was effective to provide the ratings.
2.5 Considering the instructions and guidance provided by SSA, did Abt’s methodology provide a sufficient nexus to its findings?
The committee understood question 2.5 to mean: Considering the guidance provided by SSA, did Abt apply sound methods that addressed SSA’s objectives and supported the study findings?
Based on the detailed instructions and guidance from SSA, the committee concluded that Abt’s methodology was step-wise and systematic, addressed the stated objectives, and supported the study findings. First, Abt identified an appropriate data source for the study, although some relevant details about the environmental scan were not provided in the report. Next, it prepared a substantial amount of data in advance of the expert committee meetings to inform the work group’s deliberation process. For instance, core tasks for each of the 134 specified occupations were rated by importance and relevance. The frequency with which core tasks were performed was identified. SVPs were collapsed and each occupation was rated for each of three SVP ranges on five categories of mental function (reflecting adaptation and work interactions). Complex statistical modeling was performed to characterize the latent trait of “Adaptation” and tie it to job tasks. All of these materials were summarized in tables and provided to the expert panel to inform its ability to (1) assign tasks in each occupation to a level of SVP, (2) rate the frequency of required work interactions, (3) rate the five aspects of mental functional capacity as defined by SSA, (4) determine the minimum required functional capacity for each occupation across these categories, and (5) collapse these rating to the occupational level. The materials in Appendix D
of the Abt report operationalize SSA’s thinking about interactive and adaptive aspects of work demands. Abt’s data preparation process is reflected, in part, in Exhibits 3-1 and 3-2 of the Abt report (Abt Associates, 2020, pp. 8–9). The work of the expert panel is, in part, reflected in Appendix B, Exhibit 5-1 (Abt Associates, 2020, pp. 21–37), Appendix A, and Exhibit 5-2 of the Abt Report (Abt Associates, 2020, pp. 38–47).
The expert work group recruited by Abt (see Box 6) included eight work group members whose backgrounds included vocational rehabilitation, nursing, labor economics, psychology, physical therapy, and occupational therapy. In addition to the expert work group members, two advisory group members joined the first five meetings to contribute to the initial ratings of tasks and mapping them to occupational requirements. These advisory group members had backgrounds in vocational and psychiatric rehabilitation.
3.1 Did the composition of the expert work group include appropriate expertise?
The committee agreed that the list of experts compiled by Abt (see Box 6) included the appropriate expertise for the report.
The expert group included individuals with backgrounds in nursing, occupational therapy, physical therapy, rehabilitation counseling, and social work, reflecting professions with expertise in function, occupational requirements, and their intersection. The committee noted that physicians were not included on the expert panel. Because physicians in occupational medicine specifically evaluate injury and its impact on function in relationship to work requirements, the
inclusion of a representative with this disciplinary expertise may have added to the panel’s composition. The committee concluded that there were no additional highly relevant professional fields that should be included.
4.1 Were there any likely sources of bias or systematic error resulting from the data or methodology?
The committee defined bias as a systematic error leading to an incorrect estimate of effect or association, and systematic error as an inaccuracy in a measurement or system that is not introduced by chance. The committee concluded that the Abt report does not demonstrate any apparent sources of systematic bias or error. It acknowledged sources of potential error such as the data aggregation methods (see Box 7) and that the ratings of core tasks are not entirely straightforward (Abt Associates, 2020, p. 16). In addition the absence of reliability and validity checks in the consensus methodology could have contributed to potential error, but the committee concluded that any potential errors introduced by these issues are unlikely to be systematic. Additional examples of potential sources of error would have helped enhance confidence in the committee’s assessment of this issue.
4.2 Was it clear how Abt and the expert work group reached its conclusions?
The committee agreed the report was clear about the way in which Abt and the expert work group reached their conclusions. Section 4, Narrative of Expert Work Group Process (Abt Associates, 2020, pp. 13–20), describes in detail how the work group reached its consensus, and the approach seems reasonable.
4.3 Did Abt present its methodologies clearly?
In terms of Abt’s overall methodology, the committee concluded that while the approach was sound, it could have been presented more clearly in the report. For instance, the report used different terms interchangeably, such as required functional abilities, mental function, and functional capacity, when referring to social interaction and adaptation, which is how these concepts were operationalized for the purpose of this study. Similarly, the terms core tasks and essential tasks were used interchangeably. This made it difficult to follow the logic of the workflow without re-reading the report several times.
In addition, although it is clear that SSA specified many of the parameters of this study, it would have been helpful for Abt to have articulated the rationale for these parameters, such as the need to collapse SVP categories and then link them to essential job tasks. Similarly, articulation of the rationale for operationalizing mental function by the five variables noted in the report (i.e., 1. whether adaptation was required, 2. whether complex work interactions were required or not, 3. if so, how frequently, 4. the frequency of verbal work interactions, and 5. the nature of interactions with the public) would have been helpful since it significantly narrows the concept of mental function applied to this study.
A flow chart or graphic depicting the essential study tasks would have helped clarify the study process, particularly in how ratings were assigned. Inclusion in the report of the materials that the expert work group were given would have helped facilitate the readers’ understanding of Phases 2–4. Additional detail would have been useful in the description of Phase 1, in particular the various crosswalks that were conducted to inform the expert panel.
Finally, details about how the latent modeling was conducted were not clear and the results were not presented or interpreted by Abt. The description of the latent model made it difficult to understand how those results informed the study, particularly because the expert group did not seem to utilize the modeling in its work. Additional appendices expanding on the latent model and environmental scan would have been useful, so that these aspects of the process could be understood better. Inclusion of terminology definitions (a glossary) would have allowed faster and more accurate assimilation of the report. The committee noted that O*NET is complex so the process of linking elements is complex, and Abt likely optimized this process.
Overall, the committee concluded that Abt utilized the most appropriate data resources available and selected an expert work group with necessary and relevant expertise. In addition, while the committee noted that further explanation and details would have been helpful, the committee concluded that both the methodology and meeting process Abt used were rigorous and sound.
Abt Associates. 2020. Synthesizing information about vocational preparation requirements, occupational tasks, and required functional abilities in the Standard Occupational Classification (SOC) System: Final report. Rockville, MD.