Youth Employment and Training Programs The YEDPA Years (1985) / Chapter Skim
Currently Skimming:
Appendix A: Standardized Data Collection for Large-Scale Program Evaluation: An Assessment of the YEDPA-SAS Experience
Appendix A: Standardized Data Collection for Large-Scale Program Evaluation: An Assessment of the YEDPA-SAS Experience
Pages 193-219
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 193... ...
And so, a key element of YEDPA's knowledge development strategy was the establishment of a standardized system for the systematic collection of data on the progress of program participants and the services provided by YEDPA programs. ~_ , _ ~ STANDARDI ZED ASSESSMENT SYSTEM In order "to document administrative outcomes, to monitor performance, and to continually assess program impacts and lessons" from YEDPA programs, the Office of Youth Programs launched a large-scale data gathering operation in collaboration with the Educational Testing Service (ETS)
|
From page 194... ...
' a battery of seven measures of occupational knowledge, attitudes, and related skills administered preprogram and postprogram; a program completion interview; interviews at 3 and 8 months postprogram; and evaluations by counselors (postprogram) and employers or work supervisors (postprogram and 3 and 8 months later)
|
From page 195... ...
Those locator tests are short reading-comprehension measures ordinarily used as screening devices for deciding which level of the full STEP achievement tests is suitable for administration. Job Knowledge and Attitudes Battery Measures chosen for incorporation in the Job Knowledge and Attitudes battery were intended to reflect YEDPA program objectives while still being compatible with the characteristics of the trainee population and the operational constraints of the youth projects.
|
From page 198... ...
The inventory contained 16 items that provided both a total score and scores for three subscales defined as "Optimism," "Self-Confidence," and "Unsocialized Attitudes." The response to each of the attitudinal statements was based on a four-point scale of degree of agreement with, or applicability of, the statement. Job Holding Skills Scale This scale dealt with respondent awareness of appropriate on-the-job behaviors in situations involving interaction with supervisors and coworkers.
|
From page 199... ...
Two questionnaires were used for this purpose: the "Program Completion Survey" and the "Program Follow-up Survey." (The same instrument was used 3 and 8 months postprogram.) Program Completion Survey This questionnaire contained 48 items, most of which were phrased as questions to be presented to the youth at the time he or she had completed or was leaving the training program.
|
From page 200... ...
Since one of the canons of testing is that the manner of test administration can have important effects on measurement, it is natural that concerns about the reliability and validity of the SAS measurements were voiced by outsiders -- as well as by ETS. Almost all of the SAS scales used previously published tests, and there did exist a literature that documented the characteristics of the scales and estimated their reliability and predictive validity with various populations.
|
From page 201... ...
. The other two scales, Job Holding and Self-Esteem, do not show significant associations with postprogram employment, but do show positive associations with evaluations given by guidance counselors and work training supervisors.
|
From page 202... ...
.24 Remedial skills instructor rating OIC (134) .18 SOURCE: Educational Testing Service (1980' CEIARACTERISTICS OF THE DATA BASE Completeness of Initial Coverage According to ETS, the Standardized Assessment System was designed to provide a complete enumeration of all participants (together with appropriate controls)
|
From page 203... ...
ETS contracted to process the data supplied by the program operators (and, in a number of cases, to analyze that data) · 2 Indeed, most ETS discussions of the SAS data base contain forceful disclaimers that "collection of all data with the Standardized Assessment System instruments remained the sole responsibility of the service delivery agents who were required to assign suitable staff at each project site for carrying out the data gathering tasks" (ETS, 1982:15, emphasis in original)
|
From page 204... ...
Incompleteness affects not only participant data, but also the process data. Analysis of the process data collected from individual program sites (shown in Table A.3)
|
From page 205... ...
(It is, of course, a leap of faith to assume that this same percentage would apply to the sites that did not provide process data, but it is not inconsistent with the informal "guesstimates" made by ETS personnel who were familiar with the SAS data collection.) Sample Attrition Subsequent to the initial data gathering, a series of data collection steps were planned for each participant (and control)
|
From page 206... ...
reveals that at 8 months after program completion, there is no interview data on the majority of program participants and controls (see Table Ado. Even at 3 months postprogram, the attrition losses amount to almost half of the original sample (45 percent of program participants and 49 percent of controls who provided initial data did not provide interview data at 3 months postprogram)
|
From page 208... ...
if the attrition rates for its follow-up measurements had been accurately anticipated. To illustrate the cumulative impact of the incompleteness of coverage discussed previously and the sample attrition that occurred over time in the SAS data base, Figure A.2 graphs an estimate of the size of the target sample of participants for the SAS data collection (using the 43 percent coverage rate computed for sites reporting process data)
|
From page 209... ...
Interview Tests Tests Interview Interview Sample coverage and attrition (Standardized Assessment employment. Since these are relatively inexpensive data to collect, there is some reason to favor such a strategy -- particularly if one suspects that the effects of training on employment may be unusually subtle or delayed in arriving.
|
From page 210... ...
. Since the Standardized Assessment System was launched with some expressed trepidations about the suitability of such measures to the YEDPA population, it is important to seek evidence within the data base as to whether these conditions are met by the SAS measurements.
|
From page 211... ...
Control 14.5 14.6 14.9 4.6 NOTE: Standard deviation of scale is computed from data for all controls and participants. SOURCE: Derived by tabulating data for every fifth record in ETS data base, i.e., 20 percent subsample.
|
From page 212... ...
Correlation with STEP Reading Score SAS Measurementa Participant Control Vocational attitudes .445 .509 Job knowledge .447 .467 Job holding skill .288 .354 Work-related attitudes .446 .383 Job search skill .569 .578 Sex stereotyping - .279 .241 (N)
|
From page 214... ...
. Correlation with Activity Statusa SAS Measurement at 3 Months8 Months Program Completion SexPostprogramPostprogram Vocational attitudes Male(.048)
|
From page 215... ...
The ability to read a job advertisement is an essential component of "job search skills." It is not, however, the case that such a simple argument can be made to defend these correlations in every instance. There is no prima facie case to be made for a correlation between the attitude measures and reading-although there are more than enough plausible paths for indirect causation to account for this correlation.
|
From page 216... ...
The correlations for the SAS data base are considerably below those found for the NYC and OIC samples reported by ETS in their 1980 report on SAS (Educational Testing Service, 1980~. They are even lower than the correlations (.10)
|
From page 217... ...
Here again, we selected 10 sites at random from the SAS data base. This time, however, we restricted our analysis to sites that had a minimum of 100 program participants from whom data had been obtained at 3 months postprogram.
|
From page 218... ...
218 lo o U]
|
From page 219... ...
. REFERENCES Educational Testing Service 1980 The Standardized Assessment System for Youth Demonstration Projects.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.