Community Programs to Promote Youth Development (2002) / Chapter Skim
Currently Skimming:
7 Generating New Information
7 Generating New Information
Pages 197-227
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 197... ...
PART 111 Evaluation and Social Indicator Data
|
From page 198... ...
Many of them tend to be relatively new and are continually changing in response to growing interest and investments on the part of foundations and federal, state, and local policy makers. In addition, the elements of community programs for youth rarely remain stable and consistent over time, given that program staff are always trying to improve the services and the manner in which they are delivered.
|
From page 199... ...
Deciding what questions to ask and the methods to use for the development of a comprehensive evaluation is important to the wide range of stakeholders of community programs for youth.
|
From page 200... ...
These two evaluative approaches can be thought of as a set of assessment options that build on one another, allowing program staff to increase their knowledge about the activities that they undertake as they incorpoiSummative evaluations can be used with such programs to assess impact and estimate its size. Such evaluations are often called for when the government has invested very large sums of money in a novel omnibus social experiment, such as those linked to recent welfare-to-work reform or large-scale school reform efforts.
|
From page 201... ...
Such information can provide very useful information to program staff to help them make changes to improve program effectiveness. And finally, program evaluation can test new or very well-developed 201
|
From page 202... ...
These two evaluative approaches can be thought of as a set of assessment options that build on one another, allowing program staff to increase their knowledge about the activities that they undertake as they incorpoiSummative evaluations can be used with such programs to assess impact and estimate its size. Such evaluations are often called for when the government has invested very large sums of money in a novel omnibus social experiment, such as those linked to recent welfare-to-work reform or large-scale school reform efforts.
|
From page 203... ...
Process and outcome evaluations both rely on the collection of two types of data qualitative and quantitative data (see Box 7-2 for elaboration)
|
From page 204... ...
204 Community Programs to Promote Youth Development EVALUATING COMMUNITY PROGRAMS FOR YOUTH In our review of studies and evaluations that have been conducted on community programs serving youth, we found that a wide range of evaluation methods and study designs are being used, including experimental, quasi-experimental, and nonexperimental methods (see Box 7-3~. Part II provided examples of community programs for youth using both experimental or quasi-experimental methods, as well as a range of other nonexperimental methods of study, including interviews, focus
|
From page 205... ...
Generating New Information 205
|
From page 206... ...
Through consideration of our review of various programs, the basic science of evaluation, and a set of experimental evaluations, quasi-experimental evaluations, and nonexperimental studies of community programs for youth, the committee agreed that no specific evaluation method is well suited to address every important question. Rather, comprehensive evaluation requires asking and answering many questions using a number of different evaluation models.
|
From page 207... ...
Answers to all of these questions may also be obtained by synthesizing the findings from many evaluations conducted with the same organization, program, or element. They may also be answered by using various evaluation methods.
|
From page 208... ...
Complex generic models are needed that involve theorizing at the community, program, cultural, and individual levels because these will help determine the classes of variables to be included in both program and outcome evaluations, and both ongoing reflective practice and program study. Gambone, Connell, and their colleagues developed one such mode!
|
From page 209... ...
. Cook and colleagues' evaluations of the Comer School Intervention provide a good example of careful implementation assessment (Cook et al., 1999; Anson et al., 1991; Cook et al., 2000~.
|
From page 210... ...
The process evaluation to provide a description of implementation quality can be carried out using a wide variety of both qualitative and quantitative data collection techniques. Among the qualitative methods are ethnographic data collection using a time-sampling basis or openended interviews with young people and service practitioners and man2Individuals interested in this issue should consult references on the "intent-to-treat" debates (Shadish et al., 2001; Angrist and Imbrens, 1995)
|
From page 211... ...
It is important to remember that in experimental program evaluations, it is highly desirable if not necessary in theory that the assessment of implementation quality be identical for the treatment and contro! groups (the same is actually also true in other forms of evaluation)
|
From page 212... ...
Various evaluation methods can be used to answer this question, including randomized experiments, quasi-experiments, interrupted treatment designs, theory-based evaluation, and qualitative case studies Randomized Experiments This effectiveness question can be rephrased in the following way: Would any observed changes in the youth exposed to the activities under evaluation have occurred without the program? In other words, would the changes have occurred because of temporal maturation alone, because of selection differences between the kinds of young people exposed to two different sets of experiences that are being contrasted, due to statistical regression, or due to being tested on two different occasions and learning from the first one what to say or write on the second one?
|
From page 213... ...
Nonetheless, they are often clifficult to implement and program staff are often reluctant to participate in experimental evaluations for a variety of reasons ranging from the clifficulty and inconvenience often imposeci by this method to very serious concerns about the ethics of clenying an opportunity or service to a large portion of the aclolescents neecling it. Even so, experiments are the method of choice for evaluating the effectiveness of programs in communities, and program staff need to be part of the ongoing clebates about the feasibility of random assignment, raising their objections and carefully considering the responses macle to these objections.
|
From page 214... ...
Done properly, both experimental and quasi-experimental methods provide quite valid information about program effectiveness. Internal validity is best addressed by the random assignment of subjects to contro!
|
From page 215... ...
At issue is noting whether there is a shift in the abbreviated series' mean or trend following entry into a program, or whether it is a program introducing some new practice that is being evaluated. The value of this approach is that it can rule out alternative interpretations to the program effect that are based on maturational changes in the young people or on statistical regression common in situations in which a program is begun or modified because the situation in a community has suddenly worsened.
|
From page 216... ...
Of course, having more demand for the program than supply also means that a randomized experiment is feasible, and a randomized experimental evaluation is superior to the regression-discontinuity design, even if only because its statistical power is greater. If evaluationary practice is to be improved in the youth development area, there will have to be greater concern for random assignment, regression-discontinuity, interrupted time-series, and quasi-experiments that have strong rather than weak designs.
|
From page 217... ...
If the causal modeling analyses suggest that the obtained data are consistent with what the program theory predicts, then it is presumed that the theory is valid and success of the program has been demonstrated. If time does not permit assessing all the postulated causal links, information on the quality of initial program implementation nonetheless will be gathered because implementation variables are usually the first constructs in the causal mode!
|
From page 218... ...
Of course, such information can be misused, particularly if it is not gathered with rigorous methods; such misuse of information is also possible using quantitative experimental methods. But when done with scientific rigor, particularly if done in conjunction with rigorous experimental and quantitative quasi-experimental studies, qualitative information can provide very important insights into the effectiveness of youth programs.
|
From page 219... ...
The object of study is a program component that repeatedly occurs across many of the organizations currently providing community services to youth; 2. An established national organization provides the program being evaluated through a system of local affiliates; and
|
From page 220... ...
Particular attention here needs to be paid to such individual and community differences as culture, age and maturity, sex, disability status, social class, educational needs, and other available community resources. Comprehensive experimental evaluations are also called for in two other contexts.
|
From page 221... ...
The final context for comprehensive experimental evaluation is when some boici new idea for a new kind of service surfaces and critical examination shows that the substantive theory behind the idea is reasonable and that it is incleeci likely to be able to be implemented. This situation is often called a demonstration project.
|
From page 222... ...
. Supported by a clear understanding of the causal processes underlying program effectiveness, practitioners at new sites can decide how the processes can best be implemented with their unique target population and their unique community characteristics.
|
From page 223... ...
If Effective, How Valuable? If a youth program is found effective, a comprehensive evaluation can then ask: Is it more valuable than other opportunities that could be pursued with the resources devoted to the program?
|
From page 224... ...
. To advance youth program evaluation in this direction will require more rigorous evaluations with adequate follow-up periods and suitable data on a broad set of impacts.
|
From page 225... ...
Nonetheless, when carefully done with attention to the findings' sensitivity to different assumptions, benefit-cost analysis can improve the basis on which youth development policy decisions rest. SUMMARY In this chapter, we reviewed fundamentals of evaluation and important questions for the development of a comprehensive evaluation strategy.
|
From page 226... ...
In order to generate the kind of information about community programs for youth needed to justify large-scale expenditures on programs and to further fundamental understanding of role of community programs in youth development, comprehensive experimental program evaluations should be used when: · the object of study is a program component that repeatedly occurs across many of the organizations currently providing community services to youth; · an established national organization provides the program being evaluated through many local affiliates; and . theoretically sound ideas for a new demonstration program or project emerge, and pilot work indicates that these ideas can be implemented in other contexts.
|
From page 227... ...
When experiments are not called for, a variety of nonexperimental methods and more focused experimental and quasi-experimental studies are ways to unclerstanci and assess these types of community programs for youth and help program planners and program staff build internal knowledge and skills and can highlight theoretical issues about the clevelopmental qualities of programs. Such systematic program stucly shouici be a regular part of program operation.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.