Improving Evaluation of Anticrime Programs (2005) / Chapter Skim
Currently Skimming:
5 How Should the Evaluation Be Implemented?
5 How Should the Evaluation Be Implemented?
Pages 45-53
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 45... ...
Some of the more common areas in which study designs break down in implementation are: · failure to obtain the necessary number of cases to construct treatment and control groups and/or attain sufficient statistical power; · failure to acquire a suitable comparison group in quasi-experimental studies; · attrition, especially when it affects the treatment and control groups differently; · dilution of the service delivery that weakens the program being tested; and · failure to identify essential covariates or obtain measures of them in observational studies. Problems such as these undermine the validity of the conclusions an impact evaluation can support and, if serious, can keep the study from being completed in any useful form.
|
From page 46... ...
Where there is significant uncertainty about the feasibility of an impact evaluation, a two-step contracting process would be advisable, with the first step focusing on developing background information and formulating the evaluation plan and the second step, if warranted, being the implementation of that plan and completion of the evaluation. Funding agencies and evaluators have used a number of approaches to developing the information needed to formulate an instructive RFP or planning the evaluation directly.
|
From page 47... ...
of Multidimensional Treatment Foster Care for delinquents, for instance, both built on preliminary studies conducted before the main evaluation. For complex evaluations, a design advisory group consisting of experts in evaluation methodology and study design might be funded to assist in developing an evaluation plan that is informed by the findings from whatever preliminary studies have been conducted.
|
From page 48... ...
This differential attendance potentially compromised the comparison across conditions because any difference favoring the child-only and family conditions might have been attributed to the greater number of contact hours rather than the content of the program. A preliminary year of funding for piloting study procedures and con ducting pipeline analyses would have strengthened this study by alerting the investigators to the challenges so that they could refine the procedures before the study began.
|
From page 49... ...
Reviewing Evaluation Proposals Knowledgeable reviewers can contribute not only to the selection of sound evaluation proposals but also to improving the methodological quality and potential for successful implementation of those selected. The comments and suggestions of reviewers experienced in designing and implementing impact evaluations may identify weak areas and needed revision in even the highest scoring evaluation proposals under review.
|
From page 50... ...
The review panels at this second stage could then focus solely on the scientific merit and likelihood of successful implementation of the proposed research. For purposes of obtaining careful reviews and sound advice for improving proposals, standing review committees rather than ad hoc ones have much to recommend them.
|
From page 51... ...
Such a fund will be counterproductive, however, if it is not carefully directed toward solvable problems that obstruct what otherwise is a high probability of a successful evaluation project. STEPS THAT CAN BE TAKEN AFTER AWARDING THE EVALUATION CONTRACT The typical grant monitoring process requires periodic reporting by the grantee.
|
From page 52... ...
Vigilance throughout the course of the evaluation project is required to catch such changes. Other mechanisms that can be used to enhance project success after funding include meetings of evaluators of similar projects and cluster conferences for evaluators.
|
From page 53... ...
Agencies might maintain a directory of experienced researchers who could be called upon to consult with grantees as situations arise. Advisory boards are often created for this purpose and may be especially helpful on large and complex projects.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.