Evaluation Design for Complex Global Initiatives Workshop Summary (2014) / Chapter Skim
Currently Skimming:
3 Framing the Evaluation
3 Framing the Evaluation
Pages 17-28
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 17... ...
Any evaluation effort starts by framing the evaluation. In the workshop's opening session, five panelists discussed various approaches for this key initial step.
|
From page 18... ...
REFLECTIONS FROM THE EXPERIENCE OF THE EVALUATION OF THE U.S. PRESIDENT'S MALARIA INITIATIVE Framing an evaluation starts with a small set of well-defined questions, said workshop planning committee member and panel moderator Jonathon Simon, who is Robert A
|
From page 19... ...
While it was important to maintain operational independence from the funders of the evaluation, Simon and his colleagues often had to rely on the PMI staff to gain the cooperation of the in-country teams. To illustrate that objectivity required a balance between independence and interdependence, Simon explained that while the evaluation team received 167 comments from the funders in response to the draft report, the evaluators chose which comments to address and which to reject.
|
From page 20... ...
"That's one of the trade-offs with these largescale, infrequent evaluations is that they can address high-level issues with a global context, but they cannot drill down as effectively as one might expect," she said. "We would like to see more internal evaluation capacity building so that that can answer specific questions in a more timely basis." To increase utilization of findings, evaluations now start with a 3-month inception phase in which her office holds conversations with potential users, reviews prior evaluations, and attempts to develop a better understanding of how an evaluation can be useful to program staff as well as to the UN as a whole.
|
From page 21... ...
In terms of the the Joint United Nations Program on HIV/AIDS (UNAIDS) program, Rugg said that ongoing internal evaluations are focusing on performance monitoring, while an external, independent evaluation is conducted every 5 years and a variety of ad hoc special evaluation studies focus on specific programmatic issues.
|
From page 22... ...
Another confounding factor is that most of the large, complex programs are conceived by what Whitty characterized as "very smart, very politically connected, and very charismatic true believers." The resulting political realities have to be considered in the initial design discussions between funders and evaluators. On the side of the evaluators, poorly performed evaluations are often the result of the difficulty of evaluating complex programs.
|
From page 23... ...
Nonetheless, the Global Fund has used the results of the evaluation to create a new funding model that has been launched recently, and it has taken steps to address the evaluation's shortcomings by emphasizing continuous smaller country-level evaluations on which to build comprehensive evaluations that can better inform ongoing grant management at the country level. This new model also reduces the logistical challenge of evaluating multiple countries, each with its own operational cycle, simultaneously.
|
From page 24... ...
For PEPFAR, Black emphasized what was in his view an unfortunate trade-off: not being able to report findings that were specific to individual countries, which was due to the framing of the original evaluation mandate and to the necessity of assuring country de-identification to receive secondary quantitative data and to maximize candor in qualitative data collection. As far as independence and objectivity in the PEPFAR study, Black said that the IOM is very strong on avoiding conflict of interest among committee members, and the process was developed and carried out with complete independence, with the sponsoring organization not receiving the report until it was finalized after extensive review by outside experts.
|
From page 25... ...
He said that issues related to funding of the implementation of an evaluation also need to be thought through carefully. "In all of the examples I have seen, funding is linked in some strong way with the program, which to me compromises independence." Black commented that the evaluations he described almost all have some kind of trade-off in their framing and design that limits what kind of evaluation can be done, for example, "in the selection of countries, the design of the implementation, the interpretation, or the control of funding." IMPACT OF AN EVALUATION In the final presentation of the session, Carmela Green-Abate, the PEPFAR coordinator in Ethiopia, discussed how both the recent IOM PEPFAR evaluation and a prior IOM evaluation earlier in the implementa
|
From page 26... ...
She also remarked that the second evaluation emphasized knowledge management, including monitoring evaluation, innovation, and research. This has contributed to a new monitoring and evaluation framework from PEPFAR that is still being rolled out, but an enormous dilemma at the country level is alignment with the countries and the speed of the roll out.
|
From page 27... ...
OTHER TOPICS RAISED IN DISCUSSION Sangeeta Mookherji of George Washington University, asked how the field can ensure that there is independence and objectivity when it comes to analyzing and interpreting data, not just collecting data. Black cited the PEPFAR evaluation as an example where the interpretation and analysis of data was independent, even though some of the data came from the program itself.
|
From page 28... ...
In response to a question from Sir George Alleyne, chancellor of the University of the West Indies and a member of the workshop planning committee, about whether all programs were evaluable, Rugg said that programs such as the UN Development Programme or WHO are evaluated not for the purpose of determining whether they should continue but to make them better, which is a different type of evaluation. Whitty said that evaluability is not a yes/no proposition but a spectrum that ranges from the obvious to the impossible.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.