Evaluation Design for Complex Global Initiatives Workshop Summary (2014) / Chapter Skim
Currently Skimming:
Appendix D: Evaluation Information Summary for Core Example Initiatives
Appendix D: Evaluation Information Summary for Core Example Initiatives
Pages 149-168
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 149... ...
Appendix D Evaluation Information Summary for Core Example Initiatives 149
|
From page 150... ...
What are the wider effects of the Global Fund partnership on country systems?
|
From page 151... ...
Congress Global Fund The Global Fund Board, PMI, USAID, CDC audiences OGAC to guide its decision on (For whom is PEPFAR implementers whether to "expand, the evaluation (USG agencies, country accelerate, modify, primarily programs, implementing terminate, or suspend the performed based partners) AMFm business line" on the statement of task or terms of reference or the intent communicated by the requestor/ funder?
|
From page 152... ...
other multilateral agencies, stakeholders (bilateral program personnel and USG the evaluation Global HIV stakeholders bilateral and foundation donors, and multilateral agencies, personnel findings (e.g., multilateral advocates) Roll Back Malaria, etc.)
|
From page 153... ...
Three additional consultant members of the Evaluation Team were paid directly by GH Tech outside of the BU contract Geographic scope "Whole of PEPFAR": Whole of Global Fund; scope All 8 Phase 1 pilot 15 PMI focus countries in of the evaluation Scope varied by data with regard to data collection countries (Ghana, Africa How many source, ranging from varied by study area; Kenya, Madagascar, countries/regions all countries receiving 16 countries for SA 2 and 18 Niger, Nigeria, Tanzania were included in PEPFAR funds (100+ countries for SA 3 (mainland) , Uganda, the scope of the countries)
|
From page 154... ...
, with subcontracted the evaluation team, subteams, and paid consultants Macro International was a subcontract to the to BU and three independent subcontractors, for both qualitative and awarded the contract for the London School of consultants and advisory quantitative methods evaluation of all three areas and Hygiene and Tropical mechanisms enlisted a consortium of different Medicine The evaluation team (made Evaluation team was universities and service providers, up of the BU team and the divided into subteams including Johns Hopkins Outlet survey data three consultants) consisted for data collection field University, Harvard School collection was contracted of members with broad visits and into working of Public Health, Washington separately by the Global public health experience, groups by topic area University, Axios International, Fund, without a direct malaria expertise, extensive Development Finance line of accountability experience in the use of both An external panel International (DFI)
|
From page 155... ...
Relationship Independent and Independent and external Independent and Independent and external of evaluator to external external evaluand For example: Independent external contractor; internal independent evaluation unit within the same organization as the evaluand; internally conducted by evaluand 155
|
From page 156... ...
The TERG TERG, AMFm Ad Hoc USAID, and CDC staff the evaluator's and avoid conflicts of organized regular consultative Committee, AMFm members. The Evaluation institution, interest; compliance meetings with the contractor Phase 1 country Team reviewed the oversight by with transparency to assess progress and discuss officials, Expert comments and responded, the evaluand or requirements of the issues Advisory Group, and as they deemed appropriate the evaluand's Federal Advisory key technical partners, advisory bodies Committee Act; prepared the initial external report review evaluation design which was put out to tender OGAC appointed a liaison, the head of AMFm Ad Hoc Strategic Information, Committee met to whom the IOM staff approximately twice could communicate per year and received their needs for process updates and evaluation (i.e., data interim evaluation and documentation findings (e.g., baseline requests, interview results)
|
From page 157... ...
that detailed the independent consultants for with the results of the were incorporated into the reports the strategic approach each of the three study areas secondary analysis of final report as appropriate. to the evaluation and household data on The Evaluation Team had evaluability assessments A synthesis report, based on antimalarial use was full discretion to accept or but no preliminary findings and recommendations published separately reject comments as they findings of the final study area reports, from the main report saw fit (2)
|
From page 158... ...
GOVERNANCE STRUCTURE AND REPORT 158 PEPFAR Global Fund AMFm PMI Authorship Final report authored Final reports for each of the All reports authored by Final report authored and ownership by the convened three study areas, as well as a the IE team; the content by the Evaluation Team; of evaluation committee and issued synthesis report, were authored of the report belongs to content of the report findings/report by the IOM; content of by the independent contractors the IE team belongs to the Evaluation the report belongs to and released by Marco Team the IOM International The TERG released a summary for the synthesis report and each of the three study areas In the summaries for each study area, the TERG provided an "assessment" of and comments on each of the contractor's recommendations Other notable Congressional mandate The TERG emphasized that the While the original terms None noted aspects of for study afforded contractor should ensure a clear of reference for the the Terms of limited ability to focus on capacity building in the evaluation called for Reference negotiate terms, which conduct of evaluations under household surveys to Any other issues lead to necessity SA 3, not necessarily building or measure antimalarial not mentioned of communication/ strengthening systems, but that use, these were not above explanation to dormant capacity at the country included in the final clarify appropriate level should be mobilized by evaluation design expectations with the contractor and in-country because of budget respect to complying Impact Evaluation Task Force implications. Instead, with specific terms of data from national the mandate surveys that were
|
From page 159... ...
appropriately timed (close to baseline and endline) were to be used to report on use.
|
From page 160... ...
into an annual report. exploration evaluation mandate evaluation, and upon approval stakeholders, including This report provides a of program's • Research on by the Global Fund Board, the TERG, AMFm summary of the various objectives, PEPFAR's legislative identified the independent Ad Hoc Committee, data collected during the activities, and and programmatic consultant to carry out AMFm Phase 1 country year, examines progress theory of change; intent and evaluation activities, after a officials and technical and trends, and points out data mapping, objectives, program competitive bidding process partners, which areas for improvements assessment design, complexity, included elements of the or enhancements in the of available/ and operational proposed independent coming year accessible data; structure evaluation and data assessment of • Design planning to mapping for national feasibility of choose conceptual level surveys planned data collection; framework; methods to be implemented by evaluation design to be applied others development.
|
From page 161... ...
country and by topic system record reviews in the Assessment of context area; benchmarking areas of HIV/AIDS, TB, and and process factors analysis for legislative, malaria using key informants policy, and other and document review. programmatic Theory of change objectives used to interpret and attribute changes, based on logic model together with context and process factors.
|
From page 162... ...
How predetermined technical Fund and through an extensive QAACT availability, Principles into Practice was sampling areas stakeholder consultation separately for rural (Obj 2) ; done for data and urban domains, Wider Partnership collection/ Comprehensive sample Sampling for data collection/ pooling across outlet Environment (Obj 3)
|
From page 163... ...
assessment of Global Fund interviews and Semi-structured interview Program clinical data governance and management, document review data Semi-structured performance review, Focus group discussions Document review interview data benchmarking of results and Secondary analysis of Globally reported malaria Document review processes, and document review household survey data data Globally reported indicator data Study Area 2: In-depth qualitative assessments (850 interviews) , extensive literature review and in-depth review and analysis of performance data on Global Fund grants Study Area 3: National health accounts, district facility censuses, household surveys, civil society organization surveys, record reviews, and follow-up studies of patients 163
|
From page 164... ...
and appendixes (287 communications Each country participating in the pages) , which includes a products beyond Report Summary booklet impact study (SA 3)
|
From page 165... ...
and the MDAG (July– report Summary reports by the Global October 2012) in lead Presentations at Fund, posting of reports on the up to board meeting conferences and meetings Global Fund website November 2012 Published commentaries The TERG briefings of the Global Presentations at Fund Board International Health Economics Association, American Society of Tropical Medicine and Hygiene (ASTMH)
|
From page 166... ...
DISSEMINATION AND UPTAKE OF EVALUATION FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 166 PEPFAR Global Fund AMFm PMI Responses to PEPFAR did an internal The TERG summaries of final Written feedback The PMI generated a evaluation written response to the study area reports received from the TERG management report recommendations and from the AMFm identifying which Global Fund Secretariat Working Group recommendations they Bidirectional exchange implemented provisions to wished to act upon and between the IOM and address some observations and Commentaries in describing how they would implementing agencies recommendations scientific journals and implement the proposed during technical media coverage changes briefings about specific considerations for The report was highlighted recommendations and on the PMI website and agency implementation broadly disseminated electronically Commentaries in journals and blog postings authored by stakeholders external to the IOM Mechanisms The IOM has Unknown No formal mechanism Beyond the scope of the for tracking limited formal and for tracking impact of Evaluation Team implementation resourced internal the evaluation of mechanisms for tracking recommendations recommendation or impact of the implementation and evaluation other report impact
|
From page 167... ...
contribution collection to the Global Fund or Evaluation Team was of programming and Preliminary findings constrained by lack of funding to NIH presented to multiple time data points consultative forum of to assess country trends Committee interpreted country stakeholders and impacts additional areas of for review and debate study not explicitly stated in mandate but determined to be needed to be responsive to the charge (i.e., funding, knowledge management, country ownership, and sustainability) SOURCE: Information compiled from evaluation summary documents and members of the respective evaluation teams.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.