Evaluation Design for Complex Global Initiatives: Workshop Summary (2014)

Ann Kurth, New York University, Planning Committee Chair

Simon Hearn, Overseas Development Institute/BetterEvaluation Initiative

• Identify the evaluation goals, develop and prioritize the evaluation questions, and define the audience and the intended uses.
• Understand how decisions and trade-offs in

• choosing the type of evaluator and in setting up the governance structure for the evaluation may be affected by the different aims/audiences and the degree to which they are aligned.

• Understand how the options and trade-offs in choosing the type of evaluator and in setting up the governance structure contribute to designing and conducting the best evaluation for the available funds.
• Identify how to understand, ensure, and assess the value and utility of the evaluation and its findings.

Discussants: Chris Whitty, UK Department for International Development

Ryuichi Komatsu, Global Fund to Fight AIDS, Tuberculosis, and Malaria

Deborah Rugg, United Nations Inspection and Evaluation Division

Carmela Green-Abate, Country Coordinator, PEPFAR Ethiopia

Robert Black, Johns Hopkins University

• The importance of mixed methods approaches to evaluate complex initiatives and to understand not only whether an effect was achieved but also how and why.

• The importance of strategically thinking through different options for methods and how they can be matched to the evaluation aims/questions, the available data, and the feasibility of implementing the methods with appropriate rigor (design that is “fit for purpose” and feasible).
• How to recognize, understand, and grapple with the complexity of the initiative being evaluated and the context in which that initiative is implemented—this affects, for example, attribution versus contribution designs and expectations for findings.
• The importance of planning for and building flexibility into the design.

Presentation: Evaluation of PEPFAR

Deborah Rugg, United Nations Inspection and Evaluation Division

Presentation: Evaluation of Global Fund

Daniel Low-Beer, Global Fund to Fight AIDS, Tuberculosis, and Malaria

Presentation: Evaluation of the Affordable Medicines Facility–malaria

Catherine Goodman, London School of Hygiene and Tropical Medicine

Presentation: Illustration of Good Practices for Complex Evaluation Design

Elliot Stern, Lancaster University

• The importance of and strategies for identifying and assessing potential data sources and mapping these to the evaluation questions and the methodological approaches being considered (data that is “fit for purpose”).
• The key elements of assessing data sources:

• Availability
• Accessibility (including data ownership issues and data sharing agreements)
• Quality
• Feasibility of gathering/accessing the identified data

Discussants: Martin Vaessen, ICF, Planning Committee Member

Kara Hanson, London School of Hygiene and Tropical Medicine, Planning Committee Member

Jon Simon, Boston University, Planning Committee Member

Batya Elul, Columbia University

Victoria Fan, Center for Global Development

Peter Elias, University of Warwick

• Data analysis and interpretation within each data type/method according to the methodological standards of rigor for that method (i.e., avoiding “mixed-up” methods).
• Triangulation among the findings from mixed methods to achieve an overall, integrated analysis and interpretation to arrive at major conclusions and recommendations.
• Triangulation among investigators.

• Among members of the evaluation team
• Use of expert/advisory panels
• Use of internal or external reviewers

Discussants: Jon Simon, Boston University, Planning Committee Member

Catherine Goodman, London School of Hygiene and Tropical Medicine

Sangeeta Mookherji, George Washington University

Bridget Kelly, Institute of Medicine

Ann Kurth, New York University, Planning Committee Chair

Ann Kurth, New York University, Planning Committee Chair

• Complexity of diverse uses and audiences for large-scale evaluations (e.g., accountability to funders/legislators; program improvement at headquarter, implementer, and country program levels; advocacy to continue initiative; public awareness).
• Importance of matching the message, the messenger, and the audience (communication that is “fit for purpose”).
• Challenges with tracking the use of large-scale evaluation findings.

Discussants: Bernard Nahlen, President’s Malaria Initiative

Dorothy Muroki, FHI360, Planning Committee Member

Ian Goldman, Department of Performance Monitoring and Evaluation, Government of South Africa

Kimberly Scott, Institute of Medicine

Presenters: Margaret Kruk, Columbia University

Padma Chandrasekaran, The Chennai Angels

Tanya Marchant, London School of Hygiene and Tropical Medicine

Presenters: Geoff Wong, Queen Mary University of London

Emmanuela Gakidou, Institute for Health Metrics and Evaluation

Caine Rolleston, University of London

Evaluation experts: Tessie Catsambas, EnCompass LLC

Sharon Knight, East Carolina University

Presenters: Sharon Knight, East Carolina University Tessie Catsambas, EnCompass LLC

Charlotte Watts, London School of Hygiene and Tropical Medicine

Rachel Nugent, University of Washington

Sangeeta Mookherji, George Washington University

Joanna Schellenberg, London School of Hygiene and Tropical Medicine

Presenters: Sanjeev Sridharan, Evaluation Centre for Complex Health Interventions, University of Toronto/St. Michaels’s Hospital

Charlotte Watts, London School of Hygiene and Tropical Medicine

Gina Dallabetta, Bill & Melinda Gates Foundation