Design Considerations for Evaluating the Impact of PEPFAR Workshop Summary (2008) / Chapter Skim
Currently Skimming:
Overview
Overview
Pages 1-22
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 1... ...
The workshop represented a final element of the work of the congressionally mandated IOM Committee for the Evaluation of PEPFAR Implementation, which published a report of its findings in 2007 (IOM, 2007) evaluating the first 2 years of implementation, but could not address longer term impact evaluation questions.
|
From page 2... ...
The final section addresses methodological issues that were raised by participants as being important to consider in the design of future impact evaluations. Defining and Using Impact Evaluation Meeting participants proposed a working definition of impact evaluation as a measurement of net change in outcomes attributable to a specific program using a methodology that is robust, available, feasible, and appropriate both to the question under investigation and to the specific context.
|
From page 3... ...
Initially, reported PEPFAR officials, the conceptual framework for PEPFAR impact evaluation was much more narrowly defined, and country capacity for monitoring was limited, with a lack of consistent targets across countries and a lack of hard data. Evaluation efforts and reporting requirements were loosely coordinated among U.S.
|
From page 4... ...
Workshop participants identified questions for evaluating impact that can be clustered into the following nine broad categories: cost-effectiveness, logic of conceptual approach, health impacts, impacts beyond health, capacity building and health systems strengthening, coordination and harmonization, sustainability, equity and fairness, and unintended impacts. These questions are summarized in Box O-1.
|
From page 5... ...
OVERVIEW BOX O-1 Summary of Impact Evaluation Questions as Identified by Workshop Participants Cost-effectiveness Approaches, strategies, and interventions, such as prevention services and treat ment options Conceptual Approach Countries targeted Populations targeted Budget allocations for types of interventions Management and financing Health Impacts HIV/AIDS-specific health impacts • Prevalence, incidence, morbidity, mortality, longevity • Prevention of HIV transmission • Quality of life • Behavioral change • Stigma and discrimination Other health impacts, disaggregated by population • Overall mortality, lives saved, survival • Child mortality • Fertility, unintended and intended pregnancy Impacts Beyond Health Gender equality • Effectiveness of PEPFAR in addressing underlying causes of women's vulnerability • Effectiveness in building men's and women's analytical skills and competencies • Effectiveness of messages for behavioral change • Effectiveness of interventions to reduce the spread of HIV infection to women and girls • Effectiveness of the "packaging" of gender interventions Child welfare • Effectiveness in improving parenting skills • Health, nutritional, and educational status of orphans and vulnerable children Continued
|
From page 6... ...
government implementing agencies • Consistency of targets among implementing agencies • Positive and negative impacts of complementary interventions, or "wrap around" programs
|
From page 7... ...
OVERVIEW BOX O-1 Continued Harmonization and alignment with partner countries • Openness and accessibility of plans and projects at the community level • Presence of instruments and structures for joint decision making • Degree of information sharing and joint implementation among partners • Existence of mechanisms to make coordination more flexible Coordination among program implementers • Development and effectiveness of a variety of coordination tools or mecha nisms for fostering the exchange of learning • Level of harmonization of drug procurement systems Sustainability Impacts Degree to which additional resources have been leveraged from other donors Extent to which long-term learning and research have been promoted Measures of capacity building and sustained contributions to institutions and systems Degree to which local implementation, ownership, and coordination have been promoted Equity and Fairness Impacts Existence and effectiveness of processes for goal setting and implementation Fairness impacts, disaggregated by group, of program integration within the health system Existence and effectiveness of compensatory mechanisms to improve fairness Positive and Negative Unintended Impacts Impacts of earmarking on program integration Diversion of resources from neglected health care areas and the broader health care system Impact of PEPFAR on corruption Impact of PEPFAR on access to services Impact of treatment on adverse and high-risk behavior Impact of nutritional programs Impact of PEPFAR programs on reproductive health and family planning • Impact of counseling and testing on pregnancy care • Impact of ARV treatment on fertility and orphanhood
|
From page 8... ...
Workshop participants highlighted key opportunities to share the outcomes of PEPFAR's evaluation efforts with the 5-year evaluation of The Global Fund, a planned evaluation by UNAIDS, and evaluative efforts by the World Bank, the Organisation for Economic Co-operation and Development, and the World Health Organization. Participants also noted that opportunities to harmonize the design, conduct, and interpretation of evaluation results with country-level partners include the conduct of joint field evaluations by collaborating partners; the development of centralized funding, knowledge management, and data aggregation systems; and discussions leading to consensus among partners on monitoring approaches and overall program objectives.
|
From page 9... ...
Designing an Evaluation Based on Robust Methodologies Workshop participants stressed the importance of designing impact evaluation on the basis of a logical conceptual approach and robust methodologies. Case studies of evaluations of HIV/AIDS interventions were presented, and challenges and opportunities in evaluating impact were discussed.
|
From page 10... ...
Factors independent of program interventions may have a significant influence on change. Methodological Challenges and Opportunities in Evaluation Challenges and Opportunities in Measuring HIV/AIDS-Specific and General Impacts Workshop participants explored the limitations of commonly used methods in evaluation, as well as new prospects, in measuring both HIV/AIDS-specific and more general impacts.
|
From page 11... ...
OVERVIEW 11 which combines results from multiple studies as if they were a single large study, is a tool that is currently underdeveloped for application to impact evaluation. Multiple analyses also have value in the independent validation of results.
|
From page 12... ...
HIV incidence The number of new cases of HIV within a population at risk over a given period of time. Infections averted The difference between expected and actual annual incidence.
|
From page 13... ...
• Empirical data on incidence by age and sex are lacking. • Infections averted are a • A new model called Spectrum takes "nonevent," and their epidemiological contextual factors into measurement requires multiple account.
|
From page 14... ...
Behavioral change Modification of sexual, injection, and drug-adherence practices.
|
From page 15... ...
• Current methods might not be • Behavior surveys, particularly those targeting able to determine the extent and younger people, who are an early indicator coverage of behavioral change. of prevalence changes, can be useful for Incomplete behavioral change attributing changes in HIV incidence to can have worse consequences specific changes in risk.
|
From page 16... ...
16 EVALUATING THE IMPACT OF PEPFAR TABLE O-1 Continued Metric Definition Stigma and discrimination Negative attitudes, beliefs, and actions toward people who are perceived to have HIV/AIDS and those associated with them. Orphanhood prevention Prevention of the death of one or usually both parents of a child.
|
From page 17... ...
OVERVIEW 17 Challenges Opportunities • Rigorous research and data • An International Planned Parenthood collection approaches are Federation stigma index is now available. absent.
|
From page 18... ...
Health care workforce Improvement of a range of capacities related to strengthening health care personnel, including training, supervision, and job satisfaction.
|
From page 19... ...
in their first pregnancies who are likely not to be in ARV treatment can be used to track the transmission of drug-resistant HIV strains. • Therapy monitoring can be used to measure drug resistance by sampling and monitoring patients in ARV treatment from the initiation of therapy over a 1-year time period.
|
From page 20... ...
Effectiveness of coordination and Increased alignment of HIV/AIDS interventions with harmonization country-level plans and coordination of efforts among other implementing partners. Effectiveness of community- or Improved service delivery for specific populations, population-level service delivery that is, children, families, communities, HIV-infected groups, high-risk groups, etc.
|
From page 21... ...
have been developed to examine community-level service delivery and help answer questions such as when, how, and where people want testing and treatment.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.