Design Considerations for Evaluating the Impact of PEPFAR Workshop Summary (2008) / Chapter Skim
Currently Skimming:
4 Designing an Impact Evaluation with Robust Methodologies
4 Designing an Impact Evaluation with Robust Methodologies
Pages 77-110
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 77... ...
(CARE) , and The Global Fund to Fight AIDS, Tuberculosis, and Malaria (The Global Fund)
|
From page 78... ...
Examples of impact evaluation methods, provided by speaker Mary Lyn Field-Nguer of John Snow, Inc., include client satisfaction interviews and surveys, exit interviews, mystery clients, targeted intervention research, focus groups, and key informant interviews. The following case studies describe the experiences from evaluations of five HIV/AIDS assistance programs run by the World Bank, Poverty Action Lab, DFID, CARE, and The Global Fund.
|
From page 79... ...
The World Bank evaluation drew on a number of methodological approaches. As Ainsworth noted, the World Bank does not rely exclusively on a single source of information, but rather uses different types of evaluations already occurring in the context of the work, such as midterm reviews, completion reports, and annual reviews.
|
From page 80... ...
In randomized controlled trials, like medical clinical trials, those who receive the treatment and the control group are selected randomly. By construction, those who receive the proposed new intervention are no more committed, no more motivated, no richer, and no more educated than those in the control group.
|
From page 81... ...
One workshop participant noted that randomized controlled trials can be difficult to translate from the individual level to the community level, where interventions are more complex. Glennerster acknowledged that randomized controlled trials can be improperly designed and can thereby generate incorrect results.
|
From page 82... ...
Upon implementation of each program, the evaluation tracked observed changes in behavior, including school dropout rates, marriage, pregnancy, and childbirth, as determined by community interviews. Follow-up studies are also tracking HIV infection rates under each type of intervention.
|
From page 83... ...
On the basis of the costs of the interventions, the evaluators were able to calculate a cost-per-childbirth-averted rate for each intervention, with the education program about older men being the most cost-effective intervention, at $91 per childbirth averted, compared to $750 per childbirth averted for interventions to reduce the cost of schooling. Using Randomized Trials to Evaluate HIV Status Knowledge Programs in Malawi Although half of HIV/AIDS prevention spending in Africa focuses on HIV testing, many of those tested do not come back to pick up their results.
|
From page 84... ...
Glennerster also noted that the use of plausible correlation approaches -- suggested by workshop speaker Paul De Lay of the Joint United Nations Programme on HIV/AIDS (UNAIDS) as a more practical methodology applicable to work at the country level -- without doing a full trial can also lead to the wrong policy conclusion.
|
From page 85... ...
DESIGNING AN IMPACT EVALUATION WITH ROBUST METHODOLOGIES 85 cation and understanding what policy makers want -- may result in missed lessons from evaluation. A major challenge to the DFID evaluation was the declining quantity and quality of data collected at projects in-country.
|
From page 86... ...
Methodological Challenges And Opportunities in Evaluating Impact Workshop participants described methodological challenges and opportunities in evaluating the impact of PEPFAR, including those in measuring outcomes and impacts specific to HIV/AIDS, measuring broader impacts and outcomes, attributing results, and aggregating the results of impact evaluation. The discussions were wide-ranging and touched on many chal Demographic and Health Surveys including HIV prevalence measurement are known as "DHS+."
|
From page 87... ...
sentinel sites is currently the most common method for measuring changes in HIV prevalence. Workshop speaker Theresa Diaz of the U.S.
|
From page 88... ...
can also be used to distinguish recent infections from long-term infections on the basis of relative levels of anti-HIV antibodies, noted Diaz, but they tend to overestimate the proportion of most recent infections under certain circumstances, for example, in people who have taken ARV drugs immediately before the test. Finally, the potential for spread of infection and future infection can be measured through modeling techniques, noted speaker Garnett.
|
From page 89... ...
For example, noted Diaz, CDC's methodology for predicting infections averted in newborns by comparing mother-to-child transmission of HIV with or without preventive ARVs does not consider epidemiological context factors such as breast-feeding practices, efficacy differences among programs and countries, adherence and proper use, and impacts of counseling. Alternative models, population surveys, and cross-country comparisons were put forward by workshop participants as possible opportunities for more effectively measuring changes in infections averted.
|
From page 90... ...
Measuring Changes in Survival and Mortality Rates Workshop participants identified a number of challenges in measuring changes in survival -- the percentage of a group who are alive for a given period of time after diagnosis or treatment -- and in mortality rates -- the proportion of deaths in an area compared to the population of that area per unit of time. Speaker Diaz pointed out that measuring changes in mortality rate in the general population has raised questions about whether ART decreases mortality, through increasing chances of survival of HIV-infected individuals, or increases mortality, through increased opportunities for viral transmission to others.
|
From page 91... ...
Stoneburner responded that better use of existing TB and ARV surveillance data, including ARV cohort survival analyses, rather than reliance on population-level vital registration data would better identify risk factors for such adverse outcomes and better guide clinical management. Workshop participants identified a number of opportunities for more effective measurement of mortality and survival rates.
|
From page 92... ...
Based on data provided by the National AIDS Commission Trust of the Republic of Malawi, Principal Recipient for the Global Fund programs. obvious, non-HIV-related causes of death, such as accidents and maternal mortality.
|
From page 93... ...
For some interventions, such as adherence to ARV treatment, incomplete behavioral change has consequences that are even worse than no behavioral change, noted Latkin, because moderate adherence could provide a greater selective pressure for the evolution of viral drug resistance compared to poor adherence. Methods for determining such unintended impacts need to be developed.
|
From page 94... ...
Workshop participants noted several opportunities for evaluating behavioral change. Latkin urged a shift in measuring change from what is happening at the individual level to what is happening at the social and institutional levels (such as, community, network, or national levels)
|
From page 95... ...
These prevalence results in Zimbabwe have been corroborated through randomized controlled trials conducted at two time points. Surveys conducted in conjunction with the trials demonstrate that declines may have resulted from behavioral changes such as foregoing casual sexual partners and reducing simultaneous partners (Gregson et al., 2006)
|
From page 96... ...
, discussant Fowler also noted the importance of developing measures that would track the extent to which stigma is a factor in patients who seek other services, such as antenatal care. New methods for measuring stigma and new sources of data may provide opportunities that will be useful to the future impact evaluation of PEPFAR, workshop participants said.
|
From page 97... ...
New data are showing that testing, diagnosis, having the disease, physical manifestations of AIDS, status disclosure, suspicion, and rumors are all triggers to the cascade of stigma events. Measuring Changes in Orphanhood Prevention Workshop participants discussed some of the challenges and opportunities in measuring changes in orphanhood prevention -- the prevention of the death of one or usually both parents of a child.
|
From page 98... ...
Workshop participants discussed some of the challenges and opportunities in adapting a traditional evaluation framework to measure broad impacts or unintended impacts of PEPFAR interventions. This section describes the measurement of the impact of the following: health systems strengthening, complementary interventions, gender-focused activities, coordination and harmonization, and population-level service delivery.
|
From page 99... ...
Two case studies were also presented describing indicators and methodologies that can be used in evaluating health systems strengthening. Evaluating health system–wide impacts of Global Fund interventions. Workshop speaker John Novak, senior monitoring and evaluation adviser of the Office of HIV/AIDS at the U.S.
|
From page 100... ...
The study assessed 30 FHI partner health centers from 4 provinces and 14 districts in Rwanda, representing 21 faith-based centers and 9 public centers. Hospitals that do not deliver some non-HIV services and health facilities with fewer than 6 months' experience delivering basic HIV care were excluded from the study.
|
From page 101... ...
Workshop participant Laura Porter of CDC added that future studies will need to ensure that service delivery improvement is a real effect and not just an artifact of data system improvement. Measuring Impact of Complementary Interventions As described in Chapter 2, PEPFAR investments include numerous interventions in programs complementary to more narrowly focused HIV services.
|
From page 102... ...
Any intervention conducted during this 5-year period would have given the appearance of stimulating a large positive effect when there might have been none at all -- or perhaps even a negative effect. Measuring Impacts of Gender-Focused Activities Workshop participants discussed some of the challenges and opportunities for evaluating the impacts of gender-focused activities, including those interventions to promote gender equality and women's empowerment.
|
From page 103... ...
Pulerwitz shared the study design and tools used for an evaluation of gender-focused programs -- group education, community-based behavioral change communication campaigns, and clinical activities -- focused on young men in Brazil. A combination of data collection approaches were used, including the following: • Pre- and postintervention surveys and a 6-month follow-up survey for three groups of young men -- two intervention groups and a comparison group, which eventually also received the interventions after a time delay -- followed over a year • In-depth interviews with a subsample of young men and their sexual partners • Costing analysis and monitoring forms for different activities An evaluation tool called the Gender Equitable Men's (GEM)
|
From page 104... ...
Measuring Coordination and Harmonization Workshop speaker De Lay spoke of a new opportunity for measuring coordination and harmonization -- the alignment of interventions with country-level plans and coordination of efforts among other implementing partners. A new tool, known as the Country Harmonization and Alignment Tool (CHAT)
|
From page 105... ...
As PEPFAR moves increasingly toward more harmonized approaches, noted speaker Compton, it will be even more difficult to disentangle effects in an exclusive way. Many workshop participants agreed that the demand for exclusive attribution by donors may not be constructive.
|
From page 106... ...
As speaker Bertozzi pointed out, in the case of South Africa, even if outside institutions did not intervene, given the massive social mobilization potential in the country, dramatic change could have been effected without outside help. Aggregating Evaluation Results Several speakers noted that the synthesis or aggregation of evaluation results is a methodological frontier.
|
From page 107... ...
Value of Consultation and Communication Several speakers emphasized the value of consultation and communication in any evaluation approach. Speakers Compton and Glenzer observed that consultation and communication through the evaluation process are as important in effecting change and course corrections as the data from the evaluation results.
|
From page 108... ...
Speaker Compton urged that evaluations be set up at the beginning of the process, and speaker Bertozzi also spoke about some of the drawbacks of an ex-post evaluation. Speaker Glennerster noted that opportunities to use powerful randomization approaches exist, but they can be used only if the design is included at the beginning of an intervention.
|
From page 109... ...
Speaker Glenzer reinforced the point with his comment that centrally planned, mixed-method evaluation designs work best. At the same time, the use of multiple methods should be strategic, noted workshop speaker Glennerster.
|
From page 110... ...
Interventions that are successful in one country are not necessarily transferable to another country, noted workshop speaker Stoneburner. Examples provided by Stoneburner and speakers Latkin, Garnett, and Pulerwitz supported this statement.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.