Design Considerations for Evaluating the Impact of PEPFAR Workshop Summary (2008) / Chapter Skim
Currently Skimming:
2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting
2 Envisioning a Meaningful Impact Evaluation for PEPFAR: Moving Beyond Counting
Pages 37-66
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 37... ...
Workshop participants stressed the importance of identifying meaningful and appropriate impact indicators and moving beyond quantitative outputs. Workshop speakers Jonathan Mwiindi of the Kijabe HIV/AIDS Relief Program, Kenya, and Mary Lyn Field-Nguer of John Snow, Inc., observed that the use of inappropriate indicators for evaluating program impact can sometimes tell an incomplete story, mislead, or mask other problems.
|
From page 38... ...
Defining boundaries includes answering the question of whether to measure direct AIDS impacts very narrowly or to measure broader impacts such as gender empowerment or land rights or health systems. Workshop participants identified questions for evaluating impact that can be clustered into the following nine broad categories: cost-effectiveness, conceptual approach, health impacts, impacts beyond health, capacity building and health systems strengthening, coordination and harmonization, sustainability, equity and fairness, and unintended impacts.
|
From page 39... ...
Conceptual Approach A number of proposed impact evaluation questions centered on the conceptual approach of PEPFAR and the assumptions that had been made in the program's design: the countries and populations targeted, the budget allocations for different types of interventions, and the management and financing of the program. Such impact evaluation questions can be helpful, noted workshop speaker Compton, in thinking through the conceptual model of how PEPFAR inputs lead to outputs and outcomes.
|
From page 40... ...
Citing the imbalance in enrolling 90,000 people in treatment programs while 135,000 people were newly infected each year over the first phase of PEPFAR, speaker Compton urged the development of impact indicators to assess the appropriate balance among prevention, treatment, and care interventions. She also suggested an evaluation of the balance among interventions within prevention, noting that a significant proportion of the prevention funding is required to support abstinence and faithfulness (AB)
|
From page 41... ...
House of Representatives Oversight Committee warned that it may be risky to mix scientific and objective evaluation with evaluation of more difficult political questions, such as evaluating a policy that prevents a certain type of intervention from taking place. As speaker Compton noted, "An evaluation cannot answer what goals are most important; that is a political decision." Management and Financing Workshop participants also encouraged the design of impact evaluation measures for the management and financing of the PEPFAR program across a variety of levels.
|
From page 42... ...
Evaluation may reveal that some approaches may work more efficiently for certain contexts; for example, where there is high local capacity, The Global Fund approach might be more efficient, and in a country with lower capacity, the PEPFAR approach might be more efficient. Health Impacts Workshop participants suggested the development of impact evaluation measures for tracking change in health, including indicators specific to HIV/AIDS as well as more general indicators.
|
From page 43... ...
He and speaker Mwiindi urged the development of indicators to measure the completeness of viral suppression, the level of drug adherence, and the level of resistance created by PEPFAR-supported treatment programs. In terms of behavioral effects, treatment programs can on one hand encourage healthseeking behaviors, including prevention behaviors, but those receiving ART as well as HIV-positive and HIV-negative individuals in the community may on the other hand engage in more risky or other adverse behaviors TABLE 2-1 Possible Effects of ART on HIV Transmission Direction of Effect Beneficial Adverse Type of Effect (Slow transmission)
|
From page 44... ...
She cited an example from Thailand in which the mere dissemination of knowledge that 44 percent of sex workers in Chiang Mai were infected with HIV had a dramatic effect on sexual behavior. Speaker Compton suggested that evaluation could be used to assess the impacts of a lack of investment in
|
From page 45... ...
Other Health Impacts Future impact evaluation of PEPFAR could also consider other health indicators in addition to those related to HIV/AIDS. Evidence from the Rwanda context, noted speaker Allen Moore, suggests that many health indicators beyond those specific to HIV/AIDS have improved as a direct outcome of PEPFAR investments.
|
From page 46... ...
Impacts Beyond Health What have been the effects of PEPFAR interventions in the areas of gender equality, child welfare, security and development, and institutional change? Workshop participants discussed a number of measures of HIV/ AIDS interventions beyond health that could be tracked through impact evaluation.
|
From page 47... ...
Speaker Compton raised the question of what technical assistance is doing to help change political views and priorities in a country. Impacts on Sustainability, Capacity Building, and Health Systems Strengthening Workshop participants emphasized the importance of monitoring both the process and results of building local capacity, particularly in the area
|
From page 48... ...
Needs for tracking changes in the capacity of the health care system were identified in the areas of health care workforce, infrastructure, institutions, quality of care, the knowledge base, and the national-level health agenda. Health Care Workforce Measuring the Effects of PEPFAR on Workforce Shifts The HIV/AIDS epidemic and interventions have had a dramatic effect on the health care workforce in many countries, noted speaker Sepúlveda.
|
From page 49... ...
The effectiveness of training HIV-positive patients to serve as community health care workers could be evaluated, suggested speaker Mwiindi. Possible measurements of workforce training programs should go be
|
From page 50... ...
Metrics for evaluation of the effectiveness of these interventions might include, for example, whether services for health professionals are provided at different times of day from regular treatment programs, given professionals' desire not to wait in line and mix with clients. Sustaining Local Workforce Development Systems Speaker Holzemer noted the importance of evaluating how workforce development strategies complement and contribute to the sustainability of those systems in the country that controls the workforce.
|
From page 51... ...
Speakers Mwiindi and Field-Nguer suggested a variety of indicators that could be used to track the effectiveness of PEPFAR interventions in the area of quality improvement. These include the quality and appropriateness of service delivery, and existing gaps, in the areas of prevention, care, treatment, support, and mitigation; ARV retention rates; levels of client satisfaction; appropriateness of referrals; and improved community attitudes toward people living with HIV/AIDS.
|
From page 52... ...
The appropriateness of the balance between implementation and strengthening the knowledge domain is an element that can be assessed through impact evaluation. National-Level Health Agenda Speakers Nils Daulaire of the Global Health Council and Sepúlveda broadened the discussion of evaluating PEPFAR's impacts on health systems strengthening to include assessment of how well PEPFAR has integrated with other health issues and has contributed to the change in national-level health agendas and priorities over time.
|
From page 53... ...
agencies, there was initially a lack of consistent targets across focus countries because each government agency had its own reporting system; such systems have had to be coordinated and harmonized through the PEPFAR program. Among the main needs identified by workshop participants for evaluation of coordination and harmonization among U.S.
|
From page 54... ...
Potential negative impacts of complementary interventions are further described later in this chapter under "Unintended Impacts." Harmonization and Alignment with Partner Countries A second level of harmonization identified for future evaluation by workshop participants is harmonization with partner countries. Workshop participants value harmonization and alignment of PEPFAR with country priorities and plans because they believe this leads to greater success of the overall program.
|
From page 55... ...
Citing a perceived criticism in the IOM committee's report that alignment had been difficult in the early years of the Uganda partnership because of the many funding earmarks and restrictions, discussant Ambassador Kolker acknowledged that although U.S. efforts were not a perfect match to local priorities, they were congruent with the national plan.
|
From page 56... ...
Mwiindi added that the effectiveness of PEPFAR's harmonization with existing drug procurement systems also be evaluated, which might include tracking measures such as drug quality, drug cost, reliability of the drug supply, and level of engagement with organizations on the ground, such as religious groups, that function in the existing health care delivery and drug supply system. PEPFAR partner countries can apply for a waiver that allows them to reapportion their prevention funds among abstinence–faithfulness–condom use (ABC)
|
From page 57... ...
Speaker Mwiindi suggested that evaluation could help prompt the development of and measure the progress toward a clear exit strategy. Measures could be developed, for example, of the degree to which the PEPFAR program has leveraged additional resources by other donors and by national governments, remarked workshop discussant Sherry.
|
From page 58... ...
He provided examples showing the trade-offs and tensions between the competing goals of equity and efficiency in decision making. Finally, he offered guidance on indicators that can be developed to examine issues of equity as PEPFAR programs are scaled up.
|
From page 59... ...
The issue of site selection was similarly controversial. Although mobilization of resources and trained personnel in tertiary care centers may be technically the most efficient solution for rapid scale-up, a concentration of service delivery in areas where the largest numbers of people can be reached most rapidly leaves people in rural areas without a fair chance of any benefit because of poor access to services.
|
From page 60... ...
Because randomized trials traditionally require a control group, there is an ethical dilemma of excluding people from access to a program that might save their lives. Prevention and Treatment Another workshop participant raised the example of tension between the prevention and treatment approaches of PEPFAR, that is, between the people living with the disease who will die without treatment and those who will benefit from effective prevention programs -- the uninfected population plus future generations.
|
From page 61... ...
Daniels noted that the issue of brain drain of trained health workers can be an element of evaluating PEPFAR integration with the broader health system. Useful information would include data on health personnel in parts of the health system that are adjacent to the ones where scale-up sites are being established.
|
From page 62... ...
Speaker Compton pointed out that both positive and negative synergies can occur and that methodologies are needed for capturing contextual data that will allow detection of these unintended impacts. Workshop participants discussed potential unintended effects of PEPFAR on program integration, diversion of resources from other health areas, corruption, access to services, adverse and high-risk behavior, nutrition, and reproductive health and family planning.
|
From page 63... ...
Compton suggested that evaluation assess the macroeconomic impacts of PEPFAR funding on brain drain from essential services resulting from hiring and per diem practices. Speaker Mwiindi proposed that unintended effects of PEPFAR's large, multicountry supply chain program on existing supply chains be evaluated.
|
From page 64... ...
Reproductive Health and Family Planning Unintended impacts of PEPFAR on reproductive health and family planning was identified by workshop participants as an area of interest for evaluation. Several participants noted that PEPFAR's reproductive health programs may have had some positive synergies with HIV prevention, treatment, and care programs.
|
From page 65... ...
ENVISIONING A MEANINGFUL IMPACT EVALUATION 65 hood years among a woman's current children, new orphanhood years can also be created among children born during the extended life span of a woman on treatment. Speakers Daulaire and Kenyon suggested that evaluation could be used to assess positive or negative synergistic effects of PEPFAR's policies not to procure family planning commodities and contraceptives.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.