Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary (2008)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Overview A 2-day workshop on methodological, policy, and practical design considerations for a future evaluation of human immunodeficiency virus/ acquired immunodeficiency syndrome (HIV/AIDS) interventions carried out under the President’s Emergency Plan for AIDS Relief (PEPFAR) was convened by the Institute of Medicine (IOM) on April 30 and May 1, 2007. Participants at the workshop included staff of the U.S. Congress; PEPFAR officials and implementers; major multilateral organizations such as The Global Fund to Fight AIDS, Malaria, and Tuberculosis (The Global Fund), the Joint United Nations Programme on HIV/AIDS (UNAIDS), and the World Bank; evaluation experts experienced with similar kinds of evalu- ations; and representatives of partner countries, particularly the PEPFAR focus countries. The workshop represented a final element of the work of the congressionally mandated IOM Committee for the Evaluation of PEPFAR Implementation, which published a report of its findings in 2007 (IOM, 2007) evaluating the first 2 years of implementation, but could not address longer term impact evaluation questions. This overview describes core messages from the workshop’s presenta- tions and discussions. First, background is provided on the definition and uses of impact evaluation, the process of internal evaluation within the PEPFAR program, and the recommendations concerning the design of The workshop summary has been prepared by the workshop rapporteurs as a factual sum- mary of what occurred at the workshop.  

 EVALUATING THE IMPACT OF PEPFAR future impact evaluations by the IOM committee that evaluated PEPFAR implementation. Next, a vision is described for the types of questions work- shop participants would like to see addressed in future impact evaluations, along with suggestions for how the process of impact evaluation would ideally be carried out. The final section addresses methodological issues that were raised by participants as being important to consider in the design of future impact evaluations. Defining and Using Impact Evaluation Meeting participants proposed a working definition of impact evalua- tion as a measurement of net change in outcomes attributable to a specific program using a methodology that is robust, available, feasible, and appro- priate both to the question under investigation and to the specific context. Workshop participants noted that impact evaluation is not only about outcomes, but also the process that leads to outcomes; that is, it includes both means and ends. Participants argued for a definition of impact evalu- ation that is longer term, more broadly defined, and less linear. Although a more traditional definition of infectious disease impact evaluation (that is, one that is limited to metrics such as prevalence, incidence, infections averted, morbidity, and mortality) is important, the broader and deeper im- pact evaluation envisioned would also include measurement of changes in health status, systems capacity, quality of services, economic development, and social, economic, and political outcomes. Two major uses of evaluation were described: (1) use of evaluation for judging the performance of the program for purposes of accountabil- ity (summative evaluation) and (2) use of evaluation for informing the improved decision making within a program (formative, or utilization- focused, evaluation). Formative evaluation of PEPFAR is important to inform both congressional decision making and programmatic decision making in partner countries, although decision makers at different levels may have different evaluation needs, participants said. For each evaluation question, participants noted, it is important to clarify who needs the infor- mation, what information is needed, and when. Internal Evaluation of PEPFAR According to PEPFAR officials, a key aspect of PEPFAR’s monitoring process and infrastructure is to track progress toward the program’s goals in prevention, treatment, and care. PEPFAR internal monitoring supports the principles of local leadership and ownership of the HIV/AIDS response by building local capacity, using local infrastructure, implementing the pro- gram according to national guidelines, monitoring using locally developed 

OVERVIEW  indicators, and funding programs based on results, PEPFAR officials noted. Over time, PEPFAR’s evaluation activities have expanded field operations relative to central operations and have expanded reporting infrastructure and trained personnel in-country. Initially, reported PEPFAR officials, the conceptual framework for PEPFAR impact evaluation was much more narrowly defined, and coun- try capacity for monitoring was limited, with a lack of consistent targets across countries and a lack of hard data. Evaluation efforts and reporting requirements were loosely coordinated among U.S. implementing agen- cies. Over the first 5 years of PEPFAR, evaluation planners developed a monitoring and evaluation system that relied on survey data and periodic targeted evaluations on selected topics. Although this approach provided results on the deployment and use of funds, the development and delivery of services, and the beneficiaries of program services, as the complexity of PEPFAR program strategies grows and as the program undergoes a transi- tion from an emergency response to a sustained response, the definition of impact and approaches used to measure impact will need to be broadened. PEPFAR has recently developed a new, centrally managed public health evaluation (PHE) structure that can be used to aggregate results across multiple countries, multiple time points, and multiple settings. The PHE approach is designed to support evaluation by helping to set priorities, provide technical assistance, establish common protocols, and coordinate projects across countries. Future Evaluation Design: Perspective from the IOM Committee Speaker Jaime Sepúlveda, chair of the IOM committee that authored the report, PEPFAR Implementation: Progress and Promise (IOM, 2007), reviewed the recommendations of the committee on the design of impact measures—both AIDS-specific and more general indicators—for future evaluation of PEPFAR. Future evaluation should include measurement of what Sepúlveda referred to as the “three generations” of HIV/AIDS surveillance: prevalence and incidence of HIV infection, measurement of behavioral change, and measurement of stigma and discrimination. Other important indicators include measures of survival, quality of life, develop- ment of drug resistance, and the overall physical, mental, and social well- being of those people affected by HIV/AIDS. PEPFAR evaluation should also develop more general indicators, such as the empowerment of women and girls, overall health status, capacity of community-based organizations to respond, and public health infrastructure and capacity. 

 EVALUATING THE IMPACT OF PEPFAR Use of Evaluation to Focus on Meaningful Endpoints There was substantive focus during the workshop on what participants wanted to accomplish through PEPFAR. Although the workshop drew participants from wide-ranging and diverse perspectives—representatives of the U.S. government, evaluation methodological experts, representatives of partner countries, and global partners—most participants converged on the same kinds of endpoints. Participants emphasized a need to move be- yond counting numbers of people who are “touched” by the program and to instead focus on a set of more meaningful and more strategic questions about what interventions succeed. Participants cautioned that an emphasis on counting can be misleading, can drive inflated reporting, and can jeop- ardize sustainability of a program. Most participants called for a broader interpretation of impact that includes measurement of not only AIDS-specific impacts, but also more general impacts; measurement of not only results of implementation, but also the process of implementation; and measurement not only of the over- all benefits, but also the distribution of benefits. Workshop participants identified questions for evaluating impact that can be clustered into the following nine broad categories: cost-effectiveness, logic of conceptual approach, health impacts, impacts beyond health, capacity building and health systems strengthening, coordination and harmonization, sustain- ability, equity and fairness, and unintended impacts. These questions are summarized in Box O-1. Use of Evaluation to Focus on Collective Outcomes The meeting set a tone for the ideal conduct of evaluation—emphasizing collaboration, consultation, harmonization with the host countries, and co- ordination among global partners. Several workshop participants observed that the principles of coordination and harmonization need to be reflected in both the evaluation effort and in the overall implementation of PEPFAR. Workshop participants noted that many different actors—PEPFAR, The Global Fund, UNAIDS, the World Bank and others—are all working in HIV/AIDS response in many of the same countries and can learn from one another. Given the costs of evaluation, setting up evaluation in a collab- orative way can take advantage of this synergy and save resources. Many participants observed that although exclusive attribution of program suc- cesses to specific funders may not be realistic or constructive, coordinated evaluation may be able to better illuminate what types of interventions are the most effective. Workshop participants acknowledged that there have been earnest efforts to improve coordination and harmonization over the life of PEPFAR, and there is increased agreement to focus evaluation on collective outcomes. 

OVERVIEW  BOX O-1 Summary of Impact Evaluation Questions as Identified by Workshop Participants Cost-effectiveness Approaches, strategies, and interventions, such as prevention services and treat- ment options Conceptual Approach Countries targeted Populations targeted Budget allocations for types of interventions Management and financing Health Impacts HIV/AIDS-specific health impacts • Prevalence, incidence, morbidity, mortality, longevity • Prevention of HIV transmission • Quality of life • Behavioral change • Stigma and discrimination Other health impacts, disaggregated by population • Overall mortality, lives saved, survival • Child mortality • Fertility, unintended and intended pregnancy Impacts Beyond Health Gender equality • Effectiveness of PEPFAR in addressing underlying causes of women’s vulnerability • Effectiveness in building men’s and women’s analytical skills and competencies • Effectiveness of messages for behavioral change • Effectiveness of interventions to reduce the spread of HIV infection to women and girls • Effectiveness of the “packaging” of gender interventions Child welfare • Effectiveness in improving parenting skills • Health, nutritional, and educational status of orphans and vulnerable children Continued 

 EVALUATING THE IMPACT OF PEPFAR BOX O-1  Continued Security, development, productivity, and poverty alleviation • National peace and security in a nation • Poverty alleviation and economic growth • National development Institutional and societal changes • Policy changes such as property rights, inheritance laws, and human rights; political will; community ownership; food and water security; engagement of vulnerable populations; destigmatization; and discrimination measures • National priorities and political views Impacts on Capacity Building and Health Systems Strengthening Health care workforce • Effects of PEPFAR on workforce shifts • Impact of PEPFAR’s training approaches • Impact of PEPFAR’s programs to support health care workers • Effectiveness in sustaining local workforce development systems Effectiveness of institution-building efforts • Institution building of community-based organizations • Capacity building of nontraditional institutions Infrastructure • Effectiveness of strengthening supply chain management and drug delivery systems Quality of care and service delivery • Prevention, care, treatment, support, and mitigation; antiretroviral (ARV) drug retention rates; levels of client satisfaction; appropriateness of referrals; community attitudes toward people living with HIV/AIDS; and rational prescription behavior • Development of a knowledge base of what interventions work • National-level health agenda • Integration with other health issues; change in national-level health agendas Coordination and Harmonization Coordination among U.S. government implementing agencies • Consistency of targets among implementing agencies • Positive and negative impacts of complementary interventions, or “wrap- around” programs 

OVERVIEW  BOX O-1  Continued Harmonization and alignment with partner countries • Openness and accessibility of plans and projects at the community level • Presence of instruments and structures for joint decision making • Degree of information sharing and joint implementation among partners • Existence of mechanisms to make coordination more flexible Coordination among program implementers • Development and effectiveness of a variety of coordination tools or mecha- nisms for fostering the exchange of learning • Level of harmonization of drug procurement systems Sustainability Impacts Degree to which additional resources have been leveraged from other donors Extent to which long-term learning and research have been promoted Measures of capacity building and sustained contributions to institutions and systems Degree to which local implementation, ownership, and coordination have been promoted Equity and Fairness Impacts Existence and effectiveness of processes for goal setting and implementation Fairness impacts, disaggregated by group, of program integration within the health system Existence and effectiveness of compensatory mechanisms to improve fairness Positive and Negative Unintended Impacts Impacts of earmarking on program integration Diversion of resources from neglected health care areas and the broader health care system Impact of PEPFAR on corruption Impact of PEPFAR on access to services Impact of treatment on adverse and high-risk behavior Impact of nutritional programs Impact of PEPFAR programs on reproductive health and family planning • Impact of counseling and testing on pregnancy care • Impact of ARV treatment on fertility and orphanhood 

 EVALUATING THE IMPACT OF PEPFAR Coordination and harmonization of the evaluation process can pro- vide the benefits of mutually influencing others’ work, minimizing transac- tion costs, ensuring more efficient use of funds, and bringing to bear the strengths and perspectives of key stakeholders. Partner countries, imple- menting partners, and beneficiaries are among the perspectives that are critical to evaluation design, workshop participants said. Partner countries add value because they are accountable to their citizens and have experience in dealing with the challenges of service delivery. Implementing partners offer a familiarity with program data and lessons, along with a keen un- derstanding of the challenges of delivering services. Local people add value to the evaluation process because of their deep contextual knowledge and expertise about program impact at the community level. Although the process of coordination can be very time-consuming, there is great potential for coordination and harmonization to improve the sharing of data, approaches, and evaluation research among partners. Workshop participants highlighted key opportunities to share the out- comes of PEPFAR’s evaluation efforts with the 5-year evaluation of The Global Fund, a planned evaluation by UNAIDS, and evaluative efforts by the World Bank, the Organisation for Economic Co-operation and De- velopment, and the World Health Organization. Participants also noted that opportunities to harmonize the design, conduct, and interpretation of evaluation results with country-level partners include the conduct of joint field evaluations by collaborating partners; the development of centralized funding, knowledge management, and data aggregation systems; and dis- cussions leading to consensus among partners on monitoring approaches and overall program objectives. Use of Evaluation to Build Local Capacity The evaluation effort is an element of the larger effort to build capacity in partner countries so that they have, going forward, a lasting capacity to respond to their own epidemics, workshop participants said. Many par- ticipants reflected on the fact that evaluators often rely on their own data collection and evaluative capacities instead of helping countries to develop their own. Impact evaluation is most constructive when it is done in a way that builds countries’ capacities to collect, analyze, and use program infor- mation, thereby strengthening the sustainability of programs. Participants noted, however, that multiple constraints stand in the way of building local evaluative capacity, including weak or absent country-level systems for gathering data, funding mechanisms that limit the prioritization of monitoring and evaluation, competition for resources between evalua- tion and implementation, poor engagement of stakeholders in defining the 

OVERVIEW  monitoring agenda, and a tendency of researchers to drive a more narrow and less useful evaluation agenda. Opportunities for strengthening local capacity to conduct impact evalu- ation include the dissemination of methodologies, provision of technical as- sistance, recruitment and training of personnel in evaluation methods, and promotion of country-driven priority-setting processes for evaluation. Designing an Evaluation Based on Robust Methodologies Workshop participants stressed the importance of designing impact evaluation on the basis of a logical conceptual approach and robust meth- odologies. Case studies of evaluations of HIV/AIDS interventions were presented, and challenges and opportunities in evaluating impact were discussed. A number of general principles or observations by participants emerged from the discussions: • Prioritization is needed to narrow down what needs to be mea- sured. For long-term evaluations, for example, only those issues common to all projects might be selected. For a large portfolio of activities, a more narrowly defined set of indicators might be selected. Information about who needs the evaluative information and when can inform the prioritiza- tion process. • Formative or “learning” evaluation is a common component of many of the evaluations discussed, that is, ongoing evaluation to improve programming and to inform decision making, as opposed to evaluation at the end of a program to judge the success or failure. Negative evaluation results also offer value for learning and thus should be as widely dissemi- nated as positive results. • Multiple methodologies were used in many of the evaluations discussed and can provide richer results than just one or two methods. Many evaluations used both qualitative and quantitative methodologies. These included case studies, working papers, interviews, models, literature reviews, surveys, field work, participatory approaches, and theory. Multiple methodologies can be more valuable if selected strategically to complement each other. • Randomization is a powerful technique that can add value and credibility to evaluation. Although perceptions persist that randomization is difficult to implement and impractical at the country level, new methods are available to more easily incorporate randomization into a study. • Consultation and communication are an important part of the evaluative process. Changes resulting from the evaluation may have more 

10 EVALUATING THE IMPACT OF PEPFAR to do with the communications and consultations used during the process than with the actual results of the evaluation. Consultation is an important part of understanding what decision makers want from evaluation. • Limitations of data and models need to be well understood. The data collection systems in many partner countries are weak, and empirical data are often inadequate or inaccurate. Similarly, models have limitations that need to be understood. Models need to be validated with empirical data and made more accurate through the addition of variables. • Early design of the evaluation is critical to ensuring that the design is appropriate and that impacts can be detected early. Early design is espe- cially needed to facilitate the use of randomized approaches. • Comparison across contexts is an important attribute of evalua- tions, but change may be highly contextual. Workshop participants noted that success in one context may not necessarily be transferable to another. Factors independent of program interventions may have a significant influ- ence on change. Methodological Challenges and Opportunities in Evaluation Challenges and Opportunities in Measuring HIV/AIDS-Specific and General Impacts Workshop participants explored the limitations of commonly used methods in evaluation, as well as new prospects, in measuring both HIV/AIDS-specific and more general impacts. These are summarized in Table O-1. Challenges and Opportunities in Attributing Impact Attribution—relating the impact of a particular investment to a particu- lar donor—is one of the greatest methodological challenges in impact evalu- ation, workshop participants said. It is extremely difficult to tease out the exclusive impacts of efforts of any one donor from those of others given the number and diversity of programs and funders working in the area of HIV/ AIDS. Evaluating donor-specific attribution, however, may not be construc- tive. Several participants said that it is perhaps more useful to determine, using evidence-based approaches, what interventions are most effective and then to judge donors by whether they invest in those approaches. Challenges and Opportunities in Aggregating Evaluation Results The statistical synthesis or aggregation of the results of multiple stud- ies is a methodological frontier, workshop participants said. Meta-analysis, 

OVERVIEW 11 which combines results from multiple studies as if they were a single large study, is a tool that is currently underdeveloped for application to impact evaluation. Multiple analyses also have value in the independent validation of results. 

12 EVALUATING THE IMPACT OF PEPFAR TABLE O-1  Challenges and Opportunities in Measuring HIV/AIDS- Specific and General Impacts HIV/AIDS-Specific Impacts Metric Definition HIV prevalence The proportion of individuals within a population infected by HIV. Prevalence is a function of both the death rate of those infected and the rate at which new infections occur. HIV incidence The number of new cases of HIV within a population at risk over a given period of time. Infections averted The difference between expected and actual annual incidence. 

OVERVIEW 13 Challenges Opportunities • Measurement through testing • New tools that overcome some of the of pregnant women at antenatal limitations of HIV prevalence measurement clinics (ANCs) tends to through ANC surveillance have been overestimate prevalence because developed. ANCs are urban. • A second population-based survey of HIV testing will soon become available, allowing further analysis of prevalence. • Respondent-driven sampling methods are being developed for examining prevalence in high-risk groups. • Longitudinal cohort studies may • An adjustment formula for the HIV incidence not reflect the true incidence laboratory assay has been developed. in the population, and many • A more specific laboratory assay will improve participants in cohort studies the ability to distinguish long-term and recent may be lost to follow-up. infections. • Laboratory assays used to • A new population-based survey will provide distinguish recent infections important age-, sex-, and geography-specific from long-term infections tend incidence information. to overestimate the proportion • Modeling from prevalence data and of most recent infections. accounting for survival of infected individuals • Modeling tools are limited can be used to calculate incidence. in their ability to accurately measure population-level risk. • Empirical data on incidence by age and sex are lacking. • Infections averted are a • A new model called Spectrum takes “nonevent,” and their epidemiological contextual factors into measurement requires multiple account. assumptions. • A Futures Group model can be used to • Population projection modeling attribute infections averted to specific has limitations because of interventions. the gaps in data available in • Serial HIV population surveys can help to developing countries. deduce changing incidence and infections • Models may not account for over time. epidemiological contextual • Cross-country comparative analyses of HIV factors. dynamics and intervention uptake can be used to measure the relative effectiveness of interventions in averting infections. Continued 

14 EVALUATING THE IMPACT OF PEPFAR TABLE O-1  Continued Metric Definition Survival and mortality rates Mortality rate: the ratio of deaths in an area compared to the population of that area per unit of time. Survival rate: the percentage of people in a study or treatment group who are alive for a given period of time after diagnosis or treatment. Behavioral change Modification of sexual, injection, and drug-adherence practices. 

OVERVIEW 15 Challenges Opportunities • Measuring overall mortality • Mortality rate data quality can be improved change in response to by aggressively pursuing information treatment is challenging on those patients lost to follow-up and because increased survival of by standardizing methods for collecting HIV-infected individuals and information on deaths from hospital records. increased opportunities for • Use of “verbal autopsies” can be used to viral transmission to others follow up on deaths in households and decrease and increase mortality, determine the cause of death. respectively. • Corporate-sector surveillance systems can • Mortality data are poor provide early indicators of the impact of because of weak and inaccurate treatment programs on mortality. mortality surveillance systems • Age-specific and population-based mortality and loss of patients to follow- data can be gathered for improved up. Cause-specific and cohort- measurement of mortality impact. specific survival data are lacking. • Models frequently use only vital registration, population-level data. • Data and surveillance gaps • Change at the social and institutional levels exist, particularly in behavioral to build and sustain infrastructure for risk surveillance with biomarkers. reduction can be tracked. • Surveys and simulation models • Opinions of leaders, impediments to do not illuminate why specific behavioral change, and unintended negative populations are affected consequences of behavioral change can be differently. measured. • Current methods might not be • Behavior surveys, particularly those targeting able to determine the extent and younger people, who are an early indicator coverage of behavioral change. of prevalence changes, can be useful for Incomplete behavioral change attributing changes in HIV incidence to can have worse consequences specific changes in risk. than no behavioral change. • Models combining trends in prevalence and • Factors independent of the incidence with studies of risk behavior can be program intervention may a useful tool for retrospectively understanding influence behavioral change. how interventions might have worked to maximize declines in HIV prevalence. Continued 

16 EVALUATING THE IMPACT OF PEPFAR TABLE O-1  Continued Metric Definition Stigma and discrimination Negative attitudes, beliefs, and actions toward people who are perceived to have HIV/AIDS and those associated with them. Orphanhood prevention Prevention of the death of one or usually both parents of a child. 

OVERVIEW 17 Challenges Opportunities • Rigorous research and data • An International Planned Parenthood collection approaches are Federation stigma index is now available. absent. Most of the literature Another instrument, reflecting 33 factors is based on anecdotal evidence, measuring people’s perceptions, has also testimonials, and a few proved reliable for measuring stigma. qualitative studies. • New sources of data from focus groups have • Absence of scales to measure been useful in assessing stigma. stigma and its effects and tools to measure the effectiveness of strategies for mitigating stigma. • Measurements do not • New models are under development to distinguish between children better quantify the impact of treatment and who have lost one parent (single prevention in preventing the orphaning of orphans) and children who children. have lost both parents (double • Useful indicators, such as “years of orphans) to HIV. orphanhood averted” and “number of • Treatment has an unclear children who reach age 18 before the impact on orphanhood. death of a parent whose life is extended by Treatment of HIV-positive antiretroviral therapy (ART),” have been orphans extends years of developed. orphanhood, and while treating HIV-positive parents can reduce orphanhood years of existing children by prolonging parents’ lives, it can also generate years of orphanhood among children who are born to HIV-positive parents during treatment. • Methods do not exist for conducting cost-effectiveness analysis on interventions to prevent orphanhood. Continued 

18 EVALUATING THE IMPACT OF PEPFAR TABLE O-1  Continued Metric Definition Development of drug resistance The evolved capability of HIV to withstand a drug to which it was previously sensitive. General Impacts Metric Definition Health systems strengthening Improvement of a broad range of factors related to health care service delivery, including accessibility, quality, efficiency, and equity of services; management; procurement and distribution systems; human resource use; policy environment; and infrastructure. Health care workforce Improvement of a range of capacities related to strengthening health care personnel, including training, supervision, and job satisfaction. 

OVERVIEW 19 Challenges Opportunities • The threshold survey can be used to assess transmitted HIV infection using blood tested at ANC sentinel surveillance sites. Blood sampled from young women (age 25 or younger) in their first pregnancies who are likely not to be in ARV treatment can be used to track the transmission of drug-resistant HIV strains. • Therapy monitoring can be used to measure drug resistance by sampling and monitoring patients in ARV treatment from the initiation of therapy over a 1-year time period. Indicators of drug resistance such as outcome, viral load, and drug adherence can be monitored. Challenges Opportunities • Effects on the health system can • Facility surveys, provider surveys, and be positive or negative, intended qualitative interviews can be used to measure or unintended. a variety of attributes of the health system. • Health systems represent a • The quantity of non-HIV health services diverse set of institutions that delivered before and after the introduction may or may not be easily of basic HIV care can be compared, using compared. regression analysis to control for independent • Health systems include a diverse effects. range of elements. • Empirical estimates of impacts are lacking. • The small sample size and short time interval over which change is often evaluated limit many studies. • Impact attribution is difficult in this type of analysis. • A range of new indicators could be developed to measure impacts of interventions on capacity development, training and supervising effectiveness, gender equality, competencies, etc. • A system could be created to track health care workers over time, from registration to retirement. • Methods could be developed to evaluate the degree to which interventions strengthen institutions that regulate the workforce (that is, accrediting associations). Continued 

20 EVALUATING THE IMPACT OF PEPFAR TABLE O-1  Continued Metric Definition Effectiveness of complementary Effectiveness of programs complementary to interventions more narrowly focused HIV services, including interventions in areas such as malaria, tuberculosis, nutrition education, food security, social security, education, child survival, family planning, reproductive health, medical training, health systems, and potable water. Effectiveness of gender-focused Improvements in gender equality and women’s activities empowerment. Effectiveness of coordination and Increased alignment of HIV/AIDS interventions with harmonization country-level plans and coordination of efforts among other implementing partners. Effectiveness of community- or Improved service delivery for specific populations, population-level service delivery that is, children, families, communities, HIV-infected groups, high-risk groups, etc. 

OVERVIEW 21 Challenges Opportunities • Prospective randomized evaluation can be used to compare later program enrollees to earlier program enrollees by monitoring a range of indicators (that is, education and health indicators). • Given the multidimensional, • The Gender Equitable Men’s (GEM) scale can open, complex, nonlinear, and be used to look at gender norm attitudes and adaptive nature of gender, how they change over time. The scale is an it is difficult to define what index of 24 items, including home and child constitutes success. care, sexual relationships, health and disease • Few outcome evaluations and prevention, violence, homophobia, and few tools have been developed relations with other men. on how gender-focused activities affect HIV risk, and few good indicators exist that are useful in understanding social dynamics. • Evaluations of gender activities tend to underrepresent the perspectives of local people. • The Country Harmonization and Alignment Tool (CHAT) can be applied to the standardization of approaches for alignment of interventions with country-level plans and coordination of efforts among partners. • Community-level program information reporting systems (CLPIR) have been developed to examine community-level service delivery and help answer questions such as when, how, and where people want testing and treatment.