Informing Priorities for PEPFAR-Supported Programs
• Despite some exceptions, PEPFAR has implemented evidence-informed programs that have been modified as new knowledge and scientific evidence emerged. Target setting has been used to focus PEPFAR activities, program planning, and accountability. PEPFAR has utilized epidemiological and intervention effectiveness data to drive program activities.
• PEPFAR has provided financial and technical support for collecting epidemiological information in partner countries. This was widely seen as a positive contribution to inform decisions and priorities in planning the HIV/AIDS response and implementing HIV programs, encouraging and facilitating responsiveness to the epidemic and the needs in partner countries.
Program Monitoring Data
• PEPFAR’s program monitoring indicator system has faced technological challenges limiting the ability of both PEPFAR and external stakeholders to utilize and access both current and historical trend data; resolving these challenges is critical for successful program monitoring.
• PEPFAR’s program monitoring has evolved over time: the number of centrally reported indicators was reduced, indicators to monitor new program activities were introduced, and indicators identified as problematic, removed. PEPFAR needs a program monitoring strategy that can adapt over time to respond to feedback, reflect emerging program priorities, and accurately capture program activities and outcomes. However, this needs to be balanced with the reality that changes in indicators place a burden on partner country programs and limit comparability of data, hampering the ability to monitor trends.
• PEPFAR’s current indicators do not capture sufficient information on its stated prioritized goals and activities and are focused primarily on
input and outputs. As a result, the program monitoring system has limited utility for determining the effectiveness of PEPFAR’s efforts.
• The need to quickly measure results at the onset of PEPFAR contributed to the development of PEPFAR-specific data collection systems, which has limited harmonization with partner countries and the global HIV/AIDS community. More recently, the Office of the U.S. Global AIDS Coordinator (OGAC) has worked with other global actors to harmonize indicators and validate reporting. OGAC has modified the PEPFAR monitoring system to reduce reporting burden and improve alignment with partner country programs; however, further modifications could be made by eliminating PEPFAR-specific language in the indicator guidance; further reducing the reporting burden; improving indicator harmonization with global indicators; and advancing alignment with partner country data collection at the program level.
• There are some good examples of PEPFAR data use at the implementing partner, mission team, and headquarters (HQ) levels, but the preponderance of data collected does not seem to be routinely utilized. PEPFAR’s requirement for collection and reporting of a large amount of program monitoring data places a large burden on implementing partners and mission teams that has limited the ability to analyze and use data.
• PEPFAR has invested in building the capacity of partner countries to plan for, collect, manage, and use HIV data, which has implications for the larger health system. As a result, PEPFAR has contributed to fostering a culture of evidence among partner countries.
PEPFAR-Supported Evaluation and Research
• The manner in which PEPFAR initially approached research activities was a missed opportunity to establish, from its inception, mechanisms to evaluate programs, assess impact, contribute to the global knowledge base, and develop in-country research capacity.
• PEPFAR has made progress in carrying out evaluation and research activities over time: moving from an early proscription against research, to using Targeted Evaluations and Public Health Evaluations to work within research restrictions, to the recent creation of what holds promise as a more useful process for establishing priorities, managing activities, documenting “what works,” expanding PEPFAR’s technical leadership, disseminating findings, and continually improving the effectiveness and impact of PEPFAR. Defining appropriate and allowable research activities within PEPFAR, however, was and remains a challenge, specifically clarity around the activities and aims for evaluation and research within PEPFAR.
Knowledge Transfer and Learning Within PEPFAR
• PEPFAR has successfully established and used a variety of mechanisms to transfer knowledge throughout PEPFAR; however, more progress is needed to address limitations in current systems and to establish formal mechanisms to systematically transfer experiences across countries, implementing partners, and sites. Without this, there will be missed opportunities to capitalize on best practices and internal lessons learned.
Knowledge Dissemination External to PEPFAR
• OGAC would benefit from developing a formal system to track and manage PEPFAR-funded dissemination products (e.g., publications, reports, abstracts, guidelines, and tools) from which to measure contribution to the global knowledge base, and the global HIV/AIDS community would benefit from a publicly available central repository of these products from which to share, collaborate, and accelerate knowledge creation.
• PEPFAR has had some success in external dissemination of PEPFAR knowledge, including establishing formal and informal mechanisms to share knowledge externally and contributing vast amounts of evidence and publications to the global knowledge base. Despite this, more progress is needed to develop routine formal mechanisms for knowledge exchange with partner country governments and other partners, increase the amount of PEPFAR data that is publicly available for use by researchers and evaluators, and track and measure PEPFAR’s contribution to the global knowledge base.
• PEPFAR has made progress in managing knowledge by developing systems for data creation and collection, streamlining program monitoring data, advancing PEPFAR’s role and approach to evaluation and research, and utilizing a wide variety of mechanisms to transfer knowledge. Yet, like other entities involved in the global HIV/AIDS response, it struggles with creating, acquiring, and transferring the right knowledge, at the appropriate scale, and in a manner that facilitates use. PEPFAR has the potential to lead the global HIV/AIDS community in knowledge management by adopting a conceptual framework that articulates the vision, purposes, intended audiences, and goals of knowledge; how knowledge will be acquired, created, transferred, used, and disseminated to achieve these goals; and the complementary roles of program monitoring, evaluation, and research. PEPFAR has the opportunity to optimize program efficiency and effectiveness through an improved strategy that (1) streamlines and focuses knowledge creation within PEPFAR; (2) increases acquisition of knowledge external to PEPFAR; (3) improves the ef-
ficiency and effectiveness of knowledge transfer within and external to PEPFAR; and (4) institutionalizes the use of knowledge to improve the way work is accomplished.
Recommendations Presented in This Chapter
Recommendation 11-1: The Office of the U.S. Global AIDS Coordinator (OGAC) should develop a comprehensive knowledge management framework, including a program monitoring and evaluation strategy, a prioritized and targeted research portfolio, and systems for knowledge dissemination. This framework should adapt to emerging needs to assess PEPFAR’s models of implementation and contribution to sustainable management of the HIV response in partner countries.
This knowledge management framework will require that PEPFAR implement and strategically allocate resources for the following:
A. To better document PEPFAR’s progress and effectiveness, OGAC should refine its program monitoring and evaluation strategy to streamline reporting and to strategically coordinate a complementary portfolio of evaluation activities to assess outcomes and effects that are not captured well by program monitoring indicators. Efforts should support innovation in methodologies and measures where needed. Both monitoring and evaluation should be specifically matched to clearly articulated data sources, methods, and uses at each level of PEPFAR’s implementation and oversight.
B. To contribute to filling critical knowledge gaps that impede effective and sustainable HIV programs, OGAC should continue to redefine permitted research within PEPFAR by developing a prioritized portfolio with articulated activities and methods. The planning and implementation process at the country and program level should inform and be informed by the research portfolio, which should focus on research that will improve the effectiveness, quality, and efficiency of PEPFAR-supported activities and will also contribute to the global knowledge base on implementation of HIV/AIDS programs.
C. To maximize the use of knowledge created within PEPFAR, OGAC should develop systems and processes for routine, active transfer and dissemination of knowledge both within and external to PEPFAR. As one component, OGAC should institute a data-sharing policy, developed through a consultative process. The policy should identify the data to be included
Further considerations for implementation of Recommendation 11-1A: Program monitoring and evaluation
• OGAC’s current tiered program monitoring indicator reporting structure (illustrated in Figure 11-10) should be further streamlined to report upward only those indicators essential at each PEPFAR level:
o Tier 1: A small set of core indicators, fewer than the current 25, to be reported to central HQ level. These data should be used to monitor performance across PEPFAR as a whole, for congressional reporting, and to document trends; as such these indicators should remain consistent over time. Whenever possible and appropriate, these indicators should be harmonized with existing global indicators and national indicators; therefore, some centrally reported indicators will reflect PEPFAR’s contribution rather than aim to measure direct attribution.
o Tier 2: A larger menu of indicators defined in OGAC guidance, from which a subset are selected for their applicability to country programs to be reported by implementing partners to the U.S. mission teams but not routinely reported to HQ. These data should be used to monitor the effectiveness of the in-country response and to support mutual accountability with partner countries and their citizens. These data could be considered for occasional centralized use to inform special studies or respond to congressional requests but aggregation and comparability across countries may be limited in this tier as all mission teams may not collect the same data.
o Tier 3: Indicators selected by implementing partners to monitor and manage program implementation and effectiveness that are not routinely reported to mission teams. Implementing partners should select appropriate indicators defined in OGAC guidance and augment these with other indicators as needed for their programs. Implementing partners should work with mission teams in developing their program monitoring plans with selected indicators. Mission teams should provide oversight and technical assistance to ensure implementation of these plans and to promote local quality data collection, use, and mutual accountability. Although not routinely reported, some of these data could be considered for occasional country-level and centralized use.
o OGAC should create mechanisms for implementing partners, mission teams, and agency headquarters to mutually contribute to a periodic review across all tiers of indicator development, applicability, and utility and to make modifications if necessary.
o Tier 1 indicators should be harmonized whenever possible and appropriate with existing global indicators and national indicators. For indicators that are not routinely reported centrally (Tiers 2 and 3), country program planning should facilitate alignment of
indicator selection and data collection with partner country HIV monitoring and health information systems.
• OGAC should complement program monitoring with a unified evaluation portfolio that includes periodic program evaluation at the PEPFAR country program and implementing partner levels to assess process, progress, and outcomes as well as periodic impact evaluations at the country, multi-country, and headquarters levels.
o OGAC evaluation guidance should provide information about prioritizing areas for evaluation, the types of evaluation questions, methodological guidance, potential study designs, template evaluation plans, examples of key outcomes, and how evaluation results should be used and disseminated. PEPFAR should support a range of appropriate methodologies for program evaluation, including mixed qualitative and quantitative methods, and should shift emphasis from probability designs to plausibility designs that provide valid evidence of impact.
To allow for some comparability across countries and programs, OGAC and HQ technical working groups should, with input from country teams, strategically plan and coordinate a subset of evaluations within programmatic areas that include (but are not limited to) a minimum set of centrally identified and defined outcome measures and methodologies.
Within PEPFAR-supported evaluation activities there should be an emphasis on the use of in-country local expertise to enhance capacity building for program evaluation and contribute to country ownership.
• For both program monitoring and evaluation OGAC should continue its work on defining and developing measures to assess progress in the currently under-measured areas of country ownership, sustainability, gender, policy, capacity building and technical assistance.
Further considerations for implementation of Recommendation 11-1B: Research
• OGAC should clearly define which activities and methodologies will be included under the umbrella of PEPFAR-supported research, as distinguished from program evaluation.
• OGAC should draw on input from implementing agencies, mission teams, partner countries, implementing partners, the Scientific Advisory Board, and other experts to identify and articulate research priorities and appropriate research methodologies. The research proposals and funding mechanisms should be designed to ensure that these priorities are met and that methodologies are applied through requests for applications and other investigator-driven research pro-
posals as well as through targeted solicitations of research in gap areas not met through open requests.
• Given PEPFAR’s legislative and programmatic objectives to support research that assesses program quality, effectiveness, and population- based impact; optimizes service delivery; and contributes to the global evidence base on HIV/AIDS interventions and program implementation, at the time of this evaluation the committee identified the following gaps in PEPFAR’s research activities:
o Behavioral and structural interventions, especially in areas such as prevention, gender, nonclinical care and support, care and support for orphans and vulnerable children, and treatment retention and adherence. These research activities should employ appropriate methodologies and study designs, without being unduly limited to random assignment designs.
o Costs, benefits, and feasibility of integrating gender-focused programs with clinical and community-based activities.
o Health systems strengthening interventions across the World Health Organization building blocks, with a prioritized goal of determining setting- and system-specific feasibility, effectiveness, quality of services, and costs for innovative models.
• To contribute to country ownership, PEPFAR should facilitate in- country local participation and research capacity building through simplified, streamlined, and transparent application and review processes that encourage submissions from country-based implementing partners and researchers.
Further considerations for implementation of Recommendation 11-1C: Knowledge transfer and dissemination
• The knowledge created within PEPFAR that should be more widely documented and disseminated includes program monitoring data, financial data, research results, evaluation outcomes, best practices, and informal knowledge such as implementation experience, and lessons learned.
• To institutionalize internal and external knowledge transfer and learning, PEPFAR should develop appropriate systems and processes for the most needed types and scale of knowledge transfer. To achieve this, PEPFAR should draw on broad stakeholder input to assess the strengths and weaknesses in current processes and to identify needs and opportunities for improved knowledge transfer.
• PEPFAR should invest in innovative mechanisms and technology to facilitate knowledge transfer across partner countries and implementing partners. Mechanisms currently used successfully on a small scale and an ad hoc basis could be formally scaled up across PEPFAR.
OGAC should also look to other organizations with wide geographic reach and organizational complexity, such as multi-country PEPFAR implementing partners, other large global health initiatives, and global corporations, for models of successful knowledge transfer systems.
• OGAC should develop a policy for data sharing and transparency that facilitates timely access to PEPFAR-created knowledge for analysis and evaluation. The purpose of this policy would be to ensure that, within a purposefully and reasonably defined scope, specified program monitoring data and financial data, evaluation outcomes, and research data and results generated with PEPFAR support by contractors, grantees, mission teams, and U.S. Government (USG) agencies be made available to the public, research community, and other external stakeholders. OGAC and the PEPFAR implementing agencies should consult with both internal and external parties who would be affected by this policy to help identify the data that are most critical for external access and that can be reasonably subject to data-sharing requirements, as well as to help develop feasible mechanisms to implement a data- sharing policy.
o For routinely collected financial and program monitoring data, a limited set of essential data should be identified and made available for external use in a timely way.
o Evaluation and research reports and publications using data collected through PEPFAR-supported programs should be tracked and made available in a publicly accessible central repository. USG agencies with similar repositories can be considered as models.
o For research data and other information that is expressly generated for new knowledge, the policy should respect time-bound exclusivity for the right to engage in the publication process, yet also ensure the timely availability of data, regardless of publication, for access and use by external evaluators and researchers. OGAC should look to USG agencies with similar research data policies as models.
o In developing the policy and specifying the scope of data to be included, several key factors and potential constraints that can affect the implementation of the policy will need to be addressed. These include patient and client information confidentiality; the financial resources, personnel, and time needed to make data available; and issues of data ownership, especially in the context of increasing responsibility in partner countries and the provision of PEPFAR support through country systems or through activities and programs supported by multiple funding streams.