Skip to main content

Evaluation of PEPFAR (2013) / Chapter Skim
Currently Skimming:

Appendix C: Evaluation Methods
Pages 741-792

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.


From page 741...
... Global AIDS Coordinator (OGAC) and PEPFAR implementing partners, benchmarking progress against stated programmatic targets and goals, reviewing extensive documents, and analyzing primary data collected through more than 400 semi-structured interviews with a range of stakeholders on visits to 13 PEPFAR partner countries, at the U.S.
From page 742...
... For example, analysis of country visit interview data was limited to the countries selected for visits by the committee. In addition, some analyses drew on existing data sources that were available only for some countries, programs, and partners, such as Track 1.0 partner data.
From page 743...
... • The type of data necessary to answer the question (e.g., financial data; program monitoring, surveillance, and clinical data; qualita tive interview data; literature and document review) • A description of potential data sources that had been identified • Limitations associated with the data sources, such as issues related to availability, the feasibility of accessing the data, and any other
From page 744...
... The categories of available data sources that were mapped and ultimately used for the evaluation included financial data; program monitoring, surveillance, and clinical data; qualitative interview data; and literature and document review. The sources included central OGAC data, data from multilateral organizations, data from implementing partners, and data from publicly available documents and other sources.
From page 745...
... . PEPFAR Financial Data: Annual Expenditure Data Calculated from Agency Reporting In May 2012, in response to a committee data request, SCI received from OGAC PEPFAR funding obligations and outlays for FYs 2004 through 2011 for all countries receiving PEPFAR funding.
From page 746...
... Data Description OGAC sent 78 Excel spreadsheets containing financial data for the 6 agencies that received PEPFAR funding between FY 2004 and FY 2011: • Department of Defense (DOD) • Department of Health and Human Services (HHS)
From page 747...
... For all three of these situations -- embedded comments, footnotes, or unexpected changes in funding -- the following information was recorded for each instance: • Agency/bureau -- which agency's spreadsheets contained the com ment, footnote, or inconsistency • Country/region -- the country or region affected by the comment, footnote, or inconsistency • Reporting year -- the reporting year with the observation • Budget year -- the year during which the budget was issued • Comment, footnote, or inconsistency -- verbatim comments and footnotes from the spreadsheet; inconsistencies were described as clearly as possible • Detected by agency or IOM -- an indicator variable reflecting whether the comment or footnote was already in the spreadsheet or whether the inconsistency was encountered by IOM staff during the data extraction process To further assess the most notable discrepancies in the available totals by country and by year, these were compiled in a separate spreadsheet and compared from the inception year through 2011. In particular, major discrepancies occurred when the dollar amounts reported as available for a given budget year changed (both increases and decreases were observed)
From page 748...
... PEPFAR Financial Data -- Planned/Approved Funding for All PEPFAR Countries Planned/approved funding reflects how OGAC and PEPFAR mission teams plan to obligate and outlay funds. Each year, OGAC releases an operational plan for PEPFAR that includes summary budget information 2  SAS and all other SAS Institute Inc.
From page 749...
... Each staff member extracted data on total PEPFAR funding by implementing agency and year, as well as total PEPFAR funding by budget code and year. Data Presentations The data extraction was validated and SCI converted the data into constant 2010 USD to allow for a consistent interpretation of funding over time.
From page 750...
... Data Extraction Quality Control Two SCI consultants extracted the data independently into comparably formatted spreadsheets. Each consultant extracted a spreadsheet of funding data by year, country, and agency, as well as a second spreadsheet by year, country, technical area, and budget code.
From page 751...
... The committee chose to compile these data for the same 13 countries purposefully selected for country visits, as described later in this appendix. Within this subset of countries, the committee was able to compare partner data and planned/approved PEPFAR funding for the focus countries for FY 2004 through FY 2010 and for non-focus countries for FY 2008 through FY 2010.
From page 752...
... Before releasing the dataset, CGD added data on central funding that is obligated from OGAC headquarters to partners to implement programs in countries. For FY 2004 and FY 2005, CGD obtained central funding information from the Center for Public Integrity; FY 2006 funding was estimated by CGD based on previous funding amounts and the total allocation of PEPFAR funding for focus countries in FY 2006 (Oomman et al., 2008)
From page 753...
... Data Limitations The prime partner data compiled by IOM staff for the committee were limited by the incompleteness of the data sources, as described above. Overall, the total amount of partner funding compiled for this analysis reflects only 77 percent of the total planned/approved funding for this subset of countries for FY 2005 to FY 2010 (as reported in the operational plans)
From page 754...
... to reduce the number of PEPFAR-specific reporting requirements and, where possible, to align with globally harmonized and reported indicators in partner countries. As a result, starting in FY 2010 PEPFAR classified indicators according to three reporting levels: essential reported to OGAC HQ, essential not reported to OGAC HQ, and recommended (OGAC, 2009b)
From page 755...
... If a partner country has a signed Partnership Framework, the country is also required to report a 26th programmatic indicator to OGAC (H6.1.D)
From page 756...
... . The NGIs include eight primary indicators, a subset of the 25 programmatic indicators required to be reported to OGAC (see Table C-2)
From page 757...
... After the phone interview and again during the in-person interview, IOM staff and consultants formally requested from OGAC all centrally reported PEPFAR program monitoring data corresponding to FY 2004 through FY 2010. Data requests were not
From page 758...
... At the time of the initial data request, OGAC indicated that it would make only the core programmatic indicators described in the previous section (seven from FY 2004–FY 2009 and eight from FY 2010) available to IOM.
From page 759...
... Data Presentations Once SCI received the PEPFAR indicator data from OGAC, it created tabular and graphical presentations of the data over time using SAS software versions 9.2 and 9.3. Data presentations were provided to the committee in June 2011, September 2011, April 2012, and June 2012.
From page 760...
... The intent of this approach was to rapidly initiate and scale up prevention, care, and treatment services in PEPFAR focus countries. These partners reported to both in-country Mission Teams and to CDC and HRSA directly (McCullough and Miller, 2009; Sessions, 2006)
From page 761...
... Data Summary In April 2012, SCI received from CDC an Excel spreadsheet containing 7 years of quarterly, facility-level data (Q1 FY 2005 through Q4 FY 2011) from all four Track 1.0 partners, representing programs in thirteen PEPFAR partner countries.
From page 762...
... Relationship of Track 1.0 Data to OGAC Indicators This Track 1.0 dataset contained three variables that are reported to OGAC and that ultimately contribute to the following PEPFAR programmatic indicators (OGAC, 2009b) : • C2.1.D – Number of HIV-positive adults and children receiving a minimum of one clinical care service (Care and Support Sub Area, Clinical Indicator)
From page 763...
... Analyses SCI prepared data presentations for the committee pertaining to changes over time in enrollment into clinical care services and treatment services, active facilities, and persons trained. The disaggregated nature of
From page 764...
... Data presentations were provided to the committee in June 2012. Individual Track 1.0 Partner Data As described previously, during semi-structured interviews, teleconferences, and e-mail communication between March 2011 and October 2012, IOM communicated with all of the Track 1.0 partners individually to learn more about their programs and discuss the feasibility of their sharing data and analyses with IOM.
From page 765...
... GLOBAL DATA SOURCES In addition to pursuing PEPFAR-specific data through the centrallyreported indicators and the Track 1.0 partner data, IOM staff and consultants reviewed additional potential data sources for global HIV/AIDS data. As the committee finalized its PEQs for each content area, the committee work groups, staff, and consultants mapped data from these global data sources to the PEQs to determine which data to prioritize.
From page 766...
... The WHO Communicable Disease Atlas database contains reports, documents, and data on some of the major infectious diseases of poverty, although at the time of the data mapping for this evaluation it had no specific data on HIV. The WHO Global Atlas of the Health Workforce contains global data corresponding to the health care workforce, including community health workers and laboratory health workers.
From page 767...
... • Public health and environment (household and air pollution, out door air pollution, water, sanitation and hygiene, lead, second-hand smoke, UV radiation, climate change, occupational risk factors, total environment, and children's environmental health) International epidemiologic Databases to Evaluate AIDS (International epidemiologic Databases to Evaluate AIDS, n.d.)
From page 768...
... • Demographics (birth rate, crude death rate, and population under 18 and under 5) • Economics (gross domestic production [GDP]
From page 769...
... Ideally the committee would have liked to design a model to determine if a larger annual investment of PEPFAR funding, as a continuous variable over time, had led to a greater impact on health. The committee focused on a possible model of impact on adult allcause mortality with potential outcome and explanatory variables, using the country as the unit of analysis.
From page 770...
... . The committee also discussed modeling child mortality as a function of PEPFAR funding over time but decided against it due to the small number of countries where HIV is a sizeable contributor to childhood mortality (see Chapter 7 for a more in-depth discussion of child mortality in PEPFAR partner countries)
From page 771...
... This site contains estimates of adult mortality rates between ages 15 and 60 years, for both sexes separately and combined. It also provides estimates of the crude death rate (deaths per 1,000 people)
From page 772...
... In addition, information on PEPFAR funding disaggregated by partner countries is not publicly available. As described previously, after several iterative funding data requests, SCI received from OGAC cumulative country-specific and agency-specific funding reports from which annual expenditures by country could be manually derived.
From page 773...
... As part of the data collection process associated with the evaluation, the team's desk review included reviewing several types of documents: those providing PEPFAR-specific process, policy, and planning guidance documents; PEPFAR operational plans; reports from PEPFARsupported activities and evaluations; global guidance documents related to HIV; country-specific HIV/AIDS reports; reports from multilateral agencies and other organizations external to PEPFAR; and the peer-reviewed literature. The specific documents reviewed are referenced where the information gathered is used throughout the report.
From page 774...
... This section describes how these strategies were applied pragmatically in the evaluation. The process of collecting the country visit interview data is summarized at the end of this section in Figure C-1.
From page 775...
... week debriefings debriefing notes Country Individual memos for Brief exit synthesis Data coding Staff coding using N-Vivo 9 Team Exit Synthesis Notes and Summary of Key Messages Analysis and Interpretation Quantitative program monitoring, Country Visit Summary surveillance, and (Country-specific summary of data) clinical data Interim Analysis Report Financial data (Cross-country analysis of data; Preliminary findings)
From page 776...
... One additional country with similar characteristics within the original selection criteria was added to minimize the effects of the two cancellations on the selected sample. Interviewee selection process During each country visit, the evaluation team conducted qualitative interviews with key stakeholders involved in the HIV/ AIDS response.
From page 777...
... served as additional resources to identify service organizations, programs, and individuals relevant to the response within specific countries. IOM staff then initiated contact with these resources in an effort to further identify individuals with direct experience related to various elements of the HIV response in a given country and to schedule interviews and site visits for each country visit, except where protocol required communication and scheduling with partner country government officials through formal communications by the Mission Teams.
From page 778...
... Although the interviews conducted in each country were tailored to the particular country and its unique HIV-response strengths, challenges and attributes, the evaluation team aimed to systematically interview stakeholders serving in particular roles in every country visited. The roles of these individuals or groups of individuals included • U.S.
From page 779...
... Prolonged engagement afforded the evaluation team an opportunity to gain an in-depth understanding of the PEPFAR program in the context of each country visited. Country visits typically spanned 2 weeks, and qualitative data collection involved an average of 25 in-depth interviews with key stakeholders as well as several site visits in each of the selected countries.
From page 780...
... A subset of evaluation team members was present and engaged in qualitative data collection typically for a period of 12–14 days during each visit to a partner country, with a total in-country visit time across the selected countries of about 180 days. Thus, the team had a total of approximately 6 months of residence (or approximately 140 person-weeks for all of the committee, IOM staff, and committee consultants)
From page 781...
... Guides were developed by selecting and tailoring a subset of interview questions and follow-up prompts from a pre-established set of key country visit interview questions. The development of these pre-established questions was based on the Strategic Approach to the Evaluation of Programs Implemented Under the Tom Lantos and Henry J
From page 782...
... In the case of interviewees who preferred conducting the interview in a language other than English, the delegation team hired professional interpreters from the partner country and oriented them to the purpose and process of qualitative data collection and to their role in the process. Participant validation of data summaries A commitment to anonymity and confidentiality and a focus on cross-country data reporting precluded the sharing of country-specific findings with interviewees and their agencies or organizations during or after individual partner country visits.
From page 783...
... Evidentiary documents related to the process of the evaluation included • An agenda log maintained electronically for each country visit chronicled interview scheduling and contact information, evalu ation-related contacts, and information on the participants and questions covered in completed interviews. • An activity log maintained electronically throughout the evalua tion process chronicled process and methodological decisions and action items both within and across country visits.
From page 784...
... • Across-Country Debriefings and Discussions o eriodically, between clusters of country visits, IOM study staff P participated in a discussion and synthesis of the qualitative find ings according to evaluation topic and identified consonance or differences in these findings across a number of countries.
From page 785...
... Collection of Non-Country Visit Interview Data As part of the data collection effort for the evaluation, IOM staff and consultants also conducted a series of 32 non-country visit interviews with key stakeholders. The interviewees included the USG at PEPFAR HQ level (including OGAC, CDC, and USAID)
From page 786...
... The country visit summary also includes other information provided to the delegation in advance of the trip in the form of a country brief, including background research on the country context and PEPFAR program as well as basic financial data and OGAC and other programmatic or indicator data (including UNAIDS data)
From page 787...
... , the evaluation committee's development of priority evaluation questions, and the exit synthesis process and review of initial data collected from the pilot country visits and other early country visits. For synthesis and analysis, these coded data were separated and extracted by querying for a single code or combinations of the macro-level codes across interviews.
From page 788...
... In addition, interview debriefing and exit synthesis documents from all interviews, including those not in the initial coded dataset, were used to carry out supplementary deductive confirmation and disconfirmation of findings that emerged from the coded data, and to identify specific additional interview notes and transcripts for enrichment of the analysis of the coded data. These interview data findings and analyses were presented in a number of ways, including in narrative form with accompanying illustrative quotations, in summary tables, or in bulleted groupings by subconcepts.
From page 789...
... 2008. PEPFAR funding data full dataset.
From page 790...
... 2006b. The President's emergency plan for AIDS relief: FY 2007 country operational plan guidance.
From page 791...
... n.d.-b. Global health observatory data repository.


This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More information on Chapter Skim is available.