6

Measuring Performance and Progress in Public–Private Partnerships for Health Systems Strengthening

Multiple perspectives were shared on how both success and failure in partnerships for health systems strengthening have been defined and measured in the past, with the goal of illuminating opportunities for developing a shared vision among partners for what is valued and should be measured. Robert Bollinger from the Johns Hopkins Bloomberg School of Public Health stated that it is important at the beginning of a public–private partnership (PPP) to develop a shared vision, identify and define the shared values, and then recognize that as programs develop over time some of those metrics may change, so it may be important to revisit and redefine metrics as the program continues. The emphasis in this session was to explore which metrics matter for evaluating the effectiveness of PPPs.

DEVELOPING METRICS FOR HEALTH SYSTEMS

Sally Stansfield from Deloitte emphasized that when measuring the impact of a PPP on strengthening a health system, the most critical measurements will be assessing their effect on improving health outcomes. She has observed that it is the measurement and impact on health outcomes that will drive a shared commitment, and ultimately influence the resources, time, energy, and productivity committed by each partner.

The other domain of metrics to consider is the quality of the partnering process. Stansfield shared an example from the Global Malaria Action Plan that demonstrated how the organization considered measuring the

process of partnering. Some of the measures included reviewing the key players and sectors committed to the partnership and assessing the representativeness and the balance between the public and private sectors and the appropriateness to the task.

Stansfield stated that information is considered by the World Health Organization (WHO) as one of the six building blocks of the health system. However, she suggested it underpins it. Without the collection of high-quality information, the metrics are useless. Imagine managing human resources or a supply chain, or providing governance, or developing policies without meaningful information. At the core of developing metrics is the consideration of the quality of the information system through which data can be collected and used for decision making. When considering a new investment, Stansfield noted that metrics should be evidence driven. Indeed, metrics and the way in which they are used to measure progress and communicate to partners can build momentum and trust during the project.

While health information is critically important for measuring the impact of PPPs and demonstrating the value to each partner, Stansfield stated that this information is also tremendously valuable to the communities, patients, and customers who are involved in the project. As an example, Stansfield shared her experience conducting a study in Malawi that demonstrated the impact information can have on an entire community. In this study, villages were randomized to receive two different levels of information on how well they were doing with regard to utilization and coverage of life-saving maternal and child health services, such as bed nets and contraceptives. All villages received summary information of their collective progress, while half of the villages also received information about how their own village was doing and how five of their nearest neighbor villages were doing. The villages that received locally disaggregated information about how they were doing and how their neighbors were doing were empowered. They realized that the interventions were intervenable, it was changeable, and they could fix it. These villages took charge of their own health, and there was a nearly 50 percent increase in coverage and utilization in those villages relative to the ones who only received summary information. This study demonstrated the power of metrics, not just for the partnership and for mobilizing resources, but also for empowering and motivating people and communities to take charge of their own health.

Jo Boufford from the New York Academy of Medicine commented that metrics, such as the ones Stansfield described, can be motivational but, unfortunately, the rate-limiting step is often the capacity at the local and country levels to gather the information. Though vital statistics are generally maintained, other information routinely captured is highly vari-

able. Boufford also pointed out that metric systems often measure what is important now, but they may or may not be relevant to the future. Stansfield responded that there has been progress in country ownership of information systems. This ownership is, in part, a result of the new technologies that help with district health management, including open-source software packages such as DHIS 2 that can make the information readily accessible and useful. More is being learned about how to use the information-and-benchmark progress, such that there can be increased accountability and rewards for effective local interventions. Finally, Stansfield noted the growing focus on domestic resources mobilization, rather than continuous donor funding, is changing the dynamics, such that the data are owned at the country level and therefore the problems, the solutions, and the successful interventions are now being owned at the country level, too.

In terms of measuring relevant endpoints for shaping the future, Stansfield has observed that a big problem with the global architecture in health is that the funders tend to drive and shape what is measured. For example, the Centers for Disease Control and Prevention (CDC) conducts disease surveillance but not public health surveillance. As learned from the Ebola virus outbreak, CDC is not conducting surveillance for unexplained clusters of deaths or conducting surveillance for the unexpected. Stansfield stated that she would welcome a transition away from disease surveillance toward country-owned health information and targeted data collection for community issues.

Aye Aye Thwin from the U.S. Agency for International Development (USAID) also conveyed the importance of metrics for understanding how successful PPPs have been in improving health systems. Health outcomes are critically important she stated, but so are consumer satisfaction and the extent to which the partnerships can demonstrate improvements across the health system and ultimately impact the target population. Thwin added that the outcomes often used to measure the success of PPPs include examining effectiveness, efficiency, and equity. Other metrics include compliance and quality and whether or not partnerships have improved value and provided a benefit to the population.

She noted that it is important to develop an analytical framework that distinguishes contribution from attribution. When evaluating the success of the partnership, one must establish baseline measures and then examine changes over time. Thwin suggested that when evaluating incremental changes and near-time results, it is important to measure the periodic progress and allow for appropriate modifications. That said, flexible tools and systems are needed to measure and track progress. Indeed, there is a substantial need for evidence when evaluating investments for improving health systems.

Process metrics are also important to consider when evaluating the performance and progress of PPPs. Thwin shared her experiences in setting up PPPs and observed that there is often a lot of lag time in establishing the agreements, developing the shared values and vision, executing the plan, and tracking the results. It is important to measure the time for each step in the process and assess the reasons for delays. For example, did both partners have access to the metrics being captured and utilize them for process improvement?

Other metrics to include in the evaluation of successful PPPs are the investments that each partner garnered for the program, the alliance that was built, and the number of people impacted from the partnership. Specifically, evaluating the impact on equity is important, such as through the types of people who directly benefited. Was the entire health system strengthened, including reaching vulnerable populations? As an example, Thwin described her work on preventing drug-resistant malaria in the Mekong region of Asia. This is a region where, because of the job market, there is frequent cross-border passage (between Laos and Thailand). Although Thailand is a model for universal health coverage, there are almost 4 million of these migrant workers who are not accessing health services. She suggested partnerships are needed across countries to develop solutions that address these complex and often political health issues.

Finally, Thwin suggested it is important to measure the level of human resources required to deliver high-quality health care and improve health systems. USAID prioritizes setting up national health workforce accounts for both the public and the private sector. However, the metrics do not necessarily measure the ability to deliver high-quality care. She indicated the continued need to consider how new technologies can improve efficiencies and quality of care without necessarily increasing the workforce.

John Lange from the United Nations Foundation reflected on the complexities of reporting mechanisms that require a massive number of metrics and the associated expense of collecting and analyzing the metrics for each program. Lange agreed on the fundamental importance of metrics being evaluated for their utility because of the expense and effort required globally to use them; he wondered what coordination efforts are currently taking place by the European Commission, the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR), USAID, CDC, and others to address the issue that each agency has a separate, long list of indicators and metrics, which actually increases the burden on the countries receiving funding. Thwin responded that there are many ongoing efforts by USAID and others to streamline the information collected

and used. The Roadmap for Health Measurement and Accountability¹ is one such example as well as the PPPs metrics used in the past. There is also an active movement in USAID to start documenting interventions that were not successful.

Justin Koester from Medtronic spoke about the importance of PPPs and how to improve health systems on a greater scale. The key metrics he noted were patient outcomes over time, impact on quality of life, and overall patient satisfaction and customer service. In developing devices that are meaningful to patients, Koester noted that partnerships were critical for Medtronic to ensure that the health care infrastructure and health care delivery systems were of high quality to successfully deliver the devices. Indeed, to make a difference in patients’ lives, partnerships have been developed to improve the efficiency and delivery of care within operating rooms, cardiac catheter labs, and surgical centers. For example, a PPP in the Netherlands was developed to evaluate and improve patient diagnostic time at a hospital. Over a 6-month period, this partnership reduced the time of diagnosis from the moment a patient enters the facility to the time they are appropriately diagnosed from 2 weeks to 24–48 hours.

Koester suggested that metrics are needed at every level of the health care system. This includes primary care, secondary care, and tertiary care, as well as pre-hospital, in-hospital, and post-hospital care, and at every level of abstraction at the health system, as well as whether that is from the health facilities perspective, the payer’s perspective, or the overall health system’s perspective.

Metrics are most effective in driving change when they are transparently collected and shared. Koester noted that the example shared by Devi Shetty from Narayana Health is an excellent example of transparent metrics, whereby all physicians in the hospital receive daily profits and losses for their hospital and are held accountable for these metrics, which significantly impacts their daily decisions of health care treatment.

Katherine Taylor from the University of Notre Dame shared her perspective on metrics and accountability. She noted that the majority of metrics utilized in a university setting are in regard to conducting research and training; the metrics used to examine specific implementation programs can be quite different. Therefore, when considering PPPs from a university perspective, the research questions that might be embedded in the partnership may take on a whole other set of resources and needs, in terms of data collection and evaluation, than some partners are typically accustomed to experiencing. Reporting might be different, as well

___________________

¹ See http://ma4health.hsaccess.org/docs/support-documents/the-roadmap-for-healthmeasurement-and-accountability.pdf?sfvrsn=0 (accessed December 15, 2015).

as data collection, program design, impacts and outcomes measured, and timeframes.

Taylor suggested that when considering the appropriate metrics for measuring performance and progress in a PPP that is strengthening health systems, the goal should be to improve and sustain country-level health improvements and systems for accountability. This agenda was outlined in The Roadmap for Health Measurement and Accountability and a Five-Point Call to Action.² The Roadmap aims to ensure that countries have the necessary information and capacity to plan, manage, and measure their health programs, as well as monitor and achieve their national health goals and the health-related Sustainable Development Goals (SDGs). Taylor mentioned that many workshop participants have indirectly stated these same principles of research regarding what data get collected, how the data are analyzed, who has access and/or ownership of the data, appropriate use of the data to drive decisions, and that the data must all reside at the country level. She suggested that this concept will really change the nature of some of PPPs.

Questions about who is the data for, how will the data be used, what resources will be required to collect, store, share, and analyze the data are all important; but because resources are spent on collecting data, there must be a value that serves the health needs of the country. Taylor noted that too often research projects are just serving the data needs of the partners and, while those are important, the primary goal must be the health needs of the countries and communities.

Over the past 10 years, there has been a proliferation of global health organizations that have driven dramatic advances in improving health and health systems in many low-income countries. Taylor noted that these accomplishments have been primarily driven by vertical programs whereby the data collection and evaluations can be very specific for the purposes of the program. While the accomplishments have been important, the large numbers of health organizations involved have also driven a complexity in the donor environment and expectations. Indeed, a new minister of health in a low-income country lamented that too much time was spent on organizing donors, Taylor mentioned. With increasing donors and partners, Taylor suggested there is frequently a demand for more indicators and metrics with less transparency. A clear purpose for additional indicators and dissemination plans for results need to be clarified early in the process.

The risk is that the information flow becomes one-way and is not available for informing decisions about the local and national health systems. Taylor once observed a community health center with one little

___________________

² See http://ma4health.hsaccess.org/docs/support-documents/5-point-call-to-action.pdf?sfvrsn=0 (accessed December 15, 2015).

room and a stack of forms that needed to be filled in for each project, for each donor, and for the ministry, and a significant amount of resources are used to complete those forms. Reporting is conducted on a monthly basis, and little to no information returns to the clinic. This system is not sustainable or informative for improving health systems. Taylor encouraged the movement toward pushing the ownership of these research responsibilities to the country level and the required change in the paradigm over the next few years.

Margaret Kruk from the Harvard T.H. Chan School of Public Health stated that she has recently been working within sub-Saharan Africa, where many people are living on a few dollars per day, where health systems are spending USD 30 or USD 40 per person per year, not USD 8,000 per year as in the U.S. health system. She suggested, in a region such as sub-Saharan Africa, private partners are able to shift the way patients are perceived. She also spoke about the new perspectives that PPPs can bring to the delivery of health care, the functioning of health systems, and the promotion of global health goals.

Kruk shared three broad areas to which private partners bring a tremendous value to partnerships that aim to strengthen global health systems. The first area is a customer-first focus. Historically, global health provisions have not been focused on customer and patient satisfaction, but rather on delivery and access to health care. She argued that the focus has been on getting people to facilities at all costs. Whereas, the private sector is concerned with what happens once they get there. What is the patient’s experience? What is the customer’s experience? The philosophical difference may be that government partners think patient experiences are nice to have, while the private sector believes it is a must-have. Connecting with the patient builds trust and is the basis for greater adherence and continuity of care—all these important measures are often not addressed adequately but can improve health care and health systems. Populations in even the lowest-income countries are now facing chronic diseases as the primary burden of disease. Leveraging the experience and expertise from the private sector to improve loyalty, faith, and build trust among patients can ultimately improve the delivery of public health care.

Certainly, the issue of patient satisfaction ties directly into the Patient Protection and Affordable Care Act. The issue of developing a process for universal health coverage relies on a fundamental assumption that patients will support greater spending in health care and the purchase of insurance. Again, developing a system that patients want to use is essential. Kruk believes that trust and patient satisfaction are not simply side effects, they are one of the main jobs of the health system.

The second area of focus, she noted, is outcomes. Health care must demonstrate value by lengthening lives, improving quality of life, and reducing morbidity. In addition to these critical measures, however, it

is important to measure a patient’s own sense of the quality of his or her health. As people live longer, Kruk suggested that patient-reported outcomes on the global assessment of health issues become increasingly important. To demonstrate this point, Kruk spoke about the Oregon Health Study, which found that despite broad Medicaid health insurance coverage, hemoglobin A1C levels did not improve. In fact, many outcomes did not improve. But what was notable to Kruk, was what did improve. Depression scores improved and patients’ own assessment of their health improved. As a trained family physician, Kruk spoke about her experience working in northern Canada. There, she observed a sense of trust and security when the health system was working well and health services were covered. She suggested that improving the patient’s sense of his or her own health is an equally important outcome after reducing the hard outcomes.

Kruk concluded with a final set of metrics: examining the process for developing and delivering the program. She reminded attendees that many examples demonstrating the need to tackle the process have been shared; to open the “black box” and ask how do we reduce wait times, how do we improve queuing strategies, and how do we reduce diagnosis delays? The private sector can help the public systems be highly critical in examining the process for the delivery of care. Too often, Kruk suggested, failed public health programs continue without accountability and flexibility to end them quickly. Another area where private industries can assist in the process is around parsimony and transparency of metrics. Indeed, the number of metrics utilized within PPPs is flourishing, including massive lists of indicators. Kruk noted that metrics are expensive to collect, so they must demonstrate value just as any other health care service or investment. She also stated that partners must be diligent in reducing metrics to those that add value and that reporting is critical. Comparison among countries and among regions generates important peer pressures and positive effects, but the information needs to be provided both locally and regionally. Kuk notes that if the information does not trickle down in a readily useful form, the utility of that information will be limited. As an example, she described working on a field study in Tanzania where she and her team reviewed a 200-question survey. For each question, her team deliberated on the value of the metric. How will the measure be used? How will it be analyzed? If it was not deemed important enough, then the question was deleted. Kruk suggested that everyone needs to be parsimonious and diligent in the reduction of the burden of metrics, while simultaneously improving their utility. In determining which metrics are most useful, Kruk stated that patients and consumers will increasingly demand information—not just their own information, but also the aggregate information about the performance

of their local health system and their doctors. Patients should be better informed to make personal health decisions. Based on research conducted in Tanzania, Kruk’s team found that women who have a cell phone and who listen to radio or television are increasingly bypassing their local health facility to seek treatment within a hospital, and they are reporting better care there. They are not waiting around. Kruk stated that this is not a Manhattan consumer, “let me assure you.” This is often an illiterate person with very modest wealth to be able to do this, and yet, he or she is doing it. They are acting on the information they receive to seek better health care—a desire that is a global one in the health system, both here and in lower-income countries.

Patrick Kelley from the Institute of Medicine reflected on the discussion by stating that metrics seem to be used to manage health care, but also to motivate individuals and communities locally. Therefore, it is not only important to collect the right metrics, but also to identify the right people at the right time who can achieve the change with the information collected. To which Kruk agreed and stated that for any enterprise partnership to be sustained, a useful and cost-effective set of metrics must be developed and delivered in a timely manner. She stated that we must ask, “[are the data] reaching the right people on time, and is it the smallest set of data that we need to create change?”

METRICS FOR PUBLIC–PRIVATE PARTNERSHIPS FOCUSED ON HEALTH SYSTEMS STRENGTHENING

In conclusion, each panelist was asked to prioritize two to three metrics for measuring the impact of PPPs on health systems strengthening.

Kruk stated that the patient’s view of quality is critically important and has been linked to better outcomes, improved adherence, and greater retention in the health systems. Then, sustainability of the program, including the ability of the program to adapt to the market, is an important indicator of success (see Box 6-1).

Taylor agreed and stated that, as a society, we should be measuring the quality of health care and the quality of the experience within the community, as well as the health of those populations and their satisfaction with how they received health care services, how they are financed, and how they participate in their health care.

Koester reiterated the need for patient centricity. In addition, he suggested that an adequate follow-up time be used, as well as measuring success from different stakeholders. He believes that market forces, health information technology, and information transparency are all driving significant changes to the metrics of the health care system at all levels.

Thwin agreed that consumer satisfaction should be part of the part-

nership evaluation. The U.S. legislation on migrant farm workers (The Migrant and Seasonal Agricultural Worker Protection Act) states that the migrant workers themselves should represent the majority on the board of the migrant health care centers that have been set up, so they have more than 50 percent representation. Collectively, we need to be considering similar models, Thwin stated. Another key area of focus is to conduct fewer partnerships so that higher-quality partnerships can be developed. This includes addressing what has gone wrong in the past to move forward, tracking the necessary metrics, and driving improvements. Finally, sustainability is important, as well. Thwin stated that after the funding ends, we need to be disciplined in examining the impacts and sustainability of benefits. She shared how KfW, the German Development Bank, would go back 2 years after their program ended to assess the impact of what they contributed. She notes that this practice is not always routinely conducted and could be applied more widely.

Stansfield agreed with the priority metrics presented. She emphasized the need to balance the dynamic tension between privacy and the benefit to public health and quality of care of data sharing. Another important point, she noted, is that there is funding for analytics at the global level, but there is little investment in country-level and peripheral analytics. As such, she believes that there is a great opportunity for PPPs as they expand to national and subnational levels to invest in analytics to support the investment case for the highest health priorities within countries.

Bollinger concluded by saying that the ideal PPP should focus on outcomes that matter to the community and to the patients. However, he emphasized that that while there may be agreement that community health outcomes matter, all partners must come to the same definition of what those outcomes really mean in order to move forward.