7

Incorporating Demographic Research in Program Design, Monitoring, and Evaluation

ORGANIZATIONAL EFFORTS TO BECOME EVIDENCE BASED IN HUMANITARIAN SETTINGS

Jeannie Annan (International Rescue Committee [IRC]) said many barriers impede the integration of evidence into policies and programs, including a lack of access to research; a lack of clarity, relevance, or reliability of research findings; and no time or opportunity to use research. One key facilitator in overcoming these barriers is building relationships and collaboration between policy makers and researchers (Oliver et al., 2014). Although people do not always think about a relationship as a key piece, said Annan, it is a common feature that has come up in studies on building evidence into practice.

Humanitarian contexts pose additional challenges, including less evidence and an imperative to act that results in even less time to incorporate evidence into policies and programs. As an example of the former, Annan pointed out that more than 4,000 experimental and quasi-experimental studies are available on development outcomes in low- and middle-income countries, while only 150 to 200 are in humanitarian settings or fragile contexts.

The IRC, which works in fragile or conflict-affected contexts in about 35 countries, has sought to become more evidence-based in several ways, Annan said. First, it has aimed to create high-impact programs by basing them on the best available evidence, adapting them to context, continuously using data to improve, and being responsive to client needs and preferences. Second, it has worked to make evidence more accessible and relevant and to reduce the time spent on incorporating evidence by identifying and organizing the most relevant evidence, identifying what is actionable, and presenting actionable evidence to decision makers in ways that they can digest quickly to make decisions.

To accomplish these tasks, the IRC found that it needed to adopt clear and overarching organizational outcomes, and not just specific project outcomes. It defined its five overarching goals as economic well-being, education, health, safety (or protection), and power. Under these goals, 26 outcomes are more specifically defined and accompanied by theories of change for how to achieve each outcome.¹ For example, an outcome under the goal of health is, “Women and adolescent girls are treated for and protected from complications of pregnancy and childbirth.” Under this outcome are hypotheses based on evidence of the pathways to achieve that outcome. Annan indicated that this has provided a map for conversations with country programs to consider for each project.

An evidence synthesis team searched for the research and simplified it by placing the most relevant information from systematic reviews upfront. Each systematic review has an evidence card that says whether the evidence is positive, negative, or uncertain. The idea, said Annan, is that people can quickly find the evidence that is relevant and useful to them.

The IRC has also sought to make research evidence more actionable. It has synthesized evidence for specific decision makers, with a focus on impact, implementation, and cost evidence. It has used a decision framework in which technical teams come together to review interventions and make decisions about them at a global level (see Figure 7-1). The teams summarize the evidence with the overarching goal of deciding whether it is the best intervention to achieve a specified outcome. Among the questions they consider are: What is the impact across studies? What cost data are available?

___________________

¹ All of the objectives and theories of change are available on the IRC’s website: oef.rescue.org.

In what context did the studies take place? They use these questions to score the intervention. Scores are based on confidence in the evidence, after which a program decision is made about stopping, implementing, improving, or scaling up an intervention. Where evidence is insufficient, the questions that need to be answered are identified.

In each country in which it works, the IRC develops a strategic action plan. Based on evidence about country trends, the needs of the populations, who else is working in the area, funding opportunities, and so on, a strategic action plan identifies which outcomes should be pursued in a country, which are most relevant, and how to achieve outcomes. Annan used a recent project in Tanzania as an example. The country has hosted waves of refugees for more than 50 years, including more than 300,000 refugees from Burundi living in camps and in other contexts. The IRC Tanzania country team used a variety of data, much from the United Nations High Commissioner on Refugees (UNHCR), to focus its work. A needs assessment helped determine which of the 26 organizational outcomes to emphasize. In refugee camps, for example, this assessment found an inadequate school infrastructure marked by limited textbooks, poor teaching materials, a shortage of qualified teachers, a high student-teacher ratio, and an up to

60 percent dropout rate at the primary school level. The assessment also found high rates of sexual violence, forced marriage, and intimate partner violence, with women and girls exposed to increased risk of sexual violence during firewood collection and adolescent girls encountering sexual exploitation in schools. Against this backdrop, IRC Tanzania prioritized education for school-aged children (6 to 14 years), safety in the home, and prevention, reduction, and response to gender-based violence.

IRC’s country plans have a 5-year horizon, a long time in the humanitarian context because of how quickly circumstances can change. Annan said they try to have a long-term horizon while recognizing the need to adapt and review as needed. In the case of Tanzania, for example, after the IRC strategic plan was adopted, Tanzania withdrew its support from a comprehensive refugee response framework that it had supported previously. It also increased restrictions on refugees’ movement and economic activities. This necessitated a review and shift of the strategy.

This example, Annan concluded, demonstrates the need to think more carefully about complexity in the context of forced migration. More and better evidence on population trends can improve understanding of needs and challenges and of the most cost-effective interventions to address these challenges. More generally, Annan said, there needs to be continued focus on how to improve evidence use and decision making in complex and changing settings.

During the general discussion, Annan noted that the IRC uses population data to inform strategic action plans for each country. The needs of countries differ widely, as do the goals of interventions and the approaches taken. The IRC encourages countries to gather available information and then discuss the needs, challenges, and likely scenarios for future planning.

Annan also acknowledged the challenges of applying new evidence in the field and that shifts in approaches or interventions need appropriate technical assistance. The IRC does this in different ways, including training and on-the-job guidance and accompaniment. Annan indicated that they do not expect people to immediately take up new practices, and technical teams are tasked with planning what is needed to teach new skills where needed.

USING RESEARCH TO IMPROVE ADVOCACY

Betsy Plum (New York Immigration Coalition [NYIC]) described approaches taken by the coalition to represent the collective interests of immigrant communities by advocating for laws, policies, and programs that lead to justice and enhance the power of immigrants and the organizations that serve them. As context, immigrant communities in New York State are among the most diverse in the country, she said. She explained that NYIC works with 210 member organizations that represent the diversity of the

state, from “rust belt” cities, many of which have large refugee populations, to agricultural regions to New York City. It also works with partner organizations, government leaders, policy experts, and affected individuals. The coalition brings together groups and individuals to work not just on immigration policy, but also on education, health, democracy, and legal policy issues. It in turn uses this work to do local, state, and federal advocacy.

Listening to on-the-ground needs begins the cycle of change, said Plum. Based on these needs, the coalition develops policy proposals from the local level to the national level. It then devises advocacy strategies that might be focused on budgets, litigation, or other policies and interventions based on research.

The NYIC participates in research through partnerships with other organizations. These partnerships, said Plum, are critical to quantifying and contextualizing populations and demographic changes, they justify new policies and programs, and they grant legitimacy to ideas. Research partnerships also allow for the marriage of broad concepts—for example, making the global phenomenon of migration relevant and understandable at a hyper-local level, or drawing connections between a refugee crisis in another country and a refugee population in New York in order to help explain the pre-migration, migration, and postmigration factors that are affecting those communities. Finally, research partnerships offer the confidence to go into high-stakes conversations with policy makers, and they hold the tools to evaluate implementation.

Plum provided an example of the research partnership approach. New York State has seen dramatic cuts in the budgets of resettlement agencies, in part because refugee arrivals are at their lowest levels in recent history. Federal funding has never been sufficient to resettle refugees in New York State, and refugees need services long after the 90 days assumed in federal funding levels. The state also has many secondary refugees—people who settled elsewhere but then move to New York. Plum explained an NYIC partnership with refugee resettlement agencies and researchers at the Fiscal Policy Institute to underscore the importance of the New York State Enhanced Services for Refugees Program (NYSESRP), which supports a flexible funding stream to make up for lost dollars and support for individuals past 90 days. Researchers also have been able to analyze the implementation of this program. As refugee admissions continued to decline, the research showed that the $2 million provided through the program was not enough to compensate for the loss of funding for refugee services. As a result, a revised policy proposal was created calling for NYSESRP to be expanded from $2 million to $4.5 million, and the ability to serve people seeking asylum with those dollars was made explicit. This proposal demonstrated the contribution this proposal would make to upstate New York’s revitalization and to downstate New York’s welcoming rhetoric.

Having a research partner that can step in and support such work is not always possible, Plum said, so the NYIC also has been doing more research on its own. For example, it is working on a project to collate on-the-ground information to demonstrate the impact of U.S. federal policies on immigrant mental health, with the goal of collectively generating policy and service delivery solutions around immigrant behavioral health. This research uncovered an extreme deficit of bilingual behavioral health providers, lengthy wait times at safety net institutions, restrictions on insurance eligibility, and a lack of awareness about behavioral health services. It led to a policy proposal for increased staffing and workforce, increased system capacity, and steps to make people feel comfortable accessing those services. According to Plum, the use of research methods offered the ability to elicit information (through five roundtable discussions and four informant interviews) that had buy-in from participants and created opportunities for information-sharing that would not usually exist. Even without a formal research partnership, research methods underscore and contextualize needs that lead to stronger policy proposals and service design alternatives and provide an opportunity to be proactive in creating research projects.

Plum concluded by saying that it is vital to bridge the often artificial divide between “academia” and the “trenches,” and to contest the idea that “some of us are practitioners and some of us are researchers.” When advocates understand basic research principles, they can draw on those tools to strengthen their own policy and advocacy processes. They also have a stronger shared language for partnerships with researchers, allowing those partnerships to be more equitable and mutually beneficial. Advocates do better advocacy with research, she said, but researchers also do better research with advocates involved, and the science is ultimately better and more directly applicable to effecting change.

In response to a question about barriers to sharing data between researchers and practitioners, Plum said building relationships is critical. As examples, she described regularly scheduled phone calls between practitioners and researchers, as well as collaborative development and design of projects and funding proposals. Plum also commented on the need for both sides to be flexible and to examine their expectations.

Annan agreed with Plum about the usefulness of meetings between researchers and practitioners. Annan also pointed to groups that conduct matchmaking workshops in which practitioners and researchers make presentations focused on content or on potential partnerships. She expressed a preference for long-term partnerships that allow for the creation of a shared research agenda, indicating that this option also lessens the transaction cost of new partners getting to know each other and how they work. Annan also pointed to the importance of brokers. Brokers can interface with researchers and practitioners and translate between sides, while growing each

side’s understanding of the other; brokers can also clarify and emphasize the constraints facing various partners while encouraging the flexibility needed in partnerships.

NARRATIVE AND PARTICIPATORY RESEARCH APPROACHES

Michael Wessells (Columbia University) explored the idea of paying greater attention to the voices and agency of affected people, which has major implications for program design, monitoring, and evaluation. He focused on children. People under age 18 typically constitute about half of affected populations, but are typically “invisible” and “voiceless,” despite the fact that they have tremendous resources and agency. They face many risks, including sexual and gender-based violence, recruitment into armed groups, family separation, and trafficking, he pointed out.

Wessells said a listening gap characterizes support for children. Researchers and service providers often hold theoretical preconceptions about the problems confronting children, but the situation of children is seldom apparent to adults. Girls face different situations from boys, the poorest of the poor have different circumstances, and so do children with disabilities. Engaging with children is often difficult, he added. Those in the most difficult circumstances tend to hide from researchers, perhaps because they fear being detected by the government or are involved in trafficking or dangerous labor.

Ethical dilemmas also loom large in this research. He commented that evidence is generated mostly by privileged people in the Global North, and questions about power and benefit emerge. Pointing to assessment fatigue as a problem, Wessells shared a reaction expressed by some of those who are studied: “Why do people come here and keep asking all these questions? Our lives do not change. Our children are starving.” In addition, Wessells said, research raises do-no-harm issues. Researchers might ask re-traumatizing questions or questions that lead respondents to believe their lives will improve. Research usually does not result in the return of information or resources to local people, which, Wessells said, can itself produce harm.

Communities are not victims unable to solve their own problems, Wessells observed. That inability may apply to a small percentage of the population but not to most displaced people. Putting power in local people’s hands can be part of a reshaping of epistemology so that it is not all derived from outsider knowledge. As research becomes more consultative and participatory, power is shared between researchers and populations. Furthermore, local people can teach researchers new insights, skills, and the like.

Wessells suggested narrative methodology as a key way to understand local people. A narrative methodology begins the process of learning about and building upon (where appropriate) local cultural resources. It avoids

power abuses involving the imposition of outsider categories and the silencing of local beliefs and practices. It gives voice to the voiceless and invisible, and it facilitates community participation and ownership of the process. Understanding how local people have been affected requires tapping into local categories that would not otherwise be apparent to outsiders who have a different worldview. For example, someone who has nightmares and sleep problems, from a Western perspective, might be suffering from PTSD. But in a country with a spiritualistic cosmology, those nightmares might be attributed to visitation by ancestors.

Wessells provided two examples—from Angola and Sierra Leone—of how the perspectives of children and other affected people can enrich design and program effectiveness. The war in Angola produced tremendous exposures to trauma. In a sample of 100 displaced children in Luanda at the height of the war, large percentages witnessed artillery bombings (84%), saw dead people (82%), lost a parent or relative (65%), saw people being tortured (53%), had to leave school (64%), lost all their belongings (72%), or experienced starvation (83%). The impacts also were large: 51 percent relived bad experiences, 66 percent had disturbing dreams, 69 percent had rapid heartbeats, 43 percent had headaches, 49 percent had trouble concentrating, 54 percent feared something bad would happen, and 39 percent had insomnia.

The program Wessells supported had begun by working on counseling and trauma healing, which, though necessary, turned out not to be a comprehensive approach. A team in Angola was training helpers to understand child development within a socioecological perspective and understand how the effects of war impact children. It was a rich participatory methodology, he said, but what was missing was “culture.” For example, the story told by a 10-year-old girl whose community had been destroyed and whose family had been murdered highlighted the cultural belief that the spirits of dead people linger if their bodies are not buried and that the spirits, which are trapped between two worlds, are angry and can enter people to cause death, illness, misfortune, and other maladies. Another case involved a 14-year-old boy soldier who showed signs of sleeplessness and PTSD but said he could not sleep because of voices. He said, “The spirit of the person I killed comes to me at night and asks ‘Why did you do this to me?’” A belief of being haunted by an unavenged spirit is not congruent with Western beliefs of material science, but beliefs shape behavior, said Wessells, and in this case local beliefs had a profound impact.

The Angolan team hired a Mozambican social anthropologist, and the seven-province team learned how to do basic ethnography, including observing participants and conducting interviews. They learned that people had a spiritualistic cosmology that interpenetrated with Western ideas of care. For example, if a person breaks an arm, he or she goes to a Western

medical clinic to get it set. But if people have nightmares or other kinds of problems, they attribute it to spiritual causation and go to an herbalist or another traditional healer. In this way, said Wessells, the team learned about the importance of rituals and traditions wherein living people honor their ancestors.

This new understanding had a transformative impact, he said. Before then, with funding from the U.S. Agency for International Development, the project had taken a fairly conventional approach to the integration of former child soldiers. They convened participatory dialogues with families and communities, helped children reconnect with their families, and engaged in problem solving. They taught literacy, life skills, and income-generating activities and mentoring. Although all of this was important, Wessells said, it was not working very well, and they did not know why.

The new approach revealed that the former child soldiers had very low social acceptance. They were seen as carrying the angry spirits of the people from the bush who had been killed. Working with the community revealed that people saw the need for spiritual purification rituals conducted by local healers. Wessells said that although these rituals did not cure PTSD or depression, they had major effects on social acceptance, which is a key step toward integration.

The second example described by Wessells took place in Sierra Leone, which experienced a brutal, decade-long civil war. A top-down approach, where outside experts define what is needed and impose it on communities, has become the norm in the world of child protection and mental health, Wessells said. In Sierra Leone child welfare committees were established to monitor violations against children and train people to report violations and make referrals to police, social workers, or others. This top-down approach was enshrined in the Child Rights Act of 2007, which mandated a child welfare committee in each village.

However, Wessells said that most people were reluctant to report abuse to the child welfare committees or police and state authorities, instead preferring to use traditional processes through the chiefs. The main reasons for their reluctance related to distance to authorities, having to take time off farming, doubts that action would be taken, and culture. According to Wessells, ethnographic research also indicated that individuals in local communities did not like the term “child rights,” as they saw it as undermining their authority as parents and their ability to discipline their children and to teach them proper behavior and values.

Wessells and his colleagues took an alternative bottom-up approach where local people were viewed as the problem solvers. Over the next several years, the team helped communities develop their own action intervention. Facilitators who understood the local language and the local culture lived in the villages. Rather than engage in what Wessells called

“facipulation”—a mixture of facilitation and guiding toward what a grant specifies—they asked which of the particular harms to children identified by the community the community wanted to address. Wessells acknowledged this approach is somewhat problematic, because it depends partly on the definition of community, and every community has a power elite. Women, the poorest of the poor, and children with disabilities are almost always subjugated and marginalized, he noted. Nevertheless, the facilitators asked questions that triggered awareness. The communities themselves said that everyone should have a voice, which was accomplished through small-group discussions. Teenage girls, for example, were much more willing to talk openly if their statements were not attributed to any particular individual. For the poorest of the poor and people in the most difficult circumstances, home visits were made that were designed by local people. Wessells indicated that these approaches were not stigmatizing and did not reawaken very difficult issues, but rather were done with considerable attention to ethics.

The communities eventually chose to address teenage pregnancy, although Wessells acknowledged that he wanted them to consider sexual abuse by older men that results in these pregnancies. “I wanted people to take on the sexual abuse. But local people, as usually happens, were smarter than me,” he said. The girls in particular realized that if abuse were named as the problem, older men in the community would undermine the work. Instead, everyone was able to agree with the idea that teenage pregnancy is a terrible thing. Girls who were pregnant had to drop out of school. Their families typically could not afford to feed another mouth. Within a short period of time, the girls might turn to sex work and their babies would not thrive.

The communities designed actions that focused on family planning, sexual reproductive health, and life skills, though these were not the terms the communities used, Wessells noted. The messages were delivered in the local language and not orchestrated by outsiders. The intervention had many key pieces, but much of it came down to leadership by children, said Wessells. For example, girls and boys would do street drama to present the message to stay in school and avoid pregnancy. Explicit discussion of sexuality is not a norm in Sierra Leone, said Wessells, but it became a norm as a result of this slow dialogue and reflection process.

This approach had very promising results in reducing teenage pregnancy and sexual abuse before the Ebola outbreak of late 2014. Wessells said that the entire community had become so animated around reducing teenage pregnancy that it became impossible for the elder men to abuse 14- and 15-year-olds. There was also increased access to and use of contraceptives; increased intent of girls and their close friends to ask partners to use condoms; stronger linkage of communities with the formal

health system; reduced school dropout; and discussion between parents and children of sex, pregnancy, and pregnancy prevention in a constructive manner. The key, Wessells said, is that the interventions were community owned. They were led by local people, and they were interventions that local people wanted to continue.

Wessells suggested a number of implications for research practice from these examples: Researchers ought to use elicitive, respectful methods in assessment; learn about local power structures; learn about and engage with natural helpers; engage with people who are positioned differently; encourage collective reflection, planning, and action regarding children’s issues; rethink their role from “expert” to co-learner and facilitator; document and learn from Do No Harm issues; and model and enable critical, reflective practice.

During the general discussion, Wessells emphasized the importance of humility, trust, respect, and empathy when partnering with communities. He also said that it was important to go slow and move according to community time; it is a very different process, and most curricula do not build those particular skills. Susan McGrath (York University) observed that funders do not always allow for time to go slow, and that Wessells’s and others’ work discussed at the workshop demonstrates the need.

Wessells added that adapting models from elsewhere in the world to the United States is fraught with political challenges. At the time of Hurricane Katrina, for example, several organizations proposed to the U.S. government, including the Federal Emergency Management Agency, the idea of cross-learning, wherein the organizations would offer advice based on what they had learned in international settings. But U.S. agencies feared that they would be criticized for how they handled the emergency response, and cross-learning did not occur. Wessells said that although the situation today may not be as badly siloed and segmented, it is almost so. For example, when the U.S. government began separating children from families at the U.S.-Mexican border, many people with expertise in migration wanted to engage in a dialogue, but they did not have the networks and relationships to do so. Though a small amount of cross-learning has emerged across some U.S. government agencies, the activity is not formally structured, he said.

This page intentionally left blank.