Based on their experiences scaling up programs, workshop participants described several ways to meet scale-up challenges, including ways to
- build demand, capacity, and incentives in communities that are in need of effective programs;
- provide a supportive implementation infrastructure;
- create sustainable funding; and
- adapt and improve programs so they are more applicable and successful.
Scale-up could also be aided by implementation research that reveals what strategies work best.
Several participants noted programs tend to be more successful if there is consumer support for them; local, agency, or political champions of them; and if the right program is matched to the right organization.
Consumer Demand
Families are the ultimate consumers of family-centered programs, which will not be successful unless family support is engendered, Ron Prinz said. He suggested sampling the opinions not just of parents who
have participated in programs, but of parents who might participate in the future. “Get them to express what their needs and preferences are, and then build in ways to answer those needs. Parents don’t speak with one voice but have multiple perspectives. We want to try to hit as many perspectives as possible,” he said. Mary Jane Rotheram-Borus from the University of California, Los Angeles, suggested surveying consumers to also find out what type of program formats they prefer, and Lauren Supplee added, “We need to think about user-centered design and getting the end-user voice from the outset—are they going to be engaged in this model?”
Prinz also suggested having a media communications strategy. Media publicity (e.g., via Web, community posters and billboards, parent and school newsletters) encouraging the parenting practices promoted by programs not only reach potential consumers of them, but reinforce the efforts made by participating parents, as well as validate service providers for their efforts. “It’s a way of getting public recognition for the hard work that often burns out practitioners working in isolation,” Prinz said. However, Joe McCannon pointed out, “We tend to rely too much on traditional modes of communication. We trust these diffusion mechanisms that actually don’t change behavior.”
Lisa Hill noted that Invest in Kids built consumer demand for its programs by being messengers for them. Members of this organization literally “hit the road” to meet with various communities scattered in urban and rural areas of Colorado. That effort paid off, she said, as six communities adopted the Nurse–Family Partnership (NFP) program and delivered it to 100 families each. “That only took us 9 months because people were ready and waiting for this program and tremendously excited about the opportunity to bring it to their communities,” Hill said. “Building that kind of local ownership of the program is crucial to the sustainability and quality of implementation,” David Olds added.
Part of building consumer support for programs is making sure the program fits their needs. When people want to implement The Incredible Years® (IY) program in their communities, they are asked to complete an Incredible Years Agency Readiness Questionnaire, which helps them prioritize their needs and identify their target population to assess if the program is a good match, and if not, other evidence-based programs are suggested. The Readiness Questionnaire also suggests organizations develop parent recruitment and referral strategies.
Both Carolyn Webster-Stratton and Margot Kaplan-Sanoff emphasized the importance of including everyone in the workforce in the decision making about program adoption, and acknowledging the work members of the workforce already do and how the program will build on that work. “Nurses wanted to know what the Healthy Steps Specialist would provide that they don’t already provide and we would respond, ‘Yes, that’s what
you already do, but let’s add on to it and make your job easier by having somebody else help with it,’” Kaplan-Sanoff said.
Another advantage of consumer support garnered for the NFP program that Hill cited was that it fostered support at the agency, supervisor, and provider level, ultimately leading to greater sustainability and retention of nursing staff, which led to better client outcomes. Guillermo (Willy) Prado added that showing the high engagement rates for the school-based Familias Unidas programs led to their greater adoption and support by school principals, many of whom were skeptical that parents would participate, given a lack of their participation in other school programs.
To overcome the lack of participation by ethnic minorities in their family-focused program, Norway’s Center for Child Behavioral Development hired bilingual “link workers” to contact Somali and Pakistani mothers, who are from the most dominant immigrant groups in Norway. This improved their participation in the parent groups created specifically for ethnic minorities, Terje Ogden noted (Bjørknes et al., 2011, 2012).
Rotheram-Borus suggested building demand for programs for families by offering the programs along with or part of other community programs already in demand, such as martial arts or as part of summer camp or after-school club offerings. She noted that consumer surveys in Santa Monica indicated that parents wanted coverage on such topics as social media, parenting, sibling fights, and homework. For each of those topic areas, her organization provided expert talks and information websites for parents. “We integrated these educational and evidence-based activities into martial arts, coaching, yoga, divorce, and whatever else the community said they wanted and were willing to pay for,” she said. “Our martial arts program was the best parenting program you ever saw and was especially popular among men.” Shortened versions of some of these programs were also offered at shopping malls for free with the ability to sign up for more paid interventions.
Program Champions
Several participants said their programs spread successfully due to the demand for them instilled by political champions, who advocated for their adoption or funding. Hill noted that Invest in Kids was able to have the Colorado legislature fund the NFP program because a state senator’s mother had benefited from a nurse home visit. This well-respected Republican sponsored a bill that determined how tobacco master settlement agreement funds would be used in the state, and advocated for passage of the Nurse Home Visitor Act that led to $19 million per year being set aside to fund the NFP program.
Kaplan-Sanoff also suggested finding a champion in the organization
that adopts a program. That champion should not only advocate for the program but have the power to make changes in the organization. Prado agreed, noting the importance of building relationships with those that can be champions for a program “not only from the top down but from the bottom up and to engage at all levels in between.”
Because the average tenure of school superintendents in the country is about 2 to 3 years, Familias Unidas tried to find champions for their school-based programs not just with school superintendents and directors of mental health counseling for the district, but with the associate superintendents of curriculum and education, and the mental health counselors, social workers and school counselors delivering the services the program offered. “We had a lot of champions so it was an easy sell when a new person came on board,” he noted. Webster-Stratton added that it is important to have an internal agency champion or advocate to ensure clinician peer-review support and outside consultations when needed.
Agency, Provider, and Workforce Incentives
Thomas Dishion suggested providing incentives for host agencies and for the leaders that are championing adopting evidence-based practices. “I don’t see how it’s going to work unless those people are getting the same kind of benefits as the program developers,” he said. “One of the problems I’ve seen is that folks take on this enormous burden of implementing a program—how does it benefit their career?” he asked. Patricia Chamberlain responded that a lot of the benefit “is you get a reputation as an innovator and a mover and a shaker.”
But an underlining incentive that Olds noted, and several speakers reiterated, was the urge “to make the world a better place and to do so with a really deep, rigorous approach to figuring out how to spend scarce public resources in a more effective way.” McCannon added, “Business as usual, by which I mean people go into work, check their email, and have some meetings, is not what you see in these initiatives that really thrive.” Instead, he said, there is an energetic selflessness with the drive coming from feeling like “you are a part of something that will be so great to accomplish that if you do, that achievement will eclipse any sort of issues of personal return or personal credit. So creating a culture that motivates you to do these kinds of extraordinary things has a profoundly different psychology.”
Building capacity of program providers involves hiring, training, and coaching staff. Olds pointed out the success of the NFP depended on the thorough and ongoing education and consultation of nurses, which in-
cluded visit-by-visit guidelines and a corresponding information system to monitor ongoing features of the program’s implementation. Such a system should not be too burdensome, he added, noting that he ended up scaling back the kind of information nurses were required to enter into the system because information technology placed unrealistic demands on the nurses’ time.
Rotheram-Borus emphasized that training models should be different because “I’ve seen too many people who get the scripts right, but don’t have any competence about what the skills are, and if you don’t know the skills, you can’t apply them to new situations.” She noted effective training of paraprofessionals is possible as long as they have good social skills and are good role models to begin with. She reported that Chorpita and Daleiden (2009) have identified about a dozen basic skills (self-monitoring, problem solving, cognitive styles, goal setting, praise rewards, assertiveness, attention, modeling, monitoring, relaxation, response cost, and mirroring) that are needed for 80 percent of all child and adolescent evidence-based interventions. Rotheram-Borus trained the paraprofessionals she employed in all of these skills and monitored over time how often they used the skills.
In addition, Rotheram-Borus suggested paraprofessionals need to be able to carry out five basic functions, including how to
- Frame an issue.
- Apply knowledge.
- Build skills.
- Remove barriers.
- Build social support.
She stated these abilities were key because “if you arrive at a home visit and there’s a crisis going on, you’re not consulting your manual. You have to give people the flexibility, but also the know-how for applying the skills that they use to the situation in the home.”
Some participants suggested building capacity at organizations implementing programs by making use of the existing workforce. Rotheram-Borus suggested organizations have not adequately tapped “the many armies of potential prevention promoters—every soccer coach, school safety officer, child classroom aide, swim teacher, music teacher—these could be armies of paraprofessional prevention promoters but I haven’t seen any of our intervention programs go towards them.” Kaplan-Sanoff said that in Healthy Steps it is preferred that everyone from the receptionists to the physicians attend the training sessions.
The IY program suggests that at least two group leaders or clinicians per group be provided with authorized accredited training to support each other and the next person hired if one of the leaders is replaced. The
organization should also provide staff time for study, preparation, and consultations after the training workshop, although Webster-Stratton noted that often doesn’t happen. She pointed out that in addition to the initial training for staff, there needs to be ongoing coaching, consultations, or supervision by accredited coaches, mentors, or trainers. The IY Readiness Questionnaire assesses if there is managerial support and understanding for the program and clinicians on staff with adequate qualifications to provide it. Clinicians without background in cognitive and social learning theory or child development or group work will find themselves “floundering,” she said, if they are not given this post training support when providing the IY program. “Many have not been trained in the group model—they’re used to seeing families on an individual basis,” she said.
Olds concurred: “The staff needs to be well prepared to deliver the program. They need consultation in dealing with challenging situations, because if they don’t feel effective, their commitment to their role is going to be undermined, and they’ll be more likely to leave. That ongoing education, consultation, and building staff members’ efficacy is really central,” he said.
Hill noted a study that found teachers were more likely to demonstrate knowledge and skills they had acquired through training programs in the classroom if they received coaching in the classroom rather than only theory and discussion (Joyce and Showers, 2002). “It is critically important for the long term to stay invested in the consultation role to ensure the quality is sustained over time,” Hill said.
McCannon suggested what he termed “itinerant coaching” by program developers or implementers. “Having people go from site to site collecting good ideas, problems, and sharing solutions between sites seems to be effective. A field operation that assesses and understands the needs of the organizations that adopt programs and their barriers is the hallmark of these initiatives that really succeed,” he said.
PROVIDING A SUPPORTIVE INFRASTRUCTURE
Several speakers indicated the need for a supportive infrastructure in order to ensure successful implementation of evidence-based prevention programs. “Developers cannot and should not do it all, so we really need to think about having a public health infrastructure to support implementation much like private industry,” said Supplee. She noted there are a number of different models for providing that infrastructure from state, federal, or program-specific technical assistance “but none of them are necessarily getting scaled up and it’s not really clear whether particular models work better for some pathways than others. The bottom line for ‘if we fund them, they will multiply’ is that obviously funding is necessary but not sufficient for scale,” Supplee said.
Brian Bumbarger also emphasized the need for infrastructure support to ensure quality improvement, not just at the individual community and provider level but also at the state level. He noted, “We like to think we’re all in the business of better outcomes for families, but we have to recognize the reality that practitioners, policy makers, and researchers have different needs, and it is the role of the intermediary organization to accept responsibility for solving whatever problems that occur or that may interfere with the effective scale-up of these interventions.” He added, “If the policy makers aren’t really talking to the practitioners but are just giving out grants and monitoring contract compliance, they don’t realize all of the troubles occurring on the ground. We wanted to be the ones responsible for carrying those messages back and forth.”
ADAPTING AND IMPROVING PROGRAMS
A major discussion point at the workshop was how to adapt programs to suit the specific needs of the communities that adopt them, while maintaining fidelity to the program so its effectiveness is not undermined. This is less an issue if communities choose their programs carefully so they adopt ones that best meet the needs of their population, Supplee noted. Nonetheless, some experts in the field believe adopters might have to tailor even well-suited programs to fit their budget and priorities.
Webster-Stratton emphasized the importance of adopters following program protocols, but she recognized that programs also need ongoing remodeling based on feedback, research, and the setting in which they are delivered. She noted that some providers may only follow “the minimum dosage,” in an attempt to maintain fidelity to their population or budgetary constraints. However, she also pointed out that research on the IY program found that the longer the dosage, the greater the effects. This program does have flexibility in that parents identify their own goals and learn a set of principles they can apply to achieve these goals. For example, if parents want help dealing with their children in grocery stores, a vignette related to that will be part of the program. “I think that’s why it works so well cross-culturally,” she said. She added, “There are a set of principles embedded within the program that haven’t changed since 1979 when I did my first study.”
Olds noted there is an inherent tension between the core model and new augmentations or innovations that get introduced when it is applied in the real world. For example, when the NFP was first tested and introduced, it used a procedure for observing and promoting parents care of their children in the context of home visits. But that method ended up being too cumbersome and hard for nurses to use. So Olds and his colleagues developed a separate procedure and a streamlined clinical tool that would align
observations with nurses’ way of interacting with families in the home. “We didn’t put that to a separate test in a randomized controlled trial. We just started using it because the new method was clearly superior,” he said, adding, “But this tension between what needs to be tested and what can simply be improved we collectively need to address with principles that would guide our thinking about how to do that well.”
Prinz pointed out that programs tend to evolve over time into different formats as well as in response to the feedback they receive from their adopters. “You can’t say, ‘Here are the rules for doing the intervention, and it will functionally not change,’” he said, noting that the online format of the Triple P-Positive Parenting Program, for example, was not envisioned 20 years ago when the program was first developed. “The Triple P that was available 5 years ago is much different than what’s available now. It’s constantly improving, adding evidence and changing as it grows. We need a system that’s built into the program that can absorb change,” Prinz said.
Dishion added, “It’s an iterative process. The more we have contact with our clients, which are the communities, the more they’re going to shape us, in terms of the kinds of interventions we’ll be developing.” He noted the general strategy of a translational process is that interventions be revised for public health service settings, and after they have been tested and tailored for real-world conditions, the developmental model the intervention is based on may be revised. Dishion said this strategy has worked for the Family Check-Up program, whose application in American Indian communities prompted its developers to add historical trauma to the original model’s list of factors influencing families (see Figure 7-1).
Olds echoed Dishion, stating, “We believe that the NFP is a work in progress, not a final product. There are subpopulations for whom the program is not working and ways in which the program is not working. We need to figure out how to do a better job, and that commitment to rigorous development of innovations in the model and replication is really part of what we’re all about today.”
Bumbarger offered that “quality and fidelity often work at cross purposes. The greater scale you go to, the harder it is to maintain quality and fidelity.” In an ideal world there is an intervention redesign component in which evidence-based interventions would be optimized and redesigned, hopefully with the input of communities and end users after effectiveness testing and before going to scale. But in reality, Bumbarger noted, communities perceive evidence-based interventions as unnecessarily complicated, and “they come to ask us legitimately, ‘Is all this really necessary? Do I have to do all 18 lessons?’ We can’t empirically answer that question so there’s an imbalance between the things we’re trying to get communities to do and the things they’ve naturally been attracted to.”
Kaplan-Sanoff suggested balancing the requirements of a program that
FIGURE 7-1 Family Check-Up model: An iterative translational research strategy.
SOURCE: Adapted from Dishion and Patterson, 1999.
will be applied in a pediatric setting with flexibility, pointing out that “pediatric practices do not want to be jumping through hoops of fire. They already do that all the time and don’t want a new program to make them do that.” McCannon emphasized that “when we’re talking about persuading people across boundaries and multiple organizations, we have less control and so we have to invite more of their own energy and ideas and will in order to engage them.”
Rotheram-Borus noted that when the Centers for Disease Control and Prevention nationally diffused an evidenced-based program for HIV prevention to 2,500 agencies, half of the people they trained never implemented the program, and of the 50 percent who did implement, only 20 percent did so with fidelity. The main reasons she gave for that lack of fidelity was that the program was not tailored enough to the specific populations in which they were implemented.
When Norway adopted the Parent Management Training-Oregon (PMTO) model, it adapted it substantially, Ogden reported. This program was originally designed to have 20 to 50 parent sessions and is usually delivered to individual families, but it also has modules for school con-
sultation. Recognizing that many parents may manage with fewer sessions, Norway created a program built on the same principles as PMTO (Solholm et al., 2013) but divided it into brief individual parent sessions and social skill training for children, parent groups (Kjøbli et al., 2012), and school consultations. The program was also implemented in a variety of primary care settings, including public child health clinics, schools, and kindergartens, with a low threshold for intake, and is composed of fewer sessions with lower intensity and shorter duration than full-scale PMTO, Ogden reported. Within this program, the full-scale PMTO treatment is still offered, but only as a backup for those families that need it. In addition, the Norway implementation developed training and intervention manuals and a description of the core intervention components. Six months after receiving the brief parent training, a study found the beneficial outcomes were sustained on most child and parent variables as they are for the more extensive PMTO (Kjøbli and Bjørnebekk, 2013).
Dishion said that a study he did on the implementation of his Family Check-Up program found that the intervention’s effects were completely mediated by the fidelity of the therapist’s adherence. “That makes it complicated because you can’t just expect effects, you need to have effective therapists. I can see that many of our therapists, who may not have formal training, are quite skilled at working with parents. So we’re looking at first doing the handshake, then doing the training, fidelity, coaching, monitoring and then certification,” he said. “The idea is you get people to engage with the service and then fidelity will follow. If we hit them hard on fidelity right away, we may lose them.”
Ogden noted that he tested the notion that all that is needed for a program’s results is a therapeutic alliance. Fidelity, as rated by coders, and alliance, as rated by parents, both predicted parent-reported problem behavior, but whereas fidelity predicted change in problem behavior, alliance was related to less change in problem behaviors (Hukkelberg and Ogden, 2013). “It seems that when it comes to parenting programs, particularly for antisocial behavior, it is important to have high treatment integrity or fidelity, and the therapeutic alliance may not be enough in order to secure the positive outcomes,” he said. Supplee added that a 2009 study (Schoenwald et al., 2009) found the quality of supervision provided to the therapist predicted youth outcomes more so than the therapist’s interaction with the youth.
Stephanie Lee also cautioned against drifting too much from program protocols, noting a study of the effects of the Functional Family Therapy being applied to a population of juveniles in the juvenile justice system in Washington State found the outcomes were not as expected, with no difference between the children who had gone through the therapy program and those who had not. Further investigation revealed that the therapists were not equally adherent to the program model. Consequently, the juvenile jus-
tice system there set up a fidelity monitoring system to supervise and train providers, and if they do not adhere to the protocol they are retrained or moved to a different area of work, Lee reported.
Even programs that have been adapted still have to be monitored for their quality and effectiveness, several speakers pointed out. Webster-Stratton suggested that staff’s individual and group sessions with parents be videotaped and reviewed by accredited coaches, mentors, and trainers, and based on those reviews, feedback be given regularly to clinicians so they can improve their performance. The IY provides a rigorous accreditation process that includes review of protocols, attendance at workshops, peer and self-evaluation, consumer satisfaction evaluations, and review of the DVDs of their sessions.
In addition, Webster-Stratton suggested having clinician peer-support networks within the organizations delivering the intervention. “Group leaders or clinicians should videotape their sessions right from the start and then meet together to look at their group session and give feedback to each other,” Webster-Stratton said. The IY program provides standardized group process checklists for that peer review and recommends pairing more experienced clinicians with new clinicians. Agency administrators should also be promoting accreditation of the program, which provides training for these administrators. “One of the reasons our program has had such good replication by others is largely due to the accreditation process, as well as the detailed manuals, authorized trainings, and well-defined protocols,” Webster-Stratton said.
Chamberlain said technology, including video uploads of group sessions that can be used to rate fidelity, has helped immensely with the monitoring of programs as they are adopted and implemented by other organizations. “We can give a consultation using the clips of the videos, and in years to come we’re going to find that we’re going to be more able to use this mechanism to help us have better implemented interventions,” she said.
Prado seconded that notion, pointing out that to provide feedback to the social workers or school counselors who conduct parent sessions in the Familias Unidas program, sessions are videotaped by a rater who watches these 2-hour sessions two or three times. Recognizing that monitoring fidelity can sometimes be more expensive to carry out than the delivery of the intervention itself, he and his colleagues developed a semi-automated system for monitoring fidelity ratings that uses computational linguistics and speech recognition software to rate videotapes of sessions. There is about an 80 percent concordance between human and machine ratings using this system, Prado reported. “It’s certainly something to consider as
we try to begin to take this program into schools where resources are very limited,” he said.
Dishion added, “We need to set up systems where the clients themselves can monitor their outcomes, and digital technology is a great tool for that.” He noted that the developers of Family Check-Up are trying to develop an app so that after the initial visit with a clinician, parents automatically get texted once or twice a week with five quick questions about how they and their children are doing. Answers to those questions appear on a website where the clinician can watch the progress and their outcomes.
McCannon suggested that rather than evaluating how well organizations rigidly adhere to a program’s protocol, instead there should be daily data that indicate how well the organization is doing at meeting their goals. He suggested having dashboards that indicate those daily objectives and how much of them have been met, as well as a daily rhythm of reviewing and responding to the data. “We need to hold people accountable for meeting data goals week in and week out,” he said.
Programs will not be implemented properly and maintained in the long term unless they acquire sustainable funding. Several sources for that funding were discussed at the workshop, including service grants, tiered evidence-based funding and Pay for Success grants, public–private partnerships, braided funding from different agencies, and new reimbursement options under the Patient Protection and Affordable Care Act (ACA).
Supplee suggested there be funding for the infrastructure required to implement an intervention. She noted that encouragingly, there has been a gradual increase in grants that are allowing and paying for planning periods of 6 months to 1 year and then requiring implementation plans that have to be approved before the services can be provided, as well as funding that can be spent on data systems, continuous quality improvement, and infrastructure costs, which some research suggests is about 30 percent of ongoing costs. “This is not a one-time investment—30 percent of the grant might have to go to infrastructure,” she stated.
Kathryn Stack reported on new federal funding opportunities aimed at furthering evidence-based practices. These grants include tiered support programs that financially reward applicants and grantees for reviewing and using existing research on what works and how to implement with fidelity, as well as for conducting or participating in rigorous evaluations to build evidence on what works. The funding programs support a hierarchy of evidence-building studies, including those on the bottom tier that are designed to show proof of concept, those in the middle tier whose goal is to validate previous findings, and a third tier whose objective is to scale-
up effective practices. “The theory is that not all of these interventions are going to be successful, but you want to give them a chance, and if they do well they get to move up this pipeline,” Stack said. Under this tiered model, the biggest grants are given for scale-up, medium-sized grants are for validation, with the smallest grants given for proof-of-concept studies.
Pay for Success is another evidence-building funding strategy in which private investors pay for the working capital to start evidence-based prevention services and the government provides a contract that specifies it will pay these private investors back after a certain time period if they achieve better outcomes for individuals and savings for the government. This type of funding has an evaluation built into it. “The theory is you are providing all the incentives for people to drive towards what works, but in this case there’s the flexibility not to necessarily adhere to the model if you can find a more cost-effective way of achieving the outcomes. So there’s a built-in incentive for innovation,” Stack noted.
Medicaid also offers several funding streams for children’s behavioral health that providers could tap into to support evidence-based programs, MaryBeth Musumeci reported. She noted that 10 percent of children who receive Medicaid are using behavioral health services and/or psychotropic medication. Given that 20 percent of children in the United States are reported to need behavioral health services, there probably is an underuse of that service in children supported by Medicaid, she said.
The ACA expanded Medicaid eligibility for children by increasing the household income limits to 138 percent of the federal poverty level ($27,310 for a family of 3 in 2014). Medicaid’s Early Periodic Screening, Diagnostic, and Treatment Services (EPSDT) benefit for children up to age 21 includes mandatory coverage of any services necessary to correct or ameliorate physical or mental health conditions in children. “So kids have access to all these services even if the states do not offer them to adults on Medicaid,” Musumeci said. Some states also provide coverage for behavioral health services, which can include child mental health services, such as individual and group therapy, crisis intervention, and medication management, among others. “What’s appealing about using this as a funding source for behavioral health services is that these services can be provided at home or in the community,” Musumeci noted.
Since the Deficit Reduction Act of 2005, there has been a Medicaid option for states to offer home- and community-based services as part of their state benefits package instead of through a waiver (§ 1915(i)). Some states have implemented this option, and the ACA amended it by increasing the qualifying income limit. Unlike services under a waiver, § 1915(i) home- and community-based services can be offered more as preventive care because those beneficiaries do not need to meet an institutional level
of care; instead they must meet functional needs-based criteria that are less stringent than those required to qualify for institutional care.
The ACA also added the Medicaid health homes state plan option, which some states have used to help support health homes targeted to children and youth with serious and persistent mental health conditions. A health home provides care coordination and case management services. “What’s enticing for states here is that there’s 90 percent enhanced matched federal funding for the first 2 years that the state is providing health home services,” Musumeci pointed out. A few speakers noted that Maternal, Infant, and Early Childhood Home Visiting (MIECHV) funding can be used to support programs, such as the NFP, that have home visits as a major component.
Some participants suggested creative business models to support scaled implementation of their programs. Autism Navigator® is being distributed by a for-profit company, which offers the various online tools they have built for free, but charges for certification courses for professionals. All of the profits of the company will roll back into development, Amy Wetherby reported, noting that this business plan is expected to enable the scalability and sustainability of Autism Navigator®. Richard Spoth suggested the development of innovative funding mechanisms, including the creation of private–public partnerships, to support programs and braiding of funding from multiple sources. Rotherham-Borus recommended designing programs to fit existing funding streams.
OVERCOMING ORGANIZATIONAL SILOS
Several participants noted that many prevention interventions for children or families have results that spill over into a number of different domains because, as Spoth pointed out, they target similar, common risk and preventive factors. For example, a positive parenting intervention can improve children’s educational achievements as well as decrease their rates of juvenile delinquency. But support for all the outcomes a single intervention can achieve is often provided by different state or federal agencies. “We all complain that you have these sort of rigid, straightjacket programs that require or have certain kinds of eligibility criteria and allowable activities, and sometimes in a community it can be very hard to weave those together,” said Stack.
Spoth called for having more state and national systems that integrate their prevention efforts, which he said would be a key action step to take in the future. He also suggested planning and organizing for infrastructure development, including interagency collaboration that could be tied to the ACA and build on the National Prevention Strategy (HHS, 2014) and the findings and recommendations from the National Research Council and
Institute of Medicine report Preventing Mental, Emotional, and Behavioral Disorders Among Young People: Progress and Possibilities (NRC and IOM, 2009), by focusing on scaling systems for evidence-based interventions (Spoth et al., 2013). He also suggested developing a common conceptual framework for addressing multiple behavioral health outcomes, such as common risk and protective factors that are relevant across federal stakeholders. “It is critical to keep a clear focus on that outcome we desire, that population impact—and recognize the complexity of all these barriers,” he said.
The new pilot authority for Performance Partnerships for Disconnected Youth, included in the 2014 appropriations act for the Departments of Labor, Health and Human Services, and Education, should allow organizations to come out of their silos and collaborate more with one another, Stack reported. This act authorizes up to 10 pilots serving disconnected youth that allow state, local, and tribal governments to blend funds and receive waivers of statutory provisions to implement a cross-program strategy to improve education, employment, and other outcomes. The U.S. Office of Management and Budget in collaboration with relevant federal agencies is issuing a discussion draft on how it will implement the Performance Partnerships. “We want to get examples of where communities might want to integrate services, such as marrying a substance abuse program with an education program and workforce program to get better results in a particular population,” Stack said. She added, “We are trying to build partnerships and infrastructure to support a completely different way of doing business—it is really an opportunity to find a target population and test this notion that we can provide a much more holistic set of services to achieve greater impact.”
Convincing economic analyses can also prompt agencies to overcome siloing, Lee reported. For example, after the Washington State Institute for Public Policy provided different cost–benefit scenarios for various programs seeking legislative funding, the state scaled up funding for early education and the NFP while deciding not to fund a new prison forecasted to be needed by 2020. The analysis the institute provided showed that with the right prevention interventions, fewer rather than more prison beds would be needed. “People started thinking outside of the silo that intervening in criminal justice leads to better outcomes in criminal justice and opened up their minds toward thinking about those long-term prevention outcomes and their impacts on criminal justice,” Lee said. She added that programs that are attractive to legislatures are often those that either target multiple populations or those that have multiple outcomes across sectors. “The scope is the most important thing to consider,” she said.
McCannon described more general scale strategies and principles he has found worked for other organizations. He pointed out that initiatives that do especially well “are not deeply invested in exhortation or just pushing an idea but rather removing the barriers that are holding people back from adopting new practices.” He noted there are two kinds of barriers, the most obvious being operational. But he emphasized also addressing the psychological barriers. “When someone is asked to make a change, usually their reluctance to do so is less about what you’re asking them to do, and more about something they are holding onto that’s important to them and they feel they would have to give up to do the new thing. So thinking explicitly about those things they’re holding onto is crucially important,” he said.
McCannon also suggested designing programs for scale in addition to one-time success, noting that one wants to ensure that the prototype of a program works, “but we can get to a point where we’ve designed something so top-heavy that it’s difficult to scale it, whereas design for scale thinks about scale-up factors in a systematic way and explores how to make each of those as lean as possible,” he said. These scale-up factors include human resources, financial resources, physical space, technology, oversight structures or systems, delivery requirements, and supplies needed to run the program. He suggested planning for these factors during the design phase, while recognizing that they can be adapted as the program is disseminated. “You can continue to toggle and tweak these as you expand all these different dimensions,” McCannon said.
Finally, McCannon suggested giving people the opportunity to try and fail and to waste no will, which means “anytime someone comes with an idea you say ‘yes’ and you work with them on adapting it to make it work in their setting,” McCannon said. Another meeting participant added, “We have so much we can learn from failure. At the systems level in our work we have more latitude for allowing for failure and it has fueled some of the innovation and solved some of our problems at the program level.” McCannon responded, “It is another leadership job. You have to applaud people who are able to talk about failure and encourage those who are afraid to share it.”
However, Dishion noted that funding is often predicated on showing an intervention or program that works and not one that fails. “We’re highly reinforced for publishing what works, and our careers are punished for showing an intervention that didn’t work. We need to lose that paradigm and get the word out when we fail so others can benefit and do a better job of specifying the conditions under which we succeed or fail.” McCannon concurred, saying, “There is no failure, there is just learning. If that were the approach that we took, then any paper you wrote to summarize a study
would point out what we learned, including what didn’t work, but the promising components of what didn’t work. We need to move away from binary study designs to those that are more nuanced and look at the texture of what’s happening in different environments and settings.”
RESEARCHING IMPLEMENTATION STRATEGIES
Several participants noted the lack of evidence-based implementation strategies and suggested more research be done in this area. Specifically, Supplee suggested gathering more empirical evidence for the
- Importance of capacity building,
- Effectiveness across different populations and contexts,
- Workforce quality needed to implement particular interventions with fidelity,
- Moderating factors for quality implementation at scale overall, and
- Economic costs and benefits of implementing at scale.
“As a field, we have some empirical evidence about factors affecting implementation, but not a lot, so the call for implementation science has become very loud,” she said, adding that the need for such research be made apparent to decision makers and policy makers. Dr. Kimberly Hoagwood added that although state mental health agencies collect data, they are not systematically related to the implementation of programs. In addition, only a small percentage of state agencies fund a research center or institute, collaborate with one, or produce a directory of research or evaluation findings related to implementation (NRI, 2012). “In terms of understanding the implementation process itself, states typically do not see that as something they can or should do. Where is the support for this going to come from so we can learn from these rollouts?” Hoagwood asked.
McCannon noted that traditionally researchers are not rewarded academically for doing implementation and dissemination research, and breakthroughs are more valuable than successful implementation. But some universities are changing their policies about this. Johns Hopkins University, for example, is considering giving equal points in the academic advancement system for running successful implementation or scaling projects as one would acquire for publication of a breakthrough in a top-tier journal. “These are meaningful incentives to look at,” he said.
An example of implementation research Chamberlain presented at the workshop found an organization’s pre-implementation behavior before adopting a foster care treatment program predicted successful startup of the programs as well as its sustainability over time. Such pre-implementation behavior included measures of how long it took organizations to achieve
Stages of Implementation Completion (SIC)
8 Stages:
FIGURE 7-2 Stages of implementation completion.
SOURCE: Chamberlain et al., 2011.
each of the pre-implementation steps outlined in Figure 7-2. Researchers are currently conducting a study to see if these measures are predictive of successful implementation of other child mental health treatments.
Supplee gave another example of ground-breaking research on implementation called MIHOPE. This study, which stands for Mother and Infant Home Visiting Program Evaluation, is a large-scale random assignment evaluation of home visiting programs funded by MIECHV. The evaluation is expected to include about 85 program sites and 5,100 families clustered in about 12 states nationwide and will focus on those models serving expectant families and infants up to 6 months of age at enrollment (MIHOPE, 2014). The evaluation includes assessment of how baseline family and staff attributes and implementation strategies influence outcomes.
Bjørknes, R., R. Jakobsen, and A. Nærde. 2011. Recruiting ethnic minority groups to evidence-based parent training. Who will come and how? Children and Youth Services Review 33(2):351-357.
Bjørknes, R., J. Kjøbli, T. Manger, and R. Jacobsen. 2012. Parent training among ethnic minorities: Parenting practices as mediators of change in child conduct problems. Family Relations 61(1):101-114.
Chamberlain, P., C. H. Brown, and L. Saldana. 2011. Observational measure of implementation progress in community based settings: The stages of implementation completion. Implementation Science 6:116.
Chorpita, B. F., and E. L. Daleiden. 2009. Mapping evidence-based treatments for children and adolescents: Application of the distillation and matching model to 615 treatments from 322 randomized trials. Journal of Consulting and Clinical Psychology 77(3):566-579.
Dishion, T. J., and G. R. Patterson. 1999. Model building in developmental psychopathology: A pragmatic approach to understanding and intervention. Journal of Clinical Child Psychology 28(4):502-512.
HHS (U.S. Department of Health and Human Services). 2014. National prevention strategy. http://www.surgeongeneral.gov/initiatives/prevention/strategy (accessed August 26, 2014).
Hukkelberg, S., and T. Ogden. 2013. Working alliance and treatment fidelity as predictors of externalizing problem behaviors in parent management training. Journal of Consulting and Clinical Psychology 81(6):1010-1020.
Joyce, B., and B. Showers. 2002. Designing training and peer coaching: Our needs for learning. In Student achievement through staff development, 3rd ed. Alexandria, VA: Association for Supervision and Curriculum Development.
Kjøbli, J., and G. Bjørnebekk. 2013. A randomized effectiveness trial of Brief Parent Training: Six-month follow-up. Research on Social Work Practice 23(6):603-612.
Kjøbli, J., S. Hukkelberg, and T. Ogden. 2012. A randomized trial of group parent training: Reducing child conduct problems in real-world settings. Behaviour Research and Therapy 51(3):113-121.
MIHOPE (Mother and Infant Home Visiting Program Evaluation). 2014. MIHOPE project description. http://www.mdrc.org/mihope-project-description (accessed June 6, 2014).
NRC and IOM (National Research Council and Institute of Medicine). 2009. Preventing mental, emotional, and behavioral disorders among young people: Progress and possibilities. Washington, DC: The National Academies Press.
NRI (National Research Institute). 2012. Analysis of data from NRI State Profile System, 2012. http://www.nri-inc.org/#!data-sets/czyr (accessed September 9, 2014).
Schoenwald, S. K., A. J. Sheidow, and J. E. Chapman. 2009. Clinical supervision in treatment transport: Effects on adherence and outcomes. Journal of Consulting and Clinical Psychology 77(3):410-421.
Solholm, R., J. Kjøbli, and T. Christiansen. 2013. Early initiatives for children at risk—Development of a program for the prevention and treatment of behavior problems in primary services. Prevention Science 14(6):535-544.
Spoth, R., L. A. Rohrbach, M. Greenberg, P. Leaf, C. H. Brown, A. Fagan, R. F. Catalano, M. A. Pentz, Z. Sloboda, and J. D. Hawkins (Society for Prevention Research Type 2 Translational Task Force Members and Contributing Authors). 2013. Addressing core challenges for the next generation of type 2 translation research and systems: The Translation Science to Population Impact (TSci Impact) framework. Prevention Science 14(4):319-351.
This page intentionally left blank.
