Advancing the Power of Economic Evidence to Inform Investments in Children, Youth, and Families (2016) / Chapter Skim
Currently Skimming:
4 Context Matters
4 Context Matters
Pages 159-210
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 159... ...
. Apart from economic evaluations, decision makers rely on many other sources to inform their decisions, including expert opinion, community preferences, and personal testimonies (Armstrong et al., 2014; Bowen and Zwi, 2005; Orton et al., 2011)
|
From page 160... ...
The results, therefore, may not be perceived as relevant to a particular decision context or feasible to implement in a setting different from that in which the evidence was derived. In addition, findings from economic evaluations may be reported in formats that are not accessible to consumers of the evidence.
|
From page 161... ...
. As a result, the evidence produced and the evidence perceived to be relevant to a specific decision often differ as well.
|
From page 162... ...
. Ideally, economic evaluation goes beyond rigorous impact studies and associated cost studies to examine impact variability, particularly whether there are impacts for different settings, contexts, and populations, and whether and what adaptations can be effective; systems-level supports required for effective implementation; and the cost of implementation at the level of implementation fidelity required.
|
From page 163... ...
. One of the only studies of its kind on the training needs of the public health workforce in the United States identified large gaps in these decision makers' competence in the use of economic evaluation to improve their evidence-based decision making, as well as their ability to communicate research findings to policy makers (Jacob et al., 2014)
|
From page 164... ...
Finally, human resources agencies serving employees who work on interventions for children, youth, and families could provide training and opportunities for applied learning in the use of research evidence, including how to access and acquire the evidence, how to judge its quality, and how to apply it in decision making. An example of such capacity is provided in Box 4-1.
|
From page 165... ...
Their efforts represent an example of a university-community partnership that successfully improved the community's capacity to seek and use scientific evidence in its local decision making. Although the objective was to increase the community's acceptance of program evaluation data, the lessons learned could inform similar efforts to build stakeholders' capacity to use economic evaluations as an additional tool to guide investment decisions.
|
From page 166... ...
They argue that cost is an important barrier to the adoption of evidence-based practices and advocate for an approach in which intervention developers first assess what individuals and agencies are "willing to pay" for an intervention and then design interventions that are consistent with that cost range. Two recent examples illustrate potential contributions of economic evaluation to implementation studies.
|
From page 167... ...
While knowledge of the effectiveness and cost-effectiveness of interventions is growing, there remains only limited information about what is required to support effective implementation of those interventions. At the committee's June open session, panelist Mark Lipsey, director of the Peabody Research Institute at Vanderbilt University, commented that most cost-effectiveness research focuses on brand-name programs.
|
From page 168... ...
CONCLUSION: Infrastructure for developing, accessing, analyzing, and disseminating research evidence often is lacking in public agencies and private organizations charged with developing and implementing interventions for children, youth, and families. CONCLUSION: It is not sufficient to determine whether an invest ment is effective at achieving desired outcomes or provides a positive economic return.
|
From page 169... ...
The reporting of evidence derived from economic evaluation influences whether the evidence is used in decision making (National Research Council, 2012; O'Reilly, 1982; Orton et al., 2011; Tseng, 2012; Williams and Bryan, 2007)
|
From page 170... ...
Yet many of these resources currently do not incorporate economic evidence. The work of the Washington State Institute for Public Policy (see Box 4-2 and the section below on examples of efforts to improve the use of evaluation evidence)
|
From page 171... ...
. In a pluralistic society, diverse political views, cultural norms, and values help define the context within which individuals make investment decisions.
|
From page 172... ...
. A European study found that the extent of knowledge about economic evaluation, the barriers to its use, the weight given to ethical considerations, and incentives promoting the integration of economic information into health care decision making varied by country.
|
From page 173... ...
Economic evaluations then are more likely to be seen as responsive, sensitive, and relevant to the local context and to increase the demand for and uptake of such work. CONCLUSION: Political pressures, values, long-standing practices, expert opinions, and local experience all influence whether decision makers use economic evidence.
|
From page 174... ...
. Given these limitations, there are nonetheless three dimensions in which budget processes could be improved to take better advantage of the evidence derived from economic evaluations: (1)
|
From page 175... ...
Thus they need to know what economic evaluations are available, not available, planned, and not planned for programs falling within their budget. For example, OMB could list annually which programs do and do not have economic evaluations planned as part of their ongoing assessment, where the evaluations exist, and what has been evaluated.
|
From page 176... ...
Statistical entities, such as the Census Bureau and the IRS's Statistics of Income Program or those associated with state K-12 and early childhood education, could gain more from their limited budgets if they gave significant budget weight to the development of longitudinal data following individuals. Foundations interested in economic evaluation could also assess the relative importance of a new experiment requiring new data development and more investment in data that could inform multiple investments.
|
From page 177... ...
Even a particular agency, as long as its goal is the well-being of constituents, can mitigate its own tendency to localize decision making by reporting economic evaluations across program areas, even those not under their jurisdiction. Budget offices also can identify for policy makers and administrators incentives that might offset built-in tendencies to account only for local costs and benefits.
|
From page 178... ...
They also use different types of administrative data and leverage those data in different ways. Figure 4-1 depicts the potential uses of administrative data in economic evaluations.
|
From page 179... ...
. different from those used for other types of economic evaluation.
|
From page 180... ...
Administrative data often are not stored centrally. Local data systems tend to use varied formats, archive differing information, use incompatible file formats, and sometimes overwriting data instead of archiving them.
|
From page 181... ...
CONCLUSION: Without a commitment by government to the devel opment of linkages across administrative data sets on education, health, crime, and other domains, both longitudinally and across systems, ef forts to expand the evidence base on intervention impacts and evidence of economic returns will be limited.
|
From page 182... ...
Leadership and Openness to Learning Some organizations have a culture or characteristics that are supportive of the use of evidence, including the results of economic evaluation, such as leaders and managers who value economic evaluations and have sufficient BOX 4-3 The Impetus for Economic Evaluation: Examples Proposals internal to an organization, as well as legislation with a direct impact on the organization's budget, will frequently generate cost analysis (CA)
|
From page 183... ...
(See the section above on budget considerations for discussion of budgeting for the development of data in advance of future economic evaluations.) Further discussion of the importance of a culture of learning is included in the section below on performance management.
|
From page 184... ...
Agencies may focus on success of the tasks being measured or on one type of benefit, and neglect other goals or the broader picture. Accountability typically involves a top-down approach, whereas economic evaluation should be considered valuable in strengthening, not just threatening, decision makers.
|
From page 185... ...
A more modest and attainable accountability system might first empha size obtaining better measures of individual student progress that are useful to teachers and principals (e.g., as early warning signals of a student's no longer making progress) , as well as for performing multiple levels of experimentation amenable to future economic evaluation.
|
From page 186... ...
Moynihan (2008) argues that performance data (of which economic evaluation is one type)
|
From page 187... ...
A Potential Role for Funders How might the broad conclusions on organizational culture and a continuous learning process presented in this section influence public and private funders? In sponsoring economic evaluation, funders often explicitly or implicitly seek or rely on a theory of causality: How do particular activities in this particular analysis result in specific outcomes?
|
From page 188... ...
. A systematic review of 145 articles on the use of evidence in policy making in 59 different countries found that the factor that most facilitated use was collaboration between researchers and policy makers, identified for two-thirds of the studies in which use was achieved (Oliver et al., 2014a)
|
From page 189... ...
. It is also important to recognize that economic evaluations may be applied to questions and settings beyond the original purpose of the evaluation.
|
From page 190... ...
Such models are intended to integrate the best available research evidence with local contextual factors, such as community norms, political preferences, and available resources, leading to decisions better tailored to the local context (Tseng, 2014)
|
From page 191... ...
This section describes examples of ongoing efforts to address these factors. The examples in this section are intended to be illustrative of the points discussed throughout the chapter; they do not represent the total range of innovative ways in which state and local governments across the country are partnering with practitioners and intermediaries to improve the use and usefulness of research evidence in general and economic evidence in particular.
|
From page 192... ...
As a result, they set out to develop the capacity of key personnel in local government agencies in Victoria to access research evidence, assess its trustworthiness, and apply it to the local context, as well as to implement strategies that could foster an organizational culture supporting evidence-informed decision making across these agencies. A statewide survey and a series of individual interviews with members of the target audience helped shape the development of a multi pronged, resource-intensive intervention that included tailored organizational sup port, group trainings, targeted communications, and the development of evidence summaries of relevant content, all of which contributed to both individual and organizational improvements in the use of evidence.
|
From page 193... ...
. Results First also ensures that analyses are relevant to the local context by replacing Washington State's data with data specific to each jurisdiction and by helping to implement analyses requested by states.
|
From page 194... ...
Several features distinguish the UChicago CCSR from other research organizations, many of them related to factors that influence the use of evidence. First, by focusing on one place -- Chicago -- the consortium builds a perception among its main target group of users that its work is relevant to the local context.
|
From page 195... ...
In addition, because nonprofit organizations are eligible for i3 funding only if they partner with one or more LEAs or a consortium of schools, i3 incentivizes the development of relationships and a political environment conducive to the effective use of the evidence generated by funded activities. At the committee's March open session, panelist Nadya Dabby, assistant deputy secretary for innovation and improvement at the U.S.
|
From page 196... ...
Such financing models -- referred to here as pay for success (PFS) but also known by various other names, such as social impact bonds, outcome 6 See, for example, a brief discussion of the return on investment of Functional Family Therapy at http://www.episcenter.psu.edu/sites/default/files/ebp/MST-Three-Year-Report-ROI.
|
From page 197... ...
. In particular, PFS financing has garnered growing interest within the United Kingdom and the United States (Callanan and Law, 2013; Corporation for National and Community Service, 2015)
|
From page 198... ...
municipalities and states have launched PFS arrangements to fund programs with an empirical record of preventing recidivism among juvenile offenders, reducing emergency care costs for children with asthma, and reducing utilization of special education among at-risk youth (Brush, 2013; Olson and Phillips, 2013)
|
From page 199... ...
, and organizations that specialize in supporting PFS projects (Corporation for National and Community Service, 2015) .10 Two key objectives of PFS financing are identifying programs that work and limiting government financing of those that do not work.
|
From page 200... ...
Related to "wrong pocket" issues discussed in Chapter 3, without greater flexibility and coordination between levels of government, using economic evidence can be more difficult. CONCLUSION: Growing interest in performance-based financing ef forts is likely to increase the importance of economic evidence in deci sions on investments in children, youth, and families.
|
From page 201... ...
RECOMMENDATION 4: To achieve anticipated economic benefits and optimize the likelihood of deriving the anticipated outcomes from evidence-based interventions, public and private funders13 should en sure that resources are available to support effective implementation of those interventions. Strategies for implementing this recommendation might include the following: • Support intermediary organizations that can provide training and technical assistance in the implementation of evidence-based inter ventions and work collaboratively with implementing organiza tions to ensure effective implementation.
|
From page 202... ...
. Research evidence and school board deliberations: Lessons from three Wisconsin school districts.
|
From page 203... ...
. Strengthening national decision-making on immunization by building capac ity for economic evaluation: Implementing ProVac in Europe.
|
From page 204... ...
. A systematic review of the use of economic evaluation in local decision-making.
|
From page 205... ...
. Economic evaluation research in the context of child welfare policy: A structured literature review and recommendations.
|
From page 206... ...
. Economic evaluation of public health laws and their enforcement.
|
From page 207... ...
. The use of research evidence in public health decision making processes: Systemic review.
|
From page 208... ...
. Use of economic evaluation in decision making: Evidence and recommen dations for improvement.
|
From page 209... ...
. Understanding the limited impact of economic evaluation in health care resource allocation: A conceptual framework.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.