Improving Democracy Assistance Building Knowledge Through Evaluations and Research (2008) / Chapter Skim
Currently Skimming:
8 Creating the Conditions for Conducting High-Quality Evaluations of Democracy Assistance Programs and Enhancing Organizational Learning
8 Creating the Conditions for Conducting High-Quality Evaluations of Democracy Assistance Programs and Enhancing Organizational Learning
Pages 199-218
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 199... ...
and implementers from undertaking high-quality impact evaluations. Reviewing agency policies and practices with the goal of reducing barriers to and strengthening incentives for conducting sound impact evaluations is essential.
|
From page 200... ...
Incentive Issues A key problem, not unique to DG or USAID, is the question of providing incentives to DG staff and implementers to undertake and complete sound and credible impact evaluations. The DG officers and implementers the committee and its field teams met shared a strong desire to be successful in promoting democracy.
|
From page 201... ...
Without strong incentives to complete sound impact evaluations on at least some DG programs and some rewards for doing so, these pressures make it highly unlikely that such evaluations will be designed into DG programs. One task of the DG evaluation initiative recommended in the next chapter should be to address these issues and explore how to ease the task of undertaking impact evaluations within USAID's contracting and program procedures.
|
From page 202... ...
Contracting Issues The committee's research and field visits also found that the current process of awarding contracts and grants actually works against conducting impact evaluations in a number of specific ways: • DG officers are chosen for expertise in democracy assistance and aid delivery, not for expertise in evaluation designs. Thus DG officers often felt they lacked expertise among their mission staff to prescribe or judge what would be an effective, high-quality impact evaluation design.
|
From page 203... ...
This means that if a mission wants to obtain sound evidence of the impact of a particular project, staff will need to think about planning an impact evaluation before they have even drawn up the call for proposals for that project and make a suitable design for impact evaluation part of the original action and budget plan for that project. Call for Proposals When a USAID mission undertakes a new project or the next phase of a continuing one, in most cases there is a formal request for bids, called a Request for Proposals (RFP)
|
From page 204... ...
If a more detailed statement is considered preferable, a recent RFP in one of the missions that the field teams visited provides an example. As part of the performance monitoring plan called for in the RFP for the Democratic Linkages project in Uganda, bidders were told they should have "a clearly developed strategy for assessing the impact of the program at all three levels [national, district, and subcounty] by evaluating outcomes over time (comparing pre-intervention and post-intervention values on impact variables)
|
From page 205... ...
For those select projects for which DG officers want sound impact evaluations, contracting schedules for implementers need to allow for the implementation of an appropriate evaluation design, including establishing an appropriate control or comparison group and setting up and completing baseline measurements on both the assistance and the control groups. Policymakers may need to be reminded that rushing to roll out projects without allowing for careful examination of initial conditions and creation of comparison groups undermines the only way to accumulate knowledge on whether those DG projects are working as intended and those expenditures are worthwhile. Keeping Project Evaluation Independent Ideally, the individuals or contractors who implement a project should not be the only ones involved in evaluating its outcomes.
|
From page 206... ...
To minimize the risk of collusion, USAID may have to require contractors who implement a large number of projects for USAID DG offices to work with several different evaluation partners; similarly, evaluation contractors should be required to partner with several different implementers over time in order to ensure continued independence of project and evaluation agents. Resource Issues One of the major objections to impact evaluations that the committee and its field teams encountered is that they "cost too much." The collection of high-quality baseline data and indicators, especially since it must be done for both those who receive the DG support and a control group that does not, can be costly, although Chapters 6 and 7 discuss ways in which at least some of those costs could be reduced.
|
From page 207... ...
requirements, these set out "strategic objectives" and "intermediate results" with corresponding results indicators. Many missions will spend money to have consultants train mission staff in developing PMPs and/or help develop them.
|
From page 208... ...
Improving Organizational Learning The results of sound impact evaluations have value for USAID only when they become readily accessible knowledge for USAID officers and that knowledge feeds into learning processes that inform policy and planning. This section looks at what happens to the results of evaluations and other data after they are obtained.
|
From page 209... ...
Ensuring that survey data are retained would probably require an executive decision at the bureau level or higher to impose an absolute contractual requirement that the data generated would be deposited with USAID Washington. The committee recognizes that the barriers to doing so are real, as many of USAID's DG programs are carried out by consulting firms whose contractual clauses broadly prohibit the use of their data beyond the confines of the company.
|
From page 210... ...
The earliest systematic surveys of entire regions emerged in Europe with the development of the Eurobarometer and since 2001 the emergence of the European Social Survey, which now covers 25 nations in the broadened European community. Other regions of the world also are covered by such surveys, including Eastern Europe, now included in the Eurobarometer; the New Europe Democracies Barometer, which covers much of the former Soviet Union and is currently based at the University of Aberdeen; the Asian Barometer, currently based at the National Taiwan University; and, most recently, the Arab Barometer, currently based at Princeton University and the University of Michigan. Recent studies by several of these democracy barometers can be found in the July 2007 and January 2008 issues of the Journal of Democracy.
|
From page 211... ...
the Americas Barometer, organized by the Latin American Public Opinion Project of Vanderbilt University and its partner university and think tanks in Latin America, led by the University of Costa Rica. The committee believes that greater international coordination among existing surveys should be sought and supported.
|
From page 212... ...
Videoconferencing and other advanced technologies can be an important supplement, but personal contact and discussion would be extremely important to share experiences of success and failure as the evaluation initiative goes forward. This includes lessons about the effectiveness of DG projects and about successes and failures in implementing impact evaluations.
|
From page 213... ...
Whatever the methodological flaws of these case studies and process evaluations from a rigorous social sciences perspective, this expanding literature has provided important lessons and insights for crafting effective DG programs. Turning Individual Experience into Organizational Experience: Voices from the Field Realizing that its DG officers often had valuable insights and experiences gained from years of implementing projects in various conditions around the world, USAID's Democracy Office began a pilot project under its Strategic and Operational Research Agenda (SORA)
|
From page 214... ...
They expressed the desire to have a voice in sharing smaller everyday successes, which are often overlooked by bigger projects, programs, and efforts. Finally, if these interviews were undertaken on a larger scale in the The field teams in Albania and Uganda met equally experienced foreign service nationals.
|
From page 215... ...
There was great interest in learning from their experiences as well as those of colleagues around the world; therefore they hoped that information from such programs would flow both in from and out to the field missions. Depending on the interview design, information collected through a Voices from the Field project focused on systematic debriefings of DG officers, and foreign service nationals could offer very detailed information on project implementation or more general insights about potential sources of project success or failure.
|
From page 216... ...
Conclusions The potential changes to current USAID policy and practices discussed in this chapter range from specific suggestions for the contracting process to a broad shift in the organization toward a much more systematic effort to share and learn from its own work and that of others. In the next chapter we introduce a set of specific recommendations based around a DG evaluation initiative intended to increase the capacity of USAID to support and undertake a variety of well-designed impact evaluations, and to improve its organizational learning.
|
From page 217... ...
Improving Lives Through Impact Evaluation. Washington, DC: Center for Global Development.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.