Improving Democracy Assistance Building Knowledge Through Evaluations and Research (2008) / Chapter Skim
Currently Skimming:
3 Measuring Democracy
3 Measuring Democracy
Pages 71-98
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 71... ...
The committee believes that democracy is an inherently multidimensional concept, and that broad consensus on those dimensions and how Helpful comments on this chapter were received from Macartan Humphreys, Fabrice Lehoucq, and Jim Mahoney. The committee is especially grateful to those who attended a special meeting on democracy indicators held at Boston University in January 2007: David Black, Michael Coppedge, Andrew Green, Rita Guenther, Jonathan Hartlyn, Jo Husbands, Gerardo Munck, Margaret Sarles, Fred Schaffer, Richard Snyder, Paul Stern, and Nicolas van de Walle.
|
From page 72... ...
While recognizing that it may seem self-serving for an academic committee to recommend "more research," it is the committee's belief -- after surveying the academic literature and convening a workshop of experts in democracy measures to discuss the issue -- that if USAID wishes a measure of democracy that it can use to gauge the impact of its programs and track the progress of countries in which it is active, it faces a stark choice: either rely on the current flawed measures of democracy or help support the development of a research project on democracy indicators that -- it is hoped -- will eventually produce a set of indicators with the broadly accepted integrity of today's national accounts indicators for economic development. To provide just a few examples to preview the discussion below, USAID manages its DG programs with an eye toward four broad areas: rule of law, elections, civil society, and good governance.
|
From page 73... ...
Freedom House rates nations on two scales: civil liberties (which conflates rule of law, civil society, and aspects of good governance) and political rights (which conflates rule of law, elections, and aspects of good governance)
|
From page 74... ...
. Freedom House provides two indices: "Political Rights" and "Civil Liberties" (sometimes employed in tandem, sometimes singly)
|
From page 75... ...
For example, the Freedom House Political Rights Index includes the following questions pertaining to corruption: Has the government implemented effective anticorruption laws or pro grams to prevent, detect, and punish corruption among public officials, including conflict of interest? Is the government free from excessive bu reaucratic regulations, registration requirements, or other controls that increase opportunities for corruption?
|
From page 76... ...
As noted above, Freedom House scores democracy on a seven-point index (14 points if the Political Rights and Civil Liberties indices are combined)
|
From page 77... ...
Lucia, Suriname, Sweden, Switzerland, Tuvalu, United Kingdom, United States, and Uruguay. Are we really willing to believe that there are no substantial differences in the quality of democracy among these diverse polities? Measurement Errors and Data Coverage Democracy indicators often suffer from measurement errors and/or missing data.10 Some (e.g., Freedom House)
|
From page 78... ...
Note that it is the ambiguity of the questionnaires underlying these surveys that fosters this sort of premature aggregation. The committee undertook a limited statistical examination of the Freedom House scores for 2007 on their key components -- for political rights this included electoral process, pluralism and participation, and functioning of government; for civil liberties these were freedom of expression, association and organizational rights, rule of law, and personal autonomy and individual rights (see Appendix C)
|
From page 79... ...
. Convergent Validity Given the above, it is no surprise that there is significant disagreement among scholars over how to assign scores for particular countries on
|
From page 80... ...
. Since tracking progress in democracy assistance often depends on accurately measuring modest improvements in democracy, it is particularly distressing that the convergence between different scales is so low in this regard.
|
From page 81... ...
The answers to this question lie in the very different purposes that democracy indicators must serve for scholarly analysis of average or overall global trends, as against the purposes they must serve to support policy analysis of trends in specific countries. For the former purpose it is acceptable for democracy data to have substantial errors regarding levels of democracy in particular states, as long as the errors are not systematically biased.
|
From page 82... ...
described above, which demonstrate a positive impact of various forms of democracy assistance on average levels of democracy while statistically controlling for a host of background, trend, and other causal variables, also controlled for measurement errors in the democracy indices that were assumed to be evenly distributed across countries. What their results tell us is something like the following: In any four-year period, if three countries are examined in which USAID invested an average of $10 million per country per year in DG assistance, those countries' Freedom House scores will show an overall increase of three points (an average increase of one point per country)
|
From page 83... ...
Thus the errors found in current widely used democracy indicators, while still allowing them to serve well enough for purposes of scholarly research on average effects of various factors on democracy or for charting overall democracy trends, do not serve USAID at all well for the policy purposes of determining the effects of specific programs in particular countries.13 For this reason the rest of this chapter lays out an approach that the committee believes will be more fruitful for developing useful indicators of democratic change. Also for this reason, throughout this report methods are stressed for helping USAID determine the effects of its programs using more concrete indicators of the immediate policy outcomes of those programs, rather than macrolevel indicators of national levels of democracy.
|
From page 84... ...
and/or because the underlying components, while conceptually distinct, are gathered in such a way as to compromise their independence. Consider the six World Bank governance indicators -- government effectiveness, voice and accountability, control of corruption, rule of law, regulatory burden, and political instability -- which involve very similar underlying components (Landman 2003, Kurtz and Schrank 2007, Thomas 2007)
|
From page 85... ...
A similar problem besets other efforts at disaggregation, such as the recently released Freedom House measures of civil liberties and political rights, which are broken down into seven components: electoral process, political pluralism and participation, functioning of government, freedom of expression and belief, associational and organizational rights, rule of law, personal autonomy, and individual rights (Freedom House 2007)
|
From page 86... ...
, as secured through an electoral process by which leaders are selected. Where effective competition exists, democracy is also said to exist (Schumpeter 1942, Alvarez et al 1996)
|
From page 87... ...
11. Civil Society: Is civil society dynamic, independent, and politically active?
|
From page 88... ...
For example, strong civil society organizations representing one social group may pressure government to restrict other citizens' civil liberties (Levi 1996, Berman 1997)
|
From page 89... ...
. Indeed, because current democracy scales are produced by proprietary scalings and aggregations by specific organizations rather than by objective measurements, those organizations are often subjected to "lobbying" by countries that wish to shift their scores.
|
From page 90... ...
Contrasts and comparisons may become correspondingly more acute. Applying the Proposed Index to Democracy Assistance Programming It is important to remember that, although the committee's general goal is to provide a path to democracy measures that will be useful to policymakers and citizens alike, the specific charge is to assist USAID.
|
From page 91... ...
, but we have no systematic knowledge of how a country should be scored on the various components of democracy. Since a disaggregated index can be aggregated in a variety of ways, developing a disaggregated index is advantageous even if a single aggregated measure is sometimes desired for policy purposes.
|
From page 92... ...
If the results were not robust, it might be concluded that such conclusions were contingent on a particular way of putting together the components of democracy. In short, both policy and scholarly discourse might be much improved by a disaggregated index, even if the ultimate objective involves the composition of a highly aggregated index of Big D democracy.
|
From page 93... ...
As a management structure, the committee proposes an advisory group to be headed by academics -- with some remuneration, depending on the time requirements, and suitable administrative support -- in partnership with the policy community.18 This partnership is crucial, for any widely used democracy assessment tool should have both a high degree of academic credibility and legitimacy among policymakers. Major shortcomings of previous efforts to develop indices of democracy and governance resulted from insufficient input from methodologists and subject specialists or lack of broad collaboration across different stakeholders.
|
From page 94... ...
• Existing empirical indicators of democracy are flawed. The flaws extend to problems of definition and aggregation, imprecision, measurement errors, poor data coverage, and a lack of convergent validity.
|
From page 95... ...
This should not only involve multiple agencies and actors in efforts to initially develop a widely accepted set of sectoral data on democracy and democratic development but should seek to institutionalize the collection and updating of democracy data for a broad clientele, along the lines of the economic, demographic, and trade data collected by the World Bank, United Nations, and International Monetary Fund. While creating better measures at the sectoral level to track democratic change is a long-term process, there is no need to wait on such measures to determine the impact of USAID's DG projects.
|
From page 96... ...
2007. Presentation to Democracy Indicators for Democracy Assistance.
|
From page 97... ...
2002. Conceptualizing and Measuring Democracy: Alterna tive Indices.
|
From page 98... ...
2000. A New Dataset for Measuring Democracy, 1810-1998.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.