Appendix A
Validation Exercise
This appendix discusses and compares two of the most comprehensive indices of the rule of law (ROL): (a) the World Justice Project (WJP)1 and (b) the Varieties of Democracies (V-DEM) Project.2 Both projects provide a composite ROL Index through a multidimensional set of outcome indicators, each of which reflects a particular aspect of this complex concept. The two measures are highly correlated.
The evidence reviewed by the committee suggests that research can, in principle, measure whether police reforms can support basic tenets of and potentially improve the ROL. The committee identified candidate measures that satisfy requirements of basic face validity and are regularly produced for multiple countries over multiple years.
Many of the indicators in the V-DEM survey reflect specific components of indicators in the WJP indices. However, there are no explicit questions in the V-DEM survey distinguishing civil from criminal justice, or order and security. In contrast, the WJP survey does not explicitly ask about gendered rights or about white-collar crime by government and private officials, nor does it distinguish between higher and lower courts.
In order to validate the measures, we run two simple analyses. First, we correlate V-DEM and WJP indices; the idea is that a high correlation would offer face validity. Second, we explore whether the ROL Index captures which countries are more likely to violate human rights.
___________________
1 See https://worldjusticeproject.org/our-work/research-and-data/wjp-rule-law-index-2020.
Using the last year for which the two leading sources (V-DEM and WJP) have data, Figure A-1 below shows a rather high correlation (0.81) between the two composite indices, which is reassuring.
In further analyses, we compare V-DEM’s index, since it has a broader country coverage than the WJP index, to a measure of human rights violations. We use the Latent Human Rights Protection Scores constructed by Fariss, Kenwick, and Reuning (2020).3 As Figure A-2 makes clear, there is a strong positive and significant relationship between V-DEM’s Rule of Law Index and a country’s human rights score.
NOTE: Each dot represents a country, with the x-coordinate its WJP ROL Index value and the y-coordinate its V-Dem ROL Index value (data source: https://www.v-dem.net/en/data/data/v-dem-dataset-v111/). In total, there are 118 countries represented in the figure that match between the 128 countries in WJP data and 179 in V-Dem data.
___________________
3 The Latent Human Rights Protection Scores can be found at https://dataverse.harvard.edu/dataverse/HumanRightsScores.
NOTE: Each dot represents a country, with the x-coordinate its human rights score (data source: https://dataverse.harvard.edu/dataverse/HumanRightsScores) and the y-coordinate its V-Dem ROL Index value (data source: https://www.v-dem.net/en/data/data/v-dem-dataset-v111/). In total, there are 170 countries represented in the figure that match between the 194 countries in the human-rights-score data and 179 in V-Dem data.
Correlates of the Rule of Law
In this section, the correlates of the ROL are explored with no causality claims made: the goal is more modest, and limited to correlational analysis.
While in theory the ROL and a country’s level of democracy are two distinct concepts, in practice both are highly correlated, as Figure A-3 makes clear. Here, V-DEM’s index of Electoral Democracy is used; it has a 0.82 correlation with the ROL Index.
The strong correlation between regime type and the rule of law is not simply a function of how V-DEM constructs both measures. Replacing V-DEM’s continuous measure of democracy with Freedom House’s
NOTE: Each dot represents a country, with the x-coordinate its electoral democracy index, v2x_polyarchy, (data source: https://www.v-dem.net/en/data/data/vdem-dataset-v111/) and the y-coordinate its V-Dem ROL Index value. There are 172 countries included. Note that R in the plot indicates correlation score, not R-squared for regression lines.
three-category measure of democracy (Free; Partially free; Not free),4 a strong correlation is again found between regime type and the rule of law; see Figure A-4.
Interestingly, the strong positive correlation between a country’s level of democracy and its commitment to the rule of law is observed at all income levels. Using the World Bank’s income designation,5 we disaggregate Figure A-3 to three categories: high-income, middle-income, and low-income countries. As Figure A-5 shows, a positive correlation between a country’s
___________________
4 See https://freedomhouse.org/report/freedom-world.
5 See https://datahelpdesk.worldbank.org/knowledgebase/articles/906519-world-bank-countryand-lending-groups.
NOTE: There are 175 countries included. Freedom level data from Freedom House measure of democracy (https://freedomhouse.org/countries/freedom-world/scores).
level of democracy and the rule of law can be observed at all income levels (though it is admittedly weakest for low-income countries).
Figure A-5 strongly suggests that a country’s wealth is correlated with the rule of law independent of a country’s level of democracy. We explore this correlation in Figure A-6, which examines wealth per capita. As the figure illustrates, with few exceptions, on average, richer countries exhibit stronger commitment to the rule of law.
NOTE: Each dot represents a country, with the x-coordinate its electoral democracy index, v2x_polyarchy, (data source: https://www.v-dem.net/en/data/data/v-dem-daaset-v111/) and the y-coordinate its V-Dem ROL Index value, subgrouped by World Bank’s income designation. There are 172 countries included. Note that R in the plot indicates correlation score, not R-squared for regression lines.
NOTE: Each dot represents a country, with the x-coordinate its GDP per capita in 2019 (data source: (https://api-worldbank-org.proxy.library.upenn.edu/v2/en/indicator/NY.GDP.PCAP.CD?downloadformat=csv) and the y-coordinate its 2019 V-Dem ROL Index value. There are 162 countries included.
This page intentionally left blank.
