Constructing Valid Geospatial Tools for Environmental Justice (2024) / Chapter Skim
Currently Skimming:
7 Importance of and Methods for Tool Validation
7 Importance of and Methods for Tool Validation
Pages 167-188
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 167... ...
Validation confirms the accuracy and can increase community acceptance of a tool and its results, but it is also a process for justifying decisions made throughout tool development. Tool validation can also help identify outlier places that do not fit the general pattern, as analyzing them can provide insights into possible data problems and new variables.
|
From page 168... ...
. Deliberate and intentional community engagement is another form of validation that incorporates perceptions of affected parties and lived experiences (see Box 7.1)
|
From page 169... ...
. Community valida tion results from active engagement and partnership with disproportionately impacted communities, which are often also disadvantaged communities.
|
From page 170... ...
Tool developers may consider: • How do the EJ tool results and methodologies compare to other EJ tools addressing similar geographies and topics? • How well do the EJ tool results represent lived experiences and match the knowledge of those in disadvantaged communities?
|
From page 171... ...
Tool developers, such as CEQ, can validate data, approaches, and tool results through community engagement in a variety of ways: through targeted outreach to interested and affected parties, workshopping in disadvantaged communities via small-group discussions, providing office hours to respond to questions or receive input,5 or through creative approaches such as data challenges or hack-a-thons. Community validation may also take the form of community peer review (see Box 7.2)
|
From page 172... ...
Liboiron, Zahara, and Schoot (2018) describe steps in the community peer review process that can be applied to EJ tool development (see list below)
|
From page 173... ...
. However, institutionalizing these concepts and frameworks may have negative results, such as devaluing information about lived experience gained through community validation.
|
From page 174... ...
Because in-depth engagement cannot realistically occur at the national level with all communities represented by a tool, methods to identify representative communities, to design tool feedback methodologies, and to validate decisions made during indicator construction will necessarily be important in the design of an engagement program. Ground Truthing In mapping applications, ground truthing refers to the collection of reference data tied to specific locations on Earth's surface used to gauge the validity of a theoretical model (e.g., Yonto and Schuch, 2020)
|
From page 175... ...
evaluated three cumulative impacts with EJ tools in California for hazardous waste facility locational accuracy and found location errors of up to 10 km. As CEQ considers validating input datasets and output measures of disadvantage, they can build on existing ground-truthing ap proaches to achieve greater validity.
|
From page 176... ...
Although their research is focused on patient-centered medical home models, their methods can be adapted to EJ tool development and evaluation. They describe five core characteristics of well-designed mixed-methods studies: 1.
|
From page 177... ...
But because not all information that would benefit an EJ composite indicator is quantitative, mixed-method approaches provide the creative means to incorporate lived-experience data with statistical techniques. Although mixed methodologies challenge the status quo and may be difficult to plan and execute, their use will result in data interpretation and informed research practices that allow for the incorporation of lived experiences into data analyses while providing a pathway to tool validation.
|
From page 178... ...
Discussion among multiple workshop participants suggested that economic disparities can be captured with national income and poverty measures but are not currently reflected in CEJST. Input to Define Indicator Integration Community input can reveal the burdens of a particular community, measures for those burdens, and how those burdens combine to affect lived experiences.
|
From page 179... ...
It is important for CEQ to determine the effectiveness of its communication through community engagement. The committee's workshop included a hands-on CEJST exercise intended to gather input regarding tool results (for validation)
|
From page 180... ...
SUPPLEMENTAL ANALYSIS OF EXTERNAL VARIABLES Supplemental analysis in the context of EJ tools refers to additional analysis or assessment conducted outside of the tool using independent external datasets. It is distinct from robustness analysis discussed in Chapter 6 for internally validating indicators during tool construction; supplemental analysis is useful for externally validating the outputs of EJ tools.
|
From page 181... ...
For example, supplemental analysis comparing the distribution of race/ethnicity indicators and CEJST outputs could help CEQ tool developers gain a greater understanding of how well CEJST captures community disadvantage in its current formulation. Researchers at the Bullard Center for Environmental Justice developed an interactive map that does this.14 The map overlays proportional symbols showing the number of CEJST categories exceeded with choropleth symbols showing the percent people of color in communities.
|
From page 182... ...
The California Office of Environmental Health Hazard Assessment (OEHHA) has conducted supplemental analysis of race and ethnicity alongside vintages of CalEnviroScreen since 2013 (CalEPA, 2021)
|
From page 183... ...
In response to criticisms about the absence of race as an indicator for disadvantaged communities, CEQ argued that their focus on income and environmental burdens would still effectively capture communities of color due to strong correlations between environmental and social inequities and the proportion of non-white residents (Friedman, 2022)
|
From page 184... ...
. The WRI report observed that in disadvantaged communities meeting at least one indicator threshold (i.e., the current methodology in CEJST)
|
From page 185... ...
Tool validation techniques can be applied to allow tool developers to create a tool that is stable, accepted, and scientifically sound. Different validation approaches are available, for example: • Convergent validation compares tool components or results with those of similar tools.
|
From page 186... ...
Consistent engagement throughout the tool development or upgrading process allows developers to test decisions, approaches, and tool results against community member narratives, while empowering communities to accept or refute definitions being assigned to them and gaining trust in the tool development process. • Mixed methods that allow collection and analysis of both qualitative and quantitative datasets are framed withing research models to better understand multiple perspectives of any issue and are well suited for tool validation.
|
From page 187... ...
CEQ might conduct supplemental analysis to, for example, compare the distribution of race/ethnicity indicators and CEJST outputs to test the validity of CEJST's current formulation. Such analysis could help tool developers check indicator data sources for potential gaps or inaccuracies.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.