5
Building Confidence in Data and Institutions
A significant barrier facing community leaders and decision makers is a mistrust of data and institutions. In this session, speakers highlighted the challenges to gathering trusted data, such as community skepticism, discussed evaluating data for inclusion and comprehensive representation, and shared efforts to engage the broader community in the process.
5.1 DRIVERS OF DATA AND ANALYTICS USE WITHIN (SMART) CITIES
Rishee K. Jain, Stanford University, described his work as director of the Urban Informatics Lab to analyze data to understand interactions among people, buildings, and energy systems in cities. Data and analytics use cases differ by city. For example, in Chicago, lead contamination was identified with a predictive model of exposure that was then used to target home inspections. In New Orleans, an analytical model of optimal placement was used to optimize ambulance location to improve ambulance response time. Jain observed that different cities have different combined factors of importance: while the problem in Chicago related to data infrastructure, organizational structures, and analytical capability, the problem in New Orleans related to leadership, processes, and capacities.
To better understand which combination of factors generates causal pathways to cities’ use of data and analytics, Jain and his colleagues considered the following questions: What causal condition variables could enable cities to use data and analytics, and how can they be measured
within and across city case studies? How can the outcome variable of cities’ use of data and analytics be measured within and across city case studies? What potential causal pathways emerge that lead to cities’ use of data and analytics? The first phase of research included 22 comparative city case studies across the United States as well as content analysis, which included 56 unstructured interviews with chief data/analytics officers and an analysis of existing literature. The comparative case study results were used to generate a map of city condition variables to understand what was driving the outcome variable (i.e., the intention, combination, frequency, tools and methods, and purpose for a city’s use of data and analytics). The insights from the content analysis were used to classify the condition variables into six categories: organization, procedures, direction, data, competencies, and resources. The comparative case study and content analysis models were merged, and key drivers of abilities to adopt and use data and analytics surfaced. This served as an entry point to the second phase of research, the Delphi method, in which the information from Phase 1 was provided to a panel of experts to build consensus in understanding key variables. This panel of experts participated in multiple rounds of surveys, rating the significance of each variable from the model until consensus was reached: 77 percent consensus on 11 condition variables (structures, processes, leadership, strategy, culture, data infrastructure, data governance, skills, training, capacities, and budgets) and 60 percent consensus on 3 outcome variables (frequency, tools and methods, and purpose) that are important for understanding cities’ uses of data and analytics.
Jain explained that practitioners could benefit from newly generated understanding provided by this type of analysis, with the ability to prioritize factors that drive cities’ use of data and analytics; strengthen the perspective of cities as socio-technical units; and shift the focus from physical urban infrastructure to the data and analytics overlay. He cautioned that data governance will remain a challenge in this “new data world,” and that, in the words of John Naisbitt, “we are drowning in data but starving for knowledge.”1
Serving as session moderator, Jeanne Holm, City of Los Angeles, posed a question about the Delphi method as well as alternatives to that approach. Jain elaborated that the Delphi method is a way to validate findings using a group of experts. Other methods were considered (e.g., surveys with regression analysis); however, these methods were not chosen because they did not accommodate the diversity of the cities’ challenges. Thus, the Delphi method served as a more effective starting point to highlight these key challenges and make space for future work.
___________________
1 J. Naisbitt, 1982, Megatrends: Ten New Directions Transforming Our Lives, New York: Warner Books.
5.2 STRATEGIES TO SUPPORT EFFECTIVE DECISION MAKING
Richard Y. Wang, Massachusetts Institute of Technology (MIT), responded to several questions that are key to developing strategies to build confidence in data and institutions. First, how do institutions determine whether they have the right data, and how do they know if these data are sufficiently accurate? Wang explained that data quality is a multidimensional concept that extends beyond accuracy; the “right” data are high-quality data, and “high-quality data” are data that are fit for use. He described building confidence in data and institutions as a “journey”: at different stages and in different contexts, the “right” data could be accurate, timely, complete, and/or consistent, as well as believable and credible. Understanding the multidimensional nature of data enables a better framing of problems.
Second, what communication strategies are effective in building decision makers’ and community members’ confidence in the data that support decision making? Wang suggested that decision makers and the data community use the data that support their decision making and report their experiences about the strengths and weaknesses of those data. He also encouraged decision makers to perform gap analysis to identify opportunities to improve data quality as well as to learn about the best practices of chief data officers (e.g., via MIT’s Chief Data Officer and Information Quality Symposium2).
Third, what strategies lead to effective community engagement? Wang noted that both top-down and bottom-up approaches to community engagement are valuable. He proposed that all organizations establish a formal position of chief data officer as well as initiate data quality projects that better educate decision makers and data community members. He shared his motto to “think big, start small, move fast, and get the job done,” with a continual focus on value propositions. Fourth, how should organizations and governments balance the need for data with constituents’ rights to privacy? Wang advised organizations to appoint a chief data officer, chief privacy officer, chief security officer, and chief technology officer as part of a center of excellence that would report to a chief executive officer or a board of directors.
___________________
2 To watch videos from the symposium, visit https://www.youtube.com/channel/UCBkrYA_7_rZURjtPzBG_EIw/videos, accessed August 28, 2022.
5.3 CREATING A COMMUNITY OF PRACTICE FOR DATA COLLECTION
Karen Abrams, Pittsburgh Department of City Planning, displayed a historic redlining map of Pittsburgh that revealed the disinvestments in neighborhoods where Black, Latinx, and Asian people lived and still live. She noted that Pittsburgh’s Black population is dwindling as more people move to the suburbs for better opportunities amid the economic, health, education, and environmental disparities within the city. Data are a key factor in how these issues could be addressed.
Abrams observed that minority populations also have a history of distrust toward philanthropic organizations. A data analysis of 5 years of Heinz Endowment grants revealed that 77 percent of the grant funding was awarded to organizations led by White men, and a 2005 report on California foundations revealed that only 3 percent of funding was given to minority-led organizations.3 Abrams advocated for Black-led groups in Pittsburgh to receive more funding; the Pittsburgh Justice Funders are working to bring racial equity and justice to the city and to grantmaking more broadly. She explained that part of the problem is that data are often weaponized. When a White-led city like Pittsburgh is making decisions for marginalized communities without hearing their voices, data could be misinterpreted. As an example, Pittsburgh’s Inequality Across Gender and Race was written by a team made up predominantly of White people and captured all of the negative stereotypes about women of color. However, as Dara Mendez pointed out, the data told a different story: the people who were being studied were not part of the study.4 Thus, Abrams asserted that representation is needed both within institutions and through data, and that qualitative data could supplement quantitative data.
Abrams discussed the COVID-19 pandemic as another example of the consequences of racial disparities in communities as well as the value of creating communities of practice around data. She partnered with the Pittsburgh Black Worker Center5 and the UrbanKind Institute6 (both Black-led organizations) as well as the Western Pennsylvania Regional
___________________
3 Greenlining Institute, 2005, Fairness in Philanthropy, http://greenlining.org/wp-content/uploads/2013/02/FairnessinPhilanthropyPartIFoundationGivingtoMinorityledNonprofits.pdf.
4 For more information about Professor Mendez’s work, see https://publichealth.pitt.edu/home/directory/dara-d-mendez, accessed August 28, 2022.
5 For more information about the Pittsburgh Black Worker Center, see https://www.pghblackworkercenter.org/, accessed August 28, 2022.
6 For more information about the UrbanKind Institute, see https://urbankind.org/, accessed August 28, 2022.
Data Center7 and Carnegie Mellon’s CREATE Lab8 to create a Black-led group to investigate COVID-19 data. This data group began a series of virtual town halls to provide trustworthy information to communities about the pandemic as well as about Black experiences in Pittsburgh. This group has expanded to become the Black Equity Coalition,9 focusing on data collection and dissemination related to education, community health, community engagement, policy, data, and Black businesses. Abrams emphasized that building trust among communities and in data is critical to developing a comprehensive plan for Pittsburgh; the overarching questions that remain are who owns the narrative and who owns the data.
5.4 BUILDING CONFIDENCE IN DATA WITH STAKEHOLDERS IN TRANSPORTATION ASSET MANAGEMENT
Michael Cremin, Minnesota Department of Transportation’s (MnDOT’s) Asset Management Program Office, explained that physical transportation infrastructure is very complex, and in the state of Minnesota, this infrastructure connects to the social infrastructure. Therefore, asset management is only one component of multimodal programming and planning for public health and sustainability.
Cremin noted that MnDOT Asset Management experimented with many techniques to build confidence in its data and assets. It has four active data-driven plans of particular relevance: (1) Data Quality Management Plan, (2) GIS Strategic Plan, (3) Asset Management Strategic Implementation Plan, and (4) Transportation Asset Management Plan.
The enterprise-wide Data Quality Management Plan reveals gaps related to signal systems, which are one of MnDOT’s most critical assets; although progress has been made and confidence has been built around some of the data, Cremin said that more time is needed both working in teams and working on data processes to enhance data quality. The GIS Strategic Plan focuses on both short-term (data governance and services, training, and communication) and long-term (sustainability, data management, recruitment and retention) objectives for spatial and non-spatial data and the systems and people responsible for setting up these data, as well as practical resources for implementation. This plan includes an operational workgroup for asset management with monthly meetings to “connect the dots” of data activities.
___________________
7 For more information about the Western Pennsylvania Regional Data Center, see http://www.wprdc.org/, accessed August 28, 2022.
8 For more information about the CREATE Lab, see https://www.cmucreatelab.org/home, accessed August 28, 2022.
9 For more information about the Black Equity Coalition, see https://www.blackequitypgh.org/, accessed August 28, 2022.
The 2021 Asset Management Strategic Implementation Plan focuses on the five pillars of asset management—mobility, safety, health, environment, and economy—and how they are implemented via processes, people, data, and systems. As part of this plan, a special team of agency leadership and maintenance and engineering staff was convened to identify what data elements are required for each asset class (see Figure 5-1). The 2022 Transportation Asset Management Plan, a federally required plan for state DOTs, revolves around building effective data. Cremin explained that these data-driven plans are critical and are integrated with other capital planning efforts, including community outreach.
5.5 DISCUSSION
Holm asked how to ensure that the right people are “in the room” for decision making. Jain cautioned that because some tools have been used in ways that negatively affect communities, people might distrust them. Thus, he explained that both people with the historical context of a city as well as those with data analytics skills are key to enabling decision making. Abrams described her experience building a team of “knowledge experts” around data, including those with philanthropic experience, elected officials, and trusted community members. She stressed that building trust, especially in marginalized communities, takes time; the groundwork has to occur at the community level, and the community should be provided with the right tools. Doing this work is far more difficult, she continued, when “strangers” are in the room.
SOURCES: This graphic is part of the presentation to the workshop given by M. Cremin, MnDOT Asset Management Program Office, July 20, 2022, adapted from a figure courtesy of Applied Pavement Technology, Inc.
Holm posed a question about policies to identify underrepresented data during the decision-making process to avoid inequitable implementation. Abrams highlighted the value of increased engagement from city leaders; once a city makes its methods for data collection public, progress could be made toward policy making. She proposed scouring existing data used by nongovernmental agencies and nonprofits and partnering with university data centers to help make these policies. She emphasized that because policies are rarely tested on the people who will be impacted by them, it is also important to understand cultural differences and avoid making assumptions about experiences of people in the community—another reason to ensure that the right people are in the room for decision making. Holm said that Los Angeles set aside $10 million to invest in communities that have been historically affected by racism. Those communities are asked how the budget should be spent, and nonprofits and community organizations (instead of the government) help select the projects.
Holm wondered how to deal with data that do not exist or do not align with stakeholder goals. Cremin noted that data do not exist to support planning 10 years into the future and forecasting asset conditions, and assumptions are made based on network percentages. Although this is feasible, it is not ideal when the funding is not connected directly to the asset. He also noted that when data do not align with the objective, challenges arise, and a step back could be taken to understand if perhaps a published plan was incorrect and the goal could evolve.
Holm inquired about practical advice for managing and integrating data. Cremin replied that although the focus of enterprise data in MnDOT is state assets, the network includes counties and local areas, and knowing where those assets intersect is valuable. He noted that MnDOT is in the early stages of integrating those data and is currently showing simplified representations of local data. MnDOT has eight subdistricts across the state, and knowledge is being converted to data where it makes sense to do so. Holm mentioned that Los Angeles has encouraged its departments to publish their data, and community meetings are held around specific data sets. The city also created new standards about the minimal amount of data that departments have to share to keep infrastructure safe. Jain emphasized that even if only some data are being shared, that is better than not sharing any data. He added that transparency is key to addressing existing fears that public data could be misused.
Holm posed a question about how cities balance their needs for data with constituents’ rights to privacy. Jain mentioned that technologies such as differential privacy have emerged to mitigate some of the privacy risks, but trade-offs are unavoidable. For example, targeted intervention is more difficult when data are anonymized, but privacy is still paramount. He emphasized that no one-size-fits-all approach exists. Abrams described a
privacy debate that emerged in Pittsburgh with the identification of African Americans impacted by COVID-19. Although the city wanted people to be aware of high-transmission areas, care had to be taken to protect individuals because biases and prejudices often accompany particular illnesses. Holm explained that cities have a difficult task in evaluating trade-offs, especially in communities that already distrust the government. Holm highlighted the Cities Coalition for Digital Rights,10 which has developed a digital code of ethics that Los Angeles has adopted to empower people to identify and discuss bias. Cremin remarked that people view certain infrastructure elements as having more sensitive data than others and might be hesitant to crowdsource data for fear that they would be misrepresented, especially in a digital world. Although no one-size-fits-all software exists to address these concerns, he suggested the use of the geographic information system (GIS) as a starting point.
Holm inquired about additional approaches both for effective community engagement and for building confidence in institutions. Cremin indicated that a challenge lies ahead to improve community engagement, especially because it varies by sector. Abrams asserted that decisions should never be made in a silo; input from affected communities is essential. Democratization of data allows for this level of increased, active community engagement. Holm explained that no matter the issue, cities should develop a clear strategy for community engagement that includes the appropriate staffing capacity. Jain urged continued discussion on this issue and added that this is an exciting time for people interested in both the technical and social aspects of data.
5.6 JUSTICE40’S PARTICIPATORY TECH AND DATA
Shelby Switzer, Georgetown University and U.S. Digital Service (USDS), explained that Justice40 is a government-wide effort to ensure that federal agencies work with states and local communities to fulfill President Biden’s promise to deliver at least 40 percent of the overall benefits from federal investments in climate and clean energy to disadvantaged communities. The program focuses on climate change, clean energy and energy efficiency, clean transit, affordable and sustainable housing, remediation of legacy pollution, clean water infrastructure, and training and workforce development. The Executive Office of the President, the White House Environmental Justice Advisory Council (WHEJAC), the InterAgency Council, the National Climate Advisor, the Council on Environmental Quality (CEQ), the Office of Management and Budget, USDS,
___________________
10 For information about the Cities Coalition for Digital Rights, see https://citiesfordigitalrights.org/, accessed August 28, 2022.
the Justice40 Open Source Community, and several federal agencies are partners in this effort.
Switzer indicated that USDS built the Climate and Economic Justice Screening Tool (CEJST),11 which is an open-source geospatial tool that will, along with the open-source community, help policy makers and federal program offices make more informed decisions about where to direct the benefits of their programs. CEJST includes an interactive map with indicators, displays the communities (identified by CEQ’s definition) on the map, and makes the list of communities and other data sets available for download or use by federal employees and the public. USDS also launched an open data platform to make the source data sets, tooling, and related discussion publicly available; brought together working groups; and enriched and “ground-truthed” the data (i.e., validated the data to ensure that they represent lived experiences) with the participation of the most affected communities. USDS’s goals for the Justice40 initiative include ensuring that the government benefits and investments go to the most historically overburdened and underserved communities; empowering communities to participate in decisions that could affect them; improving the government’s ability to measure environmental, economic, and climate justice metrics; and building community trust.
According to Switzer, USDS’s guiding principles for CEJST are as follows: (1) transparency and trust are key to success and are gained through community involvement; (2) it is acceptable that the first version of a screening methodology will not be perfect; (3) the screening methodology is and should be clearly communicated as an iterative process that will continue to evaluate new data sources; (4) the tool is only as good as how well it explains the data to the community and federal users; and (5) the tool and the data/technology it uses should be open and replicable—that is, anyone can host and use the tool in their environment, run the algorithm, and get the same results. Key CEJST users include communities, public data users and technologists, and federal program officers; the architecture itself begins with community input.
Switzer described the Justice40 Open Source Community, a Github repository12 that is free and available for public observation and reuse under a Creative Commons 1.0 license. Instructions for contributing to the code and to the conversation are readily available, and architectural decisions are documented and reviewed before implementation. The Justice40 Open Source Community also has a Google group for discussion
___________________
11 For more information about CEJST, see https://screeningtool.geoplatform.gov/en, accessed August 28, 2022.
12 “GitHub is a code hosting platform for version control and collaboration.” For more information, see https://docs.github.com/en/get-started/quickstart/hello-world, accessed December 9, 2022.
and organization.13 Switzer asserted that both technologists and community members could build bridges among government, academia, nonprofits, the private sector, and environmental justice organizations to better develop CEJST and improve the development of technology and data tools to advance environmental justice causes. The Justice40 Open Source Community encourages and supports participation from all communities and individuals, no matter their technology experience. It values (1) encouraging transparency and participation, (2) using a modular and modern approach to software development, (3) deploying open-source software and an open-source process, (4) implementing with ease, (5) building capacity and making software and processes accessible to participants with diverse backgrounds and skill sets to foster community, (6) building open data sets where possible because data and data science are as important as software and process, (7) advancing transparency in algorithms and identifying places where biases could be introduced, and (8) prioritizing data sets that address community vulnerabilities for programs in Justice40.
Switzer also discussed Justice40’s process to collect participatory data: solicit input from WHEJAC on data needs; crowdsource data set ideas from the open-source community; identify how to assess the data for use; investigate data sets based on identified criteria; help CEQ experiment with methodologies using data sets; launch CEQ’s beta methodology on CEJST; solicit public input; and continue to iterate on and develop the tool and methodology with CEQ, WHEJAC, and the public. Challenges of working with participatory data include aligning participatory technology and data with participatory policy; systematizing participatory data and ensuring long-term success as well as integrating diverse routes for public participation; accurately representing lived experiences; engaging a wide spectrum of audiences and users, levels of trust, and technical skills; and building capacity for participation.
In response to a question from Holm about advice for building community trust, Switzer suggested listening carefully, creating avenues to foster communication, and being transparent about plans to take action after listening. It is vital to show people how the actions taken will connect specifically to their complaints. Holm wondered about advice for communities that wish to make their data open source, and Switzer emphasized that a culture change is often first required so that people understand the benefits. Many open-source communities are already well established and could help ease the transition for new communities.
___________________
13 To visit Justice40’s Open Source Community, see https://github.com/usds/justice40-tool and https://groups.google.com/u/0/g/justice40-open-source, accessed August 28, 2022.
