2

Data Equity and Communication Strategies

This June 2022 workshop was organized to assemble a diverse group of presenters to reflect the underlying data used to measure plurality in the person and housing composition of the United States. The Demographic and Housing Characteristics (DHC) data have important ramifications for measuring diversity, as noted in the opening remarks from the National Academies of Sciences, Engineering, and Medicine staff and the presentation by the U.S. Census Bureau director and the subsequent discussion.

OPENING REMARKS

The workshop opened with welcome remarks by Carlotta Arthur, executive director for the Division of Behavioral and Social Sciences and Education (DBASSE) at the National Academies. She noted that this was an important conversation for the National Academies to facilitate with stakeholders and the U.S. Census Bureau about using census data products for key applications while protecting privacy of respondents. After thanking the distinguished planning committee members for their countless volunteer hours to help the National Academies fulfill its mission of providing independent and objective advice to inform policy, Arthur emphasized the importance of community engagement, which is a priority for DBASSE, and also stated: “The National Academies and DBASSE are deeply committed to diversity, equity, and inclusion in all aspects of our work. It is critical to measure these data accurately to ensure that the voices and experiences of all people in the United States and territories can be counted.”

The then-director of the Committee on National Statistics (CNSTAT), Brian Harris-Kojetin, next welcomed participants to this second workshop on the 2020 Census DHC files for the U.S. Census Bureau. Harris-Kojetin noted that while much has changed since CNSTAT convened the first workshop on the first 2010-based DHC demonstration data after they were produced in fall 2019, what has not changed is that the Census Bureau is still working very hard to produce useful products from the decennial data. Harris-Kojetin stressed the importance of engagement and feedback from users to better understand the uses of these data and the implications of the disclosure avoidance procedures used by the Census Bureau.

Harris-Kojetin stated that the mission of CNSTAT is to provide advice to the federal government and the nation, grounded in the current best scientific knowledge and practice, that will lead to improved statistical methods and information on which to base public policy. “We’re celebrating 50 years of service to the federal statistical community this year, here at CNSTAT, and we very much appreciate the Census Bureau for sponsoring us and trusting us to host this workshop for them.” Harris-Kojetin specifically thanked Cynthia Davis Hollingsworth from the U.S. Census Bureau, who has been the lead collaborator with CNSTAT, for her many hours working to scope the workshop; several CNSTAT staff members; and the planning committee. He concluded by thanking the presenters, who took valuable time under very short notice to delve into these data and consider the implications for their uses.

The co-chairs of the planning committee were V. Joseph Hotz (Duke University) and Elizabeth Garner (Colorado Department of Local Affairs), who also serves as Colorado’s state demographer. In their opening remarks, Hotz and Garner reiterated Harris-Kojetin’s note that the planning process was fast paced, and many thanks were due to CNSTAT staff, as well as many others at the U.S. Census Bureau.

WORKSHOP GOALS

Co-chair Joe Hotz stated that the goals of the workshop were to provide useful feedback regarding the implications for the protection of privacy, which is an important and mandated objective for the Census Bureau, and to provide actionable input on the disclosure avoidance system for the 2020 DHC File, in order to improve data utility while protecting privacy. In addition to this overarching goal, Hotz stated that it was important to understand both the fitness-for-use of the DHC files and the consequences for equity of measuring the U.S. population accurately for the following:

• large and small geographies and various legal entities;
• demographic groups (gender, age, race and ethnicity);

• household characteristics;
• housing tenure characteristics;
• legally required uses of DHC data; and
• comparisons over time.

Hotz conveyed his hope that, over the coming months, this workshop will help the Census Bureau and users of DHC data to identify and address the DHC’s disclosure avoidance system and its consequences for fitness-for-use for alternative use cases. Users’ guiding questions include the following:

1. What will the disclosure avoidance system be tuned to?
2. What, if any, are the remaining shortfalls of the 2020 DHC data?
3. What are the margins of error for various use cases?

Hotz noted that the new disclosure avoidance system is a major undertaking, and the Census Bureau should recognize that not all users are comfortable with it. To help users, he posited that it would be helpful to catalog best practices for understanding strategies and methods for using the data efficiently, in order to address these questions and the varied set of uses.

Hotz concluded by commenting that one issue emerging from this change is concern about the potential loss of credibility by the Census Bureau from the 2020 experience. The potential loss of credibility or quality of the 2020 Census is the subject of a consensus study¹ that had not been completed at the time of writing this proceedings. Hotz attributed these concerns partly to the disclosure avoidance system, but the pandemic and the circumstances under which the 2020 Census was conducted were also pertinent. Regarding the disclosure avoidance system, he emphasized that it is important to make clear whether the changes, which were driven by privacy concerns, are meeting those objectives without eroding trust. He asked what lessons are to be learned looking forward to the 2030 Census and stated that the Census Bureau should perform that assessment collectively with users and the American public to determine what is relevant from this experience, and then implement the results of that assessment in the planning of the 2030 Census, exploring all the options developed in the scientific community and elsewhere for improving the balance between privacy protection and data utility.

Co-chair Elizabeth Garner expanded on the goals during opening remarks on the second day of the workshop by underscoring the importance of understanding how different users—such as rural and smaller population areas, different race and ethnicity groups, and different household

___________________

¹ https://www.nationalacademies.org/our-work/panel-to-evaluate-the-quality-of-the-2020-census

types—have determined fitness for use and the impacts of the disclosure avoidance system on the DHC File. She asked who the “winners and losers” are if disaggregated data cannot be released and urged viewing this question through an equity lens. Garner explained that short-term objectives for being user centric were focused on communication to understand how the disclosure avoidance system will be tuned and the reliability of different variables and characteristics by levels of geography. She advocated examining and keeping in mind best practices as the disclosure avoidance system is tuned and as the DHC File is released.

Garner stated that outside users think of the Census Bureau as a whole and do not separate who is involved in decennial versus population estimates data. She questioned how to supplement the public’s understanding of the Census Bureau and ensure it is still recognized as a national resource of data and information. She also asked, “Moving forward to the 2030 Census, what are some of the more significant lessons learned?”

KEY ISSUES FOR THE U.S. CENSUS BUREAU

Garner moderated a session with U.S. Census Bureau Director Robert Santos. He opened by saying he valued the National Academies and what it can provide through its volunteers as stakeholders in the effort to improve Census Bureau products. Santos noted that the DHC File provides a number of demographics, such as age, sex, race, ethnicity, and household characteristics, and that the DHC File uses a new disclosure avoidance methodology involving differential privacy. He stated, “It is fair to say it does not enjoy a consensus among our stakeholders.” Santos continued by saying that differential privacy is a methodology most would agree is more advanced than the data swapping and other methods previously used with decennial census data, and suggested that differential privacy is more protective against disclosure than other methods.

Santos explained that this approach addresses 21st-century confidentiality threats, which will only grow in sophistication over time. As a result, differential privacy ends up diminishing some of the granularity that data users have enjoyed and become accustomed to in the past. Because it is a complex and computationally demanding process, Santos stated that time and user input are required to find the right balance of geographic accuracy and demographic detail, while still preserving confidentiality. He encouraged presenters and attendees to engage the subject matter experts from the Census Bureau about these methods used to add noise to census tabulation cells.

Since many in attendance were seeing Santos in his role as director for the first time, he offered to talk generally about his goals for the Census Bureau. He stated that the Census Bureau is committed to producing data

that reflect an accurate portrait of the nation. He believes new ways of collecting data can be found that are more accurate, more timely, more relevant, and—most of all—have utility in various fitness-for-use arenas.

Santos described himself as a leader who champions diversity, equity, and inclusion (DEI) and has lived these values over a 40-year career, often as the sole Latino in a room full of researchers even before the approach and concepts were called DEI. By embracing DEI principles, an organization can identify innovations and advance excellence. An important part of DEI efforts at the Census Bureau includes seeking out and listening to diverse voices. He stated that stakeholder feedback is critical to understanding the needs of data users, and the feedback at the workshop would be essential for protecting data from the very real threat of disclosure while addressing the needs of stakeholders.

Santos underscored the importance of transparency and its role in communicating data quality. No statistical product is ever perfect, and each is subject to various degrees of uncertainty and error. He explained that the Census Bureau applies extensive quality checks and reviews to reduce those errors so that results are fit for their intended uses. Thus far, the Census Bureau has released several 2020 Census quality assessments, some of which are still ongoing. Knowing the strengths and limitations of any data set actually increases the value of the data by enabling users to be careful with their analyses and better interpret results.

He noted that the DHC File is in the queue as the next 2020 Census data product, which will be released in May 2023. Santos requested feedback from users so that the Census Bureau can better calibrate and test the disclosure avoidance settings for these products. As a result, another round of demonstration products was planned for release in August 2022 based on the 2010 Census. He noted that this method of feedback and recalibration led to many improvements in the disclosure avoidance system algorithm aimed at increasing accuracy for the redistricting data product. Engagement with users helps identify needs that would otherwise be missed if not provided in these types of forums. He expressed his gratitude for data users and stakeholder communities investing their time and effort to help the Census Bureau build data products that are useful to everyone in the nation.

DISCUSSION

Planning committee member Jan Vink (Cornell University) commented that he is having difficulty with the new disclosure avoidance system because it creates a synthetic database of people, and it is difficult to talk about results that are “made up.” Vink explained that people need help talking about these results because they are no longer a precise count and now include noise infusion.

William O’Hare (consultant to the Count All Kids campaign) asked what lessons the Census Bureau has learned about differential privacy after rolling it out for 3–4 years. Santos stated that the experts would be speaking to this issue during the workshop, although he offered to share his overarching perceptions. He compared the Census Bureau to many other statistical federal agencies and research organizations as “continuous learners.” As a result, repeated use and implementation over various Census products has helped the Census Bureau learn that it needs user feedback to find the right balance between geographic granularity and data accuracy while maintaining disclosure avoidance.

Santos noted that if he had a magic wand, he would invent something that reflected the quantitative notion of data utility, so that one could look at the actual trade-offs mathematically and graph them to find the sweet spot. He explained that this quantitative notion of data utility is not possible because there are almost infinite uses of census data in policy work and other applications; therefore, the Census Bureau has to rely on the diversity of the stakeholder community to provide this input. As the Census Bureau conducts outreach and asks for feedback, one of the first things he requests is, “Please provide me with your use cases,” because the more they can obtain, view, and ingest, the better they can calibrate that delicate balance to maximize utility. A webinar attendee, who did not disclose name or affiliation, asked whether it was possible for the Census Bureau to revert back to the prior disclosure avoidance system. While Santos acknowledged this sentiment, he affirmed that the Census Bureau has made a determination to meet its obligation to protect census data with today’s technology, so reverting to the prior system is not possible.

Planning committee member Vickie Mays (University of California, Los Angeles) asked Santos about the loss of credibility issue and what types of products could help not only data users but also communities that participate in the filling out of the data in preparation for 2030. Could the Census Information Centers (CICs) play a role? Santos responded that the credibility issue is not a dichotomy and can be thought of as a continuum. The best way to approach this continuum of credibility is to have frank conversations with users and to listen, absorb, and understand their input, and to then create products that reflect and address their concerns. Santos mentioned disaggregated race and ethnicity as one such area. He further underscored the importance of scientific integrity and the role of transparency in that process, which is one of his high-priority areas. He stated that, although many independent experts are conducting evaluations, he believes the Census Bureau will be found to have performed well under an incredible set of challenges.

Planning committee member Mary Craigle (Montana Department of Commerce) noted that she was the chair of the State Data Centers (SDCs)

and commended the Census Bureau’s work, particularly the disclosure avoidance system team, for its outreach and improvement of the products. She asked whether Director Santos envisioned the SDCs and CICs helping shape the outreach from the start rather than being on the back end of trying to “rally the troops” to support the Census Bureau. Santos described the Census Bureau’s outreach as an evergreen effort because so many data collection efforts are always ongoing, such as approximately 130 surveys of businesses and people. He stated that partnerships and stakeholder feedback should never come solely right before a decennial census in anticipation of a big push.

Santos observed that he thought the CICs and SDCs were critical in the effort to turn the credibility situation around, which is why use cases are so important for leveraging help with their constituent communities. He suggested that they could be working together collectively to begin a dialogue and flow of information and resources, including discussion of new data tools that speak to the specific needs of local communities, which can then be scaled to other communities. Santos stated that the Census Bureau needs to be more deliberate in how it plans for expansion, and it needs to bring the SDCs and CICs together. He noted that, while the Census Bureau is not there yet, this expansion is part of its vision.

Garner noted that the Federal-State Cooperative for Population Estimates (FSCPE) has been concerned about providing use cases because they could be contradictory and create different priorities for the Census Bureau, such as a demand or need for small-area estimates versus better data at the county level for all age groups. She asked how to balance the different characteristics that may be important because it will create winners and losers. Santos acknowledged that because it is impossible to satisfy everyone, it is an issue of prioritization, which cannot be done without knowing how the data are used. Ultimately the Census Bureau will make its best judgment with regard to how it arrives at the delicate balance. Garner suggested that looking out at 2030, it would be an interesting experiment to get some of the data producers and statisticians at the Census Bureau who create the algorithms to live a day in the life of a data user so that they can understand the complexities faced by the SDCs, FSCPE, and different nonprofits.

This page intentionally left blank.