6
Wrap-Up Discussion
The final session of the workshop was devoted to a panel discussion with members of the planning committee: Michael Davern, Jennifer Madans, Sunita Sah, and Vetta Sanders Thompson, along with John Phillips from the National Institute on Aging (NIA). As moderator of the discussion, Brian Harris-Kojetin began by asking the panelists what they saw as the take-home points from the sessions they chaired or from other sessions in the workshop. He also asked them what areas they thought NIA should prioritize for further research that would help advance some of the concerns and issues raised in the workshop.
TAKE-HOME POINTS AND RESEARCH PRIORITIES
Davern returned to his initial remarks that surveys are in crisis due to challenges related to nonresponse and nonresponse bias, as well as the global pandemic. He pointed out researchers are being forced to adapt their methodology for aging studies because that population is at high risk. Furthermore, there are risks with some of the methods currently used to collect important data, such as physical measures or biomeasures. Moving forward, he urged asking what is missing from the stories about the current crisis in survey response. Every story leaves out important factors and perspectives that are not represented, he said. He asked workshop participants to consider what is missing or not being focused on that could be essential to resolve this crisis with surveys.
Maintaining Contact Over Time
Many of the longitudinal studies discussed at the workshop represented initial investments from NIA or other federal agencies, Davern continued, and 20 or 30 years later, the studies are still contacting the same people. Yet over the life of a study, contact or engagement with respondents is not maintained because of insufficient funding. He urged thinking about how to centralize some kind of repository to maintain contact with respondents in important data collections even if there is not funding for data collection some years. Failing to maintain contact and engagement is a lost opportunity for the future, he observed, and keeping electronic contact through email, social media, and other available tools is a valuable idea that came up during the workshop.
For research priorities, Davern suggested an engagement strategy for trying to stay in touch with and maintain contact information over time with participants in large surveys, even when there might not be funding for a regular data collection. He stressed the importance of being able to recontact them effectively later if there is funding or interest in follow-on studies with that group of people.
Geographic Differences
Thompson picked up on Davern’s comment on variables and factors that no one had discussed during the workshop. She noted that no one talked about geographic differences, and that she suspects that geographic differences are likely, particularly for some minority populations simply because of their concentration and relationship in the context given their numbers and political positioning, their reaction to authority and government, and other factors. She also noted that education in the context of aging relates to the cognitive capacity to consent and is important to consider.
Overall, Thompson said she heard consensus on what to do to successfully recruit and get people to participate, especially in person. However, shifting to the web where those social dynamics that drive people to agree are absent poses more problems, she observed. She was also surprised about the consistency in the way presenters talked about their ability to identify the resources and the techniques necessary to convert refusals and to build up a response rate even on the web.
Ethical Considerations in Consent
Thompson added that what struck her is the need to shift focus, because the ethical considerations and the complexities discussed lead her
to wonder about the need to think about stakeholders differently. She referred to a question posed by Christine Grady: What gets people to feel like they will be making an important contribution that is meaningful to society and thus something that they really want to do as opposed to us just trying to figure out how to get people to say yes? Thompson added that as long as the goal is simply to get people to say yes, investigators will be less personal and more reliant on technologies that remove an in-person social dynamic, and the more surveys are going to suffer. People may avoid doing surveys because they are so prevalent. Although they do make a distinction between surveys that are very important to health and well-being compared to many other kinds, they need to hear the story in a way that is appealing to them, she said. She urged putting more resources into understanding the “whys” of various stakeholder groups, then communicating more effectively whether via email, text, or telephone. Perhaps less communication will suffice, she said, and people might be more motivated over a sustained time period without additional incentives.
Understanding Why People Consent
From her perspective of an organizational psychologist, Sah said she picked up on Thompson’s comments on understanding the “why” of participation. She noted that consent varies between people as well as within the same person over time. At one particular point of time, they may say yes; at another point of time, they may say no or vice versa. Understanding why is hugely important to figure out, she emphasized.
Sah noted that collecting data online will become more important and pointed to the tension about giving away personal data too freely online. People will agree to many things online and give personal data away, but they also worry about how much data they are giving away. She asked, “What is it that people want to tell us and how could that help us in collecting the data that is also essential?” She also described how some people say they would like to help and will participate, but then do not. Sah asked what to do to close the gap between people’s statements and actual follow-through and to understand why the gap is happening at that particular stage.
In terms of research priorities, Sah suggested trying to understand the psychological processes of why people consent, in what situations, what factors influence their consent, and what factors influence their retention is important. Trust is an important factor, she said, and highlighted the discussion on the role of online influencers, friends, and family in building trust. Every brand has a different level of trust and will bring in different people and different voices, she observed. She considers this is an important area of research because personalized consent is going to be difficult, and the
way to reach more people will be through brands and influencers. Areas for research include how to build a brand that is trusted to conduct these types of surveys and with whom to partner, Sah said.
Improving Initial Response Rates
Madans commented she was probably not as optimistic as other presenters and committee members. She identified two streams in the presentations: one approach is to improve response overall and keep people in surveys, and the second is to reduce bias after the fact. She commented most researchers are doing all the obvious things to get the highest response rates, but there is no silver bullet. People vary in their preferences. Targeting approaches to each respondent to appeal to them and their decision to consent could be done, but she recognized that it is not feasible on a large scale.
To Madans, getting people in the beginning of longitudinal surveys is critical. Response rates are going down to a point where there are huge risks of nonresponse bias. Madans asked if something structural could be done rather than trying to target individual approaches for specific groups of respondents. She noted that there is no brand for the federal statistical system and asked how a potential respondent could distinguish the good surveys from the bad ones. She urged reeducating the public writ large, rather than individual survey respondents, about the use and benefits of surveys for the public good. Madans said that approach would not be easy or extremely successful, but neither is trying to tailor information to each individual respondent.
Reducing Bias
Madans then turned to the approach of addressing bias reduction after the fact, noting that it is still necessary to get a certain level of response in order to conduct bias reduction. She asked what can be done to ameliorate the fact that one no longer has a representative sample, and how to evaluate if big groups of people have been missed. She referred to the design discussed by Andy Peytchev. It is difficult to bring more people into the sample who have different experiences or ideas that are not related to demographics, she commented.
Madans said she sees opportunities with using linked data more in survey design as opposed to only getting additional information. This would present a consent issue, she acknowledged, and also is affected by the way the data system in the United States works with many different data owners. She urged doing everything possible to reduce risk for disclosure and make people more comfortable with data linkage, and there is a lot of interest in doing that now. Madans concluded by noting that scientific surveys are
costly and resources are limited. She asked how to prioritize what is most needed given limited resources so as to maximize the overall benefit.
In terms of research priorities, Madans said that research on bias reduction after data collection is important, but that it should be included in the survey design so that information is obtained that can be used to reduce bias after collection. She said that if there are auxiliary data that might be useful for bias reduction, one needs to make sure one collects the data in the survey so that the auxiliary data can be used effectively.
DISCUSSION OF ADDITIONAL PRIORITIES FOR FUTURE RESEARCH
Coordinating Research Across Studies
Madans also noted that many surveys include experiments, but there is pressure not to use sample persons in a way that would affect the substantive analyses of the survey. As a result the experiments are limited and do not permit looking at important interaction effects. She suggested NIA consider funding innovation panels to invest in research that is best done in conjunction with an ongoing survey without being concerned about losing cases. It would be especially useful to do coordinated research across multiple surveys that are looking at the same experimental conditions so that there is more power to look at experimental effects and interactions of different surveys and conditions, which would help identify innovations for improving the methodology for everybody, Madans suggested.
Increasing Research Literacy
Thompson singled out consent for biological samples as a pain point and pointed to discussion about what people understand even when they consent. She added she has been thinking about this in the context of vaccine hesitancy. What people know about biology, science, health, and research affects their responses. She suggested working with communication and education systems to better understand people’s needs to assist agencies working to promote health and well-being. Thompson noted that her organization has done work around increasing research literacy and has seen how enthusiastic people become about research, which makes a difference in their participation.
Linking to Administrative Records
Davern mentioned the missingness that happens when surveys and administrative records are linked. He said it would be good to think about
how to create inferential tools like weights or other tools to generalize to a population and understand the inferential population of linked cases in order to conduct important health and aging research on those linked datasets. He said understanding the inferences and the limitations of the inferences using joint data products is a good use of resources.
Davern expressed interest in Timothy Smeeding’s presentation, but noted its focus on government data. In the commercial space, Davern said, the rules of data governance are different. He said that trying to work with commercial health care providers to get health care, medical record, laboratory, or payment information can be difficult but also worth the effort. Tools mentioned by Lisa Mirel around the use of Privacy Protected Record Linkage are being used by commercial entities to link data, he said, and commented it is a fascinating area to explore. He also suggested engaging with commercial enterprises to ensure continuity because they can terminate access to their data at any given time. They could feel threatened by the potential release of proprietary information or research studies that could help their competitors by divulging something about their patient population, he commented.
Smeeding agreed that commercial data are difficult because companies want to keep some things secret, and they can make money off their data. He endorsed the idea of using surveys to link to government records, adding that maintaining contact with people who are otherwise difficult to locate would be an amazing possibility. Maintaining contact is important, and he recalled the contribution of Tecla Loup, a former staff member of the Panel Study of Income, who “knew everyone and sent them a Christmas card every year.” He emphasized that using administrative and survey data together strengthens and improves surveys. Surveys provide important information that administrative records cannot and serve as the “glue between the bricks from the administrative data,” he commented.
Issues of Ethical Consent and Privacy Over Time
Davern raised another issue about what happens ethically in supporting the research with respect to consent and records after a participant dies. He noted that the rules for human subjects no longer apply, and there is a lot of variability in how this is handled. The Census statute is very clear about the 72-year rule for protecting confidentiality, but he asked about long-term studies. He said he foresees many possibilities in the next 10, 20, or 30 years for how to reuse some of those data to provide information about the life course of the people and those cohorts.
Smeeding noted that linked Census data have been used to look at African American mobility over time, and noted other opportunities to use those data to address important questions. Thompson said that she spent a
lot of time talking to people in the community about the level of protection of their privacy and the fact that they have to give consent for some of these linkages because they have real fears, particularly for government data. She advised being very thoughtful in proceeding with these linkages and how the linkages are explained to the public; otherwise, people will refuse to participate from the beginning. She underscored the need to respect people’s right to make the decisions themselves, especially those whose rights have been taken in the past. Smeeding agreed with the importance to attend to these issues and clarified that the paper on mobility he mentioned uses data from 1850 to 1940. Phillips commented while he is enthusiastic about linking to Census data, it will not be easy. He anticipates consent will be needed not just for the federal agencies to link the data, but also that Institutional Review Boards (IRBs) will have requirements for the reasons that Thompson raised.
Madans agreed with Thompson, and noted that the 2020 decennial Census devoted considerable time and money on messaging how the data are completely protected and confidential. She observed that very few people know the protection ends after 72 years, and that many other federal laws do not have end dates. She urged care and balance in moving science forward with people’s ideas and views of their rights regarding their information. To maintain trust, scientists should not be trying to convince people to do something they do not want to do, but instead building trust in ethical ways. Echoing Thompson and Sah, she stressed figuring out what is really important to people, finding a common thread to obtain participation and trust from a good majority of people, and then taking care to maintain that trust.
Sah commented that valid consent is key to building trust and retaining participants. She noted that insinuation anxiety is probably more relevant to vulnerable populations, and the social pressure to get consent needs to be reduced. For ongoing consent, she advised making participation more attractive and decreasing the cost of participation by making it more convenient, less burdensome, and by providing incentives or clear benefits so respondents see their contributions to science and society.
NIA PERSPECTIVES ON THE WORKSHOP
Harris-Kojetin turned to Phillips for his observations and questions for the panel. Phillips commented on the different ways the various longitudinal aging studies are attempting to deal with the same issues. He noted that these findings and tests are often not published or widely circulated, and that sharing this information is a priority to help all the surveys. Pointing to the discussion on the limitations on what some studies may do as a result of differences between IRBs, he said he was not sure how to address this other than greater sharing of protocols and practices across studies.
Referring to the several presentations about administrative data, including those by Andy Peytchev and Joe Sakshaug, Phillips questioned what the administrative data backbone for aging studies should be and what minimum data are necessary to use the statistical techniques presented to deal with nonresponse bias. He also identified a theme across several presentations that a combination of awareness and engagement across time seems to keep people engaged in studies. He added that when people are more aware of or had experience in health studies, they are less likely to be worried about consenting for a new one. He suggested building on this awareness, and that making the scientific and research process better understood may inherently improve people’s willingness to participate. He asked the group to consider how to make people more aware of science findings and the impact on science from these studies that might help improve participation.
Phillips said he is excited about the possibilities of self-administered biospecimen collections to get more participation in an inexpensive way. He expressed interest in hearing results on the use of the BioBox, and he hypothesized that some participants may be more willing to do some of the tasks if they have someone else present to help or participate.
Phillips closed by expressing appreciation to the presenters, the committee organizing the workshop, and all the participants. Harris-Kojetin echoed Phillips’s comments and also thanked the National Institute of Aging for sponsoring the workshop.
