8

Workshop 2, Session 2: Opportunities for AHRQ, ASPE, PCORI Collaborations to Improve Sustainability of Their Efforts

MAINTAINING PARTNERSHIPS FOR SUSTAINABLE DATA COLLECTION

Abby Collier, director of the National Center for Fatality Review and Prevention, explained that her organization, a program of the Michigan Public Health Institute, supports all aspects of child fatality review, as well as of fetal

and infant mortality review.¹ These reviews are community-based processes that seek to understand the circumstances of a child fatality. This work supports state and local programs in terms of training, technical assistance, and written guidance. They also maintain a national data system. The center is tasked with building continuity between and within these programs across the United States. It also provides a connected data network for more than 1,500 U.S. fatality review teams and provides linkages to national and state resources to help teams translate their findings into action.

One unique role the center plays is collecting real-time, community-based data by reviewing individual fatalities. When the team gathers to review individual cases, each team member shares what they know about the child, family, and community that is relevant to the death. After completing the review process, the team enters the data into their national database. Both this data as well as the content of their qualitative discussions are used to write recommendations for prevention at the local, state, and national levels. Collier noted that there is a great deal of variability in child death review. Some states have a comprehensive process. However, other states have an inconsistent approach in which not all communities conduct a comprehensive review. There are approximately 2,700 users across the United States, making data consistency and quality a challenge. States have varying resources for collecting and entering data and their child death review committees have varying access to records. Collier explained that currently 47 states use the center for child death reviews, and 18 of those use it for child death reviews and fetal and infant mortality reviews. She explained that child death review creates an opportunity to look at these sentinel events though a systems-level lens, which can be beneficial to the local community. She noted that child death reviews are not intended to place blame, but rather to look for opportunities to improve how systems work.

The success of the data system, said Collier, depends on relationships. The center prioritizes functionality for end users when making decisions related to the case reporting system. They also work to promote flexibility and ease of access for use; states, for example, are able to add individual questions to their inputs. Center staff assist users in reducing data entry burden by collaborating to prioritize the data they collect and enter. Center staff also review data quality and provide feedback to the users. Finally, the center created a data dashboard to empower state-level review teams that do not have extensive epidemiology skills or resources. The center also uses feedback from state teams to determine the focus of their national reports.

She noted that problem solving and technical support has helped the center build trust with the state-level child death review teams. Those efforts

___________________

¹ Details available at https://ncfrp.org (accessed September 14, 2022).

have also allowed the center to learn strategies that can be applied in other states facing similar challenges.

STATE-LEVEL DATA COLLABORATIONS

Lynn Blewett, professor in the School of Public Health and director of the State Health Access Data Assistance Center (SHADAC) at the University of Minnesota, began her presentation by explaining that from a state-level health policy perspective, the promotion of health equity requires a different approach than traditional clinical effectiveness research. This is because states have different objectives, different research questions, and different data resources. The goal of her presentation, then, was to encourage the Agency for Healthcare Research and Quality (AHRQ) to think strategically about how to influence and engage with state policy makers.

Blewett said that in many states, health policy focuses on equity, access, affordability of care and how to better engage priority populations to improve population health. State health policy also strives to incentivize health plans to facilitate high-quality care to people with low incomes via Medicaid, as well as identify the factors that impede access to affordable, equitable care. Policy can be developed based on the policy levers available. These include

• Medicaid eligibility and payment policy,
• managed care organization contract requirements,
• regulating the private commercial health insurance market,
• financing the state employee health plans,
• financing, and
• state cost growth benchmarking and other cost-control strategies.

She encouraged AHRQ to take its focus on equity seriously and to think strategically about how to use its resources to address equity challenges. Targeting equity requires measuring and understanding who these people are and including them in state data resources. She encouraged AHRQ, the Office of the Assistant Secretary for Planning and Evaluation (ASPE), and the Patient-Centered Outcomes Research Institute (PCORI) to focus their investments on supporting projects on these populations, include an equity lens, and consider data development and infrastructure for state Medicaid data and public programs.

Blewett emphasized the need for reliable, self-reported data on race and ethnicity. Self-reported data on race and ethnicity can be a powerful tool for Medicaid programs to measure social determinants of health and address health equity. However, research on how to collect such data is nascent, and collecting that data is challenging for providers and insurers. ASPE has completed some work on imputation of race and ethnicity, but states are reluctant to use

such methods because of concerns about transparency and difficulty engaging communities about the methodology (Branham et al., 2022). She suggested that Patient-Centered Outcomes Research Trust Fund (PCORTF) investments could support further development of measures to fill gaps in existing Medicaid claims data, electronic health record (EHR) data, and other claims-based data resources. SHADAC is working with state Medicaid data analysts on a project investigating how best to encourage Medicaid beneficiaries to share what are currently optional data on race, ethnicity, sexual orientation, gender identity, and disability status. They hope to translate those findings into best practices and policy change. Key components of engaging with Medicaid beneficiaries in that project included building trust and explaining what purpose the data will serve in terms of identifying and addressing health disparities. Blewett emphasized the need for additional funding for research on best practices and translating research into policy. She also would like to see the U.S. Office of Management and Budget (OMB) expand its efforts on the development of standards for race and ethnicity data, provide guidance to the states on flexibility, and provide incentives for state engagement in efforts to collect more granular and disaggregated data. She said that in 2019 the Centers for Medicare and Medicaid Services (CMS) released the Transformed Medicaid Statistical Information System (T-MSIS), which reports CMS-collected state Medicaid data. While quality is improving with these data there are still areas of concern about completeness in some critical areas. For example, 30 states did not submit acceptable data for inpatient managed care encounters, and 22 states submitted race and ethnicity data whose quality was of high concern or unusable. Additionally, access to the system is expensive. Blewett said there is a continued need to support state data capacity and provide incentives to collect better quality data. She noted that data would provide important insights on social determinants of health not typically found in EHRs or medical claims data.

Blewett highlighted another project, the Medicaid Outcomes Distributed Research Network (MODRN), which involves a group of states working together on a distributed research protocol (Adams et al., 2019; Zivin et al., 2022). Participating states work together on programming, conduct their own analysis on their data, and then submit tables with deidentified data to investigators at the University of Pittsburgh, who conduct further analysis. This approach avoids the need for data privacy and data use agreements because the states do the necessary programming with their own data. The MODRN team has used these data in projects on opioid use disorder that led to recommendations for Medicaid reforms (Donohue et al., 2022; Jarlenski et al., 2021; Kennedy and Sheets, 2021; Medicaid Outcomes Distributed Research Network, 2021).

Blewett described the Massachusetts Medicaid program which adjusts payments to health plans based on barriers to economic, food, and housing

security using a hybrid approach that combines administrative data and American Community Survey data. This social risk-adjustment method uses an index developed with variables from the survey. It includes enrollees’ addresses to develop census block indicators, a neighborhood stress score, and a measure of housing instability based on International Classification of Diseases (ICD) social determinants of health Z codes for housing insecurity. An early evaluation of this program found that including social determinants and related variables with risk scores strengthens the predictive power of risk adjustment and yields more accurate payments to managed care organizations (Jones and Muller, 2018). She said these results suggest that states may be able to encourage the use of Z codes by requiring managed care organizations to enter into value-based care arrangements with providers and by clarifying the rules governing providers’ collection and use of social needs data.

Blewett noted that it would be beneficial for AHRQ to support research on the role of community context, poverty, structural racism, and the social determinants of health in health outcomes. She noted there is also limited understanding of the complex network of safety-net providers that are often unrepresented in clinical trials or comparative effectiveness research. These providers serve patients from groups that have historically been made vulnerable and other priority populations and provide unique access to needed care. Another opportunity for research is investigating the role of Medicaid strategies in increasing access to quality care for its enrolled populations.

She concluded her presentation with a list of possible priorities for AHRQ research funding, which included the following:

• Patient-centered outcomes research (PCOR) studies to examine the intersection of the social determinants of health and clinical health indicators to gain greater understanding of their impact on disease burden.
• Projects that develop partnerships between state or local data organizations and local researchers, such as the State University Partnership Learning Network and MODRN, to answer relevant policy questions and build data infrastructure.
• Projects that develop regional social needs indexes.
• Promote collaboration between state policy organizations that engage with community stakeholders and policy makers to set research priorities and find opportunities for evaluation research to inform policy making.
• Research that includes the development of measures and mechanisms to capture valid and reliable self-reported data of race, ethnicity, sexual orientation, gender identity, and disability status.

AHRQ–ASPE–PCORI COLLABORATIONS TO IMPROVE EFFORT SUSTAINABILITY

Robert Phillips Jr., executive director of the Center for Professionalism and Value in Health Care at the American Board of Family Medicine, discussed examples of future opportunities for collaboration among AHRQ, ASPE, and PCORI with an emphasis on sustainable projects. The first is a collaboration among the American Board of Family Medicine, Stanford University, and the U.S. Census Bureau to develop a “gold standard” small area deprivation index, analogous to the neighborhood stress score that Massachusetts Medicaid uses as part of its risk-adjustment process. These indexes would be predictive for clinical quality, costs and use, prevalence of health conditions, and mortality at the census block group level. Researchers would compare existing deprivation indexes to understand their usefulness for payment policy and for clinical application in public health. The project aims to create a research bench, using data from the Federal Statistical Research Data Centers. One of the benefits of a research bench is that it has secure access and authorization controls. It would also enable harnessing the analytic capacity of external partners such as universities that is not available inside the U. S. Department of Health and Human Services (HHS). The current project plans for the Census Bureau to become the data steward that maintains this resource. The current project builds on an ASPE–AHRQ collaboration on county-level social determinants of health data. Phillips then posed three questions regarding this project:

1. What if PCORTF dollars brought identified data sets together in these research data centers to create a bench of resources so that questions could be answered rapidly?
2. What if federal data strategy authority allowed a core of key health data assets to be maintained for authorized access and those assets did not require 12 to 18 months to assemble?
3. What if a “lab bench” of HHS data assets were available to authorized users to answer the nation’s most pressing health equity questions and to create new tools for policy use?

The second collaboration Phillips discussed would create a “wet lab” for artificial intelligence and machine learning (AI/ML) that would focus on analyzing primary care data. He explained that more than half of all outpatient visits occur in the primary care setting. However, a large majority of data that researchers use to develop AI/ML-based clinical decision support tools does not come from the primary care setting, which lacks capacity to support such projects. Phillips contended that applying decision support tools developed using hospital data to decision-making in primary care could harm people. In

contrast, the wet lab would use data from primary care to develop and test clinical decision support tools. Those tools would then need to meet a certification requirement prior to incorporation into an EHR. He added that this project is consistent with the goals and priorities of AHRQ, ASPE, and PCORI. This project has the potential to generate an AI/ML-powered tool that would calibrate a new clinical decision support tool using data in a health system’s EHR.

The American Board of Family Medicine Foundation has allocated $3 million for three projects to build needed capacity:

1. A 5-year effort to increase capacity in family medicine departments.
2. Exploratory efforts to link doctoral students or postdoctoral scholars who have expertise in AI/ML to work one-on-one with primary care researchers.
3. Ongoing support for infrastructure development for an AI/ML learning collaborative at Stanford University’s Center for Population Health Studies.

Phillips noted that the Gordon and Betty Moore Foundation has expressed interest in joining this effort. He suggested that expanding this collaboration is an opportunity for PCORTF.

Another collaboration focuses on outcomes from federal investments in the clinician workforce. Currently, only $500 million of the $19 billion that Medicare, Medicaid, and the Veterans Health Administration (VHA) spend annually on clinician training requires outcome assessments. The size of the primary care workforce is decreasing, particularly in rural areas and in outpatient settings (Phillips et al., 2022). He said that a recent study linked a 1-year decrease in life expectancy for people living in rural areas with health professional shortages in the rural primary care workforce. Another study released in May 2022 found that only 10 percent of people trained in internal medicine are going into outpatient primary care. He posited that AHRQ, ASPE, and PCORI could collaborate to support efforts to investigate and address the primary care workforce shortage.

The final collaboration Phillips addressed would update, reconfigure, and revive clinical data infrastructure. In 2018, the National Ambulatory Medical Care Survey (NAMCS) initially included 2,999 physicians, but lost 1,352 of them to follow-up because they did not meet the inclusion criteria. The response rate of the remaining 1,647 eligible physicians was approximately 40 percent. Phillips noted that led to a small sample size on which to base the survey findings.² The Centers for Disease Control and Prevention (CDC) con-

___________________

² Full demographics can be found at https://www.cdc.gov/nchs/data/ahcd/namcs_summary/2018-namcs-web-tables-508.pdf (accessed September 14, 2022).

vened an NAMCS workgroup in 2021 that recommended evolving NAMCS to include a broader diversity of ambulatory care providers to more completely capture what is happening in ambulatory care. The workgroup also recommended looking for other opportunities to create longitudinal sampling. Phillips suggested that AHRQ, ASPE, and PCORI consider how they could collaborate to support efforts to improve NAMCS and supplement it with registry data, claims data, and EHR data to create a more valuable data set that is more representative of the care provided in the primary care setting.

Phillips acknowledged Puckrein’s comments about data infrastructure and creating a data lake and noted that there has been a significant loss of community health indicators as a result of the end of four federal health data systems in 2016. One such tool, the Community Health Status Indicators tool, was a reliable source of standardized, local health data for communities during its two decades of operation. Many stakeholders have called for a replacement tool that would enable local health needs assessments, peer comparisons, and the capacity to create communities of solutions locally (Phillips et al., 2021). Phillips wondered how PCORTF investments could support such an effort and if it could be a product of the collaboration between the Census Bureau and the Federal Statistical Research Data Centers that he discussed at the start of his presentation.

DISCUSSION

Supporting Data Stewardship, Access, and Quality

Session moderator Megan Daugherty Douglas, assistant professor of community health and preventive medicine at the Morehouse School of Medicine and director of research and policy at the Morehouse School of Medicine’s National Center for Primary Care, asked Blewett to discuss the role that SHADAC and other data infrastructures can play in data stewardship and improving or supporting improvement in the timeliness, access, and quality of the data. Blewett replied that she has seen some progress in the federal government related to obtaining more accurate race and ethnicity data and to provide more guidance on standards and methods to acquire better self-reported data. She added that states are using the data that they can access, including the administrative data from their public programs, to inform immediate policy decisions. She explained that states submit their data to T-MSIS, then use another, more granular data set to conduct their own analysis, which may or may not replicate what they send to the federal government. Blewett would focus initial efforts on determining how to best incentivize states to collect race and ethnicity data and on helping providers understand how best to encourage people to provide self-reported race and ethnicity data.

Douglas asked Collier to discuss the importance of the collaborations and relationships that go into data reporting, quality, and timeliness and how to think about scaling those best practices. Collier explained that collaborations and relationships are critical to quality and timely data reporting. She noted that good relationships make it possible for her organization to work with its partners to address conflicting or missing data; while legislation around information sharing exists, it cannot force agencies to provide quality data.

Phillips said that when he served on the National Committee on Vital and Health Statistics, there were discussions about what is needed improve the quality of data submitted to these data sets. One clear message from stakeholders was the need to show them how their data would be used. He added that collaborations and relationships are critical for ensuring that these data sets support interventions and activities that matter in the communities providing the data.

A question from the audience asked the panelists to discuss their ideas for collaboration between AHRQ, ASPE, and PCORI to address the need for improved quality of data collected in EHRs, as well as challenges in moving information across the health care system using health information exchanges. Blewett said that the nation’s multipayer system has added complexity to combining data across systems. As a result, real time data from health information exchanges would be difficult to implement. She noted that Minnesota does not have a functional health information exchange because many of the state’s health systems use the Epic EHR, which can be used to share data between health systems already. Her team has worked with a state-level all-payer claims database that has some capacity to do cross-payer analysis with Medicare, Medicaid, and the commercial market. However, that database is missing data from self-insured plans.

Phillips commented that health information exchanges were well-envisioned but have challenges around execution, limited participation, and sustainability efforts. He noted that the all-claims payer databases could be a good resource. He suggested that these databases could represent another opportunity for AHRQ, ASPE, and PCORI to collaborate on projects to support data infrastructure and application.

Considering End Users of Health Care Data

Douglas then asked the speakers to explain who they see as the end users of health care data. Phillips explained that the examples that he discussed focused on clinicians and policy makers as end users. He noted that policy makers can use this data to adjust Medicaid managed care payments so that more resources flow to patients and clinicians in systematically divested communities. He added that clinicians seek to use data to improve the quality of

the care they provide. Blewett noted that researchers and Medicaid agency representatives were interested in using data from the Medicaid equity dashboard project. She added that Medicaid staff are interested in using data to compare their work with that in other states and to learn from one another.

Douglas asked Collier to discuss how the National Center for Fatality Review and Prevention assesses different users’ ability to use and analyze the data themselves versus relying on the support that the center provides. Collier replied that the center offers new users the opportunity to meet with staff. This gives center staff an opportunity to ascertain the degree of support the new user will need. In addition, center staff engages its network to obtain feedback about user needs and opportunities to improve the reporting system.

Elaborating on the AI/ML Wet Lab

Phillips was asked to elaborate on the AI/ML wet lab he described in his presentation. He provided an example of a possible use for the wet lab. Currently, most research into long COVID uses data from hospital or subspecialty clinic EHRs. As a result, all of that data are coming from people who are already diagnosed with long COVID when they arrive in those settings. That approach does not include data from the primary care setting where patients initially present with undifferentiated symptoms and without a long COVID diagnosis. He noted that patients with long COVID present to primary care with a variety of symptoms that clinicians must differentiate from symptoms of heart or lung disease. The data captured during that process in primary care could be used in AI/ML to develop clinical decision support tools informed by more longitudinal data.

Considerations for Expanding Use of Z Codes

Douglas next asked what is needed to expand the use of Z codes. Phillips replied that payment policy would be an effective method to increase use. He added that reimbursement for using Z codes is an important incentive, but a more important reason to use Z codes is that they can help clinicians better identify and address the barriers related to the social determinants of health experienced by their patients. He suggested that integrating this information into EHRs could increase the likelihood of a clinician asking a patient about barriers related to the social determinants of health they may be experiencing. The addition of this information in EHRs to a payment policy that incentivized use of Z codes could increase their use.

He noted that Massachusetts is conducting a demonstration project to test that hypothesis. Massachusetts has established a funding model in which clinicians that identify patients experiencing barriers related to the social

determinants of health can refer that patient to community-based organizations (CBOs) for relevant support. The funds follow the patient, meaning that in this funding model, instead of the clinician receiving reimbursement, the CBOs are reimbursed based on the support they provide to each patient. The Community Care Cooperative, the managed care organization of 18 federally qualified health centers in Massachusetts, has developed a data platform that manages the identification, referral, and feedback loop for people receiving support through this model. The cooperative covers 12.5 percent of all patients in the state who are eligible based on the neighborhood stress score. Providers in the cooperative account for 80 percent of the referrals in the model, primarily because they have an infrastructure to identify and track those individuals. Phillips suggested that this could be an opportunity for AHRQ to support research to identify the data infrastructure needed to disseminate this type of program widely. Blewett agreed with this idea and said it could help address health equity in AHRQ’s priority populations.

Speakers’ Final Thoughts

Douglas then asked the panelists to share their final thoughts. Blewett replied that AHRQ could benefit from engaging state level policy makers and analysts to learn from their work related to the social determinants of health and the effect of structural racism on health equity. Phillips highlighted the opportunity that AHRQ has for helping turn data into information and tools for communities and policy makers. Collier reiterated that child death review teams are an opportunity to see how systems work when they are tested and an opportunity to make profound community change in a fairly real-time format. She emphasized that engaging with child death review from a data perspective is important, but it is also an opportunity to learn how different systems can effectively collaborate, particularly when they are under pressure.

CLOSING SUMMARY OF WORKSHOP 2

Lauren Hughes concluded the workshop by summarizing her takeaways from the presentations and discussions. She noted several speakers emphasized the importance of evidence-based community engagement at the individual, family, and community levels. She said community engagement should include co-creation, being inclusive, sharing governance, being culturally centered, and prioritizing building trust and trustworthiness.

Hughes said she learned from today’s speakers that when performing data collection, researchers should consider the purpose of collecting the data, representativeness of the data collected, and whether that data matters to patients, clinicians, policy makers, and other stakeholders. She said speakers

also highlighted the importance of developing a strategy to address incomplete data on race, ethnicity, and social risk factors. Speakers also suggested researchers should consider input from the end users of data regarding their data access and utilization needs. Speakers said partnerships and collaborations for data collection and use can play a major role in advancing PCOR. Speakers also suggested both building upon existing data infrastructure and building entirely new infrastructures to support new research methods, such as artificial intelligence.

Hughes noted that the discussion raised the importance of recruiting and developing an equity-minded research workforce. Speakers suggested research funders should consider requiring equity impact measurement in comparative effectiveness research. Speakers also said the extended time it currently takes to translate research findings into practice and to impact policy suggests a need to make translation and uptake an active and faster process rather than a passive one. Speakers also emphasized that the representativeness of the research sample used to develop an underlying body of evidence should be considered when disseminating and implementing new interventions and programs.