Appendix D

Building Data Capacity for Patient-Centered Outcomes Research: Interim Report 3–A Comprehensive Ecosystem for PCOR

(Full text of the committee’s third interim report released on January 11, 2022.)¹

___________________

¹https://www.nap.edu/catalog/26396/building-data-capacity-for-patient-centered-outcomes-research-interim-report.

This page intentionally left blank.

Building Data Capacity
for Patient-Centered
Outcomes Research

INTERIM REPORT 3–
A Comprehensive Ecosystem for PCOR

Committee on Building Data Capacity for
Patient-Centered Outcomes Research:
An Agenda for 2021 to 2030

Committee on National Statistics
Division of Behavioral and Social Sciences and Education

Board on Health Care Services
Health and Medicine Division

Computer Science and Telecommunications Board
Division on Engineering and Physical Sciences

A Consensus Study Report of

THE NATIONAL ACADEMIES PRESS
Washington, DC
www.nap.edu

THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001

This activity was supported by a contract between the National Academy of Sciences and the U.S. Department of Health and Human Services (award # HHSP233201400020B/75P00120F37102). Any opinions, findings, conclusions, or recommendations expressed in this publication do not necessarily reflect the views of any organization or agency that provided support for the project.

International Standard Book Number-13: 978-0-309-27366-4
International Standard Book Number-10: 0-309-27366-8
Digital Object Identifier: https://doi.org/10.17226/26396

Additional copies of this publication are available from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.

Copyright 2022 by the National Academy of Sciences. All rights reserved.

Printed in the United States of America

Suggested citation: National Academies of Sciences, Engineering, and Medicine. (2022). Building Data Capacity for Patient-Centered Outcomes Research: Interim Report 3–A Comprehensive Ecosystem for PCOR. Washington, DC: The National Academies Press. https://doi.org/10.17226/26396.

The National Academy of Sciences was established in 1863 by an Act of Congress, signed by President Lincoln, as a private, nongovernmental institution to advise the nation on issues related to science and technology. Members are elected by their peers for outstanding contributions to research. Dr. Marcia McNutt is president.

The National Academy of Engineering was established in 1964 under the charter of the National Academy of Sciences to bring the practices of engineering to advising the nation. Members are elected by their peers for extraordinary contributions to engineering. Dr. John L. Anderson is president.

The National Academy of Medicine (formerly the Institute of Medicine) was established in 1970 under the charter of the National Academy of Sciences to advise the nation on medical and health issues. Members are elected by their peers for distinguished contributions to medicine and health. Dr. Victor J. Dzau is president.

The three Academies work together as the National Academies of Sciences, Engineering, and Medicine to provide independent, objective analysis and advice to the nation and conduct other activities to solve complex problems and inform public policy decisions. The National Academies also encourage education and research, recognize outstanding contributions to knowledge, and increase public understanding in matters of science, engineering, and medicine.

Learn more about the National Academies of Sciences, Engineering, and Medicine at www.nationalacademies.org.

Consensus Study Reports published by the National Academies of Sciences, Engineering, and Medicine document the evidence-based consensus on the study’s statement of task by an authoring committee of experts. Reports typically include findings, conclusions, and recommendations based on information gathered by the committee and the committee’s deliberations. Each report has been subjected to a rigorous and independent peer-review process and it represents the position of the National Academies on the statement of task.

Proceedings published by the National Academies of Sciences, Engineering, and Medicine chronicle the presentations and discussions at a workshop, symposium, or other event convened by the National Academies. The statements and opinions contained in proceedings are those of the participants and are not endorsed by other participants, the planning committee, or the National Academies.

For information about other products and activities of the National Academies, please visit www.nationalacademies.org/about/whatwedo.

COMMITTEE ON BUILDING DATA CAPACITY FOR PATIENT-CENTERED OUTCOMES RESEARCH: AN AGENDA FOR 2021 TO 2030

GEORGE ISHAM (Chair), HealthPartners Institute

JOHN F.P. BRIDGES, The Ohio State University

JULIE BYNUM, University of Michigan

ANGELA DOBES, IBD Plexus, Crohn’s & Colitis Foundation

DEBORAH ESTRIN, Cornell Tech

OLUWADAMILOLA FAYANJU, The University of Pennsylvania

CONSTANTINE GATSONIS, Brown University

ROBERT GOERGE, Chapin Hall, University of Chicago

GEORGE HRIPCSAK, Columbia University

LISA IEZZONI, Massachusetts General Hospital

S. CLAIBORNE JOHNSTON, The University of Texas at Austin

MIGUEL MARINO, Oregon Health & Science University

ELIZABETH MCGLYNN, Kaiser Permanente

DAVID MELTZER, University of Chicago

PAUL TANG, Stanford University and Palo Alto Medical Foundation

KRISZTINA MARTON, Study Director

CRYSTAL BELL, Associate Program Officer

RUTH COOPER, Associate Program Officer

MARY GHITELMAN, Senior Program Assistant

BRIAN HARRIS-KOJETIN, Director, Committee on National Statistics

SHARYL NASS, Director, Board on Health Care Services

JON EISENBERG, Director, Computer Science and Telecommunications Board

SAUL RIVAS (National Academy of Medicine Fellow), University of Texas Rio Grande Valley

COMMITTEE ON NATIONAL STATISTICS

ROBERT M. GROVES, (Chair), Georgetown University

LAWRENCE D. BOBO, Harvard University

ANNE C. CASE, Princeton University, Emeritus

MICK P. COUPER, University of Michigan

JANET M. CURRIE, Princeton University

DIANA FARRELL, JPMorgan Chase Institute, Washington, DC

ROBERT GOERGE, Chapin Hall, University of Chicago

ERICA L. GROSHEN, Cornell University

HILARY HOYNES, University of California-Berkeley

DANIEL KIFER, The Pennsylvania State University

SHARON LOHR, Arizona State University, Emeritus

JEROME P. REITER, Duke University

JUDITH A. SELTZER, University of California-Los Angeles, Emeritus

C. MATTHEW SNIPP, Stanford University

ELIZABETH A. STUART, Johns Hopkins University

JEANNETTE WING, Columbia University

BRIAN HARRIS-KOJETIN, Director

MELISSA CHIU, Deputy Director

CONSTANCE F. CITRO, Senior Scholar

BOARD ON HEALTH CARE SERVICES

DAVID BLUMENTHAL (Chair), The Commonwealth Fund

ANDREW BINDMAN, Kaiser Foundation Health Plan, Inc.

NIRANJAN BOSE, Gates Ventures

MELINDA J. BEEUWKES BUNTIN, Vanderbilt University School of Medicine

NEIL S. CALMAN, The Institute for Family Health

PAUL CHUNG, Kaiser Permanente School of Medicine

PATRICIA M. DAVIDSON, Johns Hopkins University School of Nursing

MARTHA DAVIGLUS, University of Illinois at Chicago

JENNIFER E. DEVOE, Oregon Health & Science University

R. ADAMS DUDLEY, University of Minnesota

RICHARD G. FRANK, Harvard Medical School

TERRY FULMER, John A. Hartford Foundation

CINDY GILLESPIE, Arkansas Department of Human Services

ELMER HUERTA, The George Washington University Cancer Center

SHARON INOUYE, Harvard Medical School

JOHN LUMPKIN, Blue Cross Blue Shield of North Carolina Foundation

FAITH MITCHELL, The Urban Institute

DAVID B. PRYOR, Ascension Health

TRISH RILEY, National Academy for State Health Policy

WILLIAM SAGE, The University of Texas at Austin

HARDEEP SINGH, Baylor College of Medicine

SHARYL NASS, Director

COMPUTER SCIENCE AND TELECOMMUNICATIONS BOARD

LAURA HAAS (Chair), University of Massachusetts, Amherst

DAVID CULLER, University of California, Berkeley

ERIC HORVITZ, Microsoft Corporation

CHARLES ISBELL, Georgia Institute of Technology

BETH MYNATT, Georgia Institute of Technology

CRAIG PARTRIDGE, Colorado State University

DANIELA RUS, Massachusetts Institute of Technology

FRED B. SCHNEIDER, Cornell University

MARGO SELTZER, University of British Columbia

NAMBIRAJAN SESHADRI, University of California, San Diego

MOSHE VARDI, Rice University

JON EISENBERG, Senior Board Director

Acknowledgments

This Consensus Study Report was reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise. The purpose of this independent review is to provide candid and critical comments that will assist the National Academies of Sciences, Engineering, and Medicine in making each published report as sound as possible and to ensure that it meets the institutional standards for quality, objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process.

We thank the following individuals for their review of this report: Robert M. Califf, Verily Life Sciences, Google Health, and Duke University; Steven B. Cohen, Statistical and Data Sciences, RTI International; Heidi M. Crane, Clinical Cohort and Comorbidity Research Core and Center for AIDS Research, University of Washington; Beth Jarosz, U.S. Programs and KidsData, Population Reference Bureau; and Russell L. Rothman, Institute for Medicine and Public Health, Vanderbilt University.

Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the conclusions of this report, nor did they see the final draft before its release. The review of this report was overseen by Andrew B. Bindman, chief medical officer, Kaiser Foundation Health Plan and Hospitals, and Alicia L. Carriquiry, Department of Statistics, Iowa State University. They were responsible for making certain that an independent examination of this report was carried out in accordance with the standards of the National Academies and that all review comments were carefully considered. Responsibility for the final content rests entirely with the authoring committee and the National Academies.

This page intentionally left blank.

Contents

Summary

1 Introduction

2 Federal Partnerships

3 State-Level Data and Collaborations

4 Clinical Trial Networks and Collaborations

5 Public-Private Partnerships

6 Collaborations with Patient Groups

Appendixes

A Biographical Sketches of Committee Members

B Workshop Agenda

C Biographical Sketches of Workshop Speakers

This page intentionally left blank.

Boxes and Figures

BOXES

1-1 Key Data Infrastructure Functionalities in the Existing Strategic Framework for Patient-Centered Outcomes Research

1-2 Building Blocks of the Patient-Centered Outcomes Research Data Infrastructure

1-3 Statement of Task for the Overall Study

2-1 Collaborations Between the Centers for Medicare & Medicaid Services Office of Minority Health and Federal Partners

FIGURES

1-1 Patient-Centered Outcomes Research Trust Fund: Three streams of work and funding

1-2 The Office of the Assistant Secretary for Planning and Evaluation’s strategic framework for the patient-centered outcomes research data infrastructure

2-1 Prevalence of diabetes in 2018

4-1 National Cancer Institute National Clinical Trials Network structure

This page intentionally left blank.

Summary

The Office of the Assistant Secretary for Planning and Evaluation (ASPE), in partnership with other agencies and divisions of the U.S. Department of Health and Human Services (HHS), coordinates a portfolio of projects that build data capacity for conducting patient-centered outcomes research (PCOR). PCOR focuses on producing scientific evidence on the effectiveness of prevention and treatment options to inform the health care decisions of patients, families, and health care providers, taking into consideration the preferences, values, and questions patients face when making health care choices. The data infrastructure includes data sources and functionalities that support the research. Major building blocks are the services, standards, policies, and governance that enable the use of the data.

ASPE asked the National Academies of Sciences, Engineering, and Medicine to appoint a consensus study committee to identify issues critical to the continued development of the data infrastructure for PCOR. The committee’s work will contribute to ASPE’s development of a strategic plan that will guide its work related to PCOR data capacity over the next decade.

As part of its information-gathering activities, the committee organized three workshops to collect input from stakeholders on the PCOR data infrastructure, which includes a variety of types of data, such as clinical data, research data, administrative data from payer records, and patient-provided data. This report, the third in a series of three interim reports, summarizes the discussion and committee conclusions from the third workshop, which focused on ways of enhancing collaborations, data linkages,

and the interoperability of electronic databases to make the PCOR data infrastructure more useful in the years ahead. Participants in the workshop included researchers and policy experts working in these areas. The first report centered on emerging data needs¹ and the second report on data standards, methods, and policy.²

The conclusions included in this interim report are based primarily on the input collected as part of the workshop, background documentation received from ASPE and other public sources, and the committee members’ synthesis and expert judgment regarding the input received. As an interim report based on one in a series of information-gathering activities, the scope of this report is narrowly focused on a subset of key topics relevant to the committee’s charge. The conclusions reached by the committee are, at this stage, fairly high-level. After completing all of its information-gathering activities, which include but are not limited to the three workshops, the committee will also issue a final report containing the study’s overall findings and conclusions.

FEDERAL PARTNERSHIPS

The workshop discussed in this report included several stakeholder groups that are involved in collaborations focused on PCOR. While it was not possible to discuss many important collaborative projects, the first session, which was held with federal agency representatives, highlighted several key areas where additional collaborative work could continue to build and strengthen the PCOR data infrastructure.

CONCLUSION 2-1: Collaboration among federal agencies and between federal agencies and other partners (such as states, patient groups, and others) is essential for continuing to build the patient-centered outcomes research data infrastructure. The areas where additional collaboration would be particularly useful include the following:

• Increasing consistency in the use of standards for data interoperability and element definitions;
• Addressing barriers that hinder data linkages, such as the limitations associated with health identifiers and mitigating potential selection biases resulting from linkage error;

___________________

¹https://www.nap.edu/catalog/26297/building-data-capacity-for-patient-centered-outcomes-research-interim-report.

²https://www.nap.edu/catalog/26298/building-data-capacity-for-patient-centered-outcomes-research-interim-report.

• Balancing the burden of the data collections and disclosure risks with the value of the datasets;
• Communicating the usefulness of the data collections to those who are asked to provide data about themselves and those who collect the data;
• Promoting discussion and education about fitness for use of the data; and
• Working with stakeholders and patients to promote sharing of data.

While there is frequent collaboration among HHS partners on PCOR data infrastructure work, and ASPE’s public website contains a comprehensive list of past and current projects funded from the Patient-Centered Outcomes Research Trust Fund, additional dissemination efforts focused on external stakeholders could further increase the usefulness of these investments.

CONCLUSION 2-2: There is a need to increase awareness among all stakeholders about new data infrastructure developments funded by the Patient-Centered Outcomes Research Trust Fund. Increased awareness will enhance the efficiency and effectiveness of research, which will increase the impact of the investments made in infrastructure development.

STATE-LEVEL DATA AND COLLABORATIONS

Many states have robust data collection systems and can produce information that is useful to state and local policy makers. State-generated data are also valuable at the national level, including for answering broader questions about issues that may be influenced by local policy, such as health care access and disparities.

CONCLUSION 3-1: There are opportunities to learn from what states have accomplished in building data capacity.

The data collected, their quality, and ease of access to the data vary by state. Challenges associated with access make it particularly difficult to use state-generated data for research at the national level.

CONCLUSION 3-2: The usefulness of data available for Patient-Centered Outcomes Research could be increased by the sharing and adoption of best practices among the states for the data collected, their quality, and ease of access.

CLINICAL TRIAL NETWORKS AND COLLABORATIONS

The session on clinical trial networks and collaborations illustrated the need for better integration between clinical care and research in ways that align differing interests and are mutually beneficial. Better integration can improve the data available for patient care as well as the data available for research.

CONCLUSION 4-1: Infrastructure investments could enhance the utility of data routinely generated in the course of care for clinical trials.

PUBLIC-PRIVATE PARTNERSHIPS

While the benefits of data sharing are clear, the workshop also underscored the risks involved for the organizations providing the data.

CONCLUSION 5-1: Successful partnerships across health care systems require participant trust, clear evidence of mutual benefit, and the ability to control risk.

COLLABORATIONS WITH PATIENT GROUPS

Collaborations with patient organizations can help to address patient concerns about participating in research studies and to build patient engagement, which is important for achieving a patient-centered approach. Disease registries directed by patient groups can be a particularly useful additional source of data, one that provides information that would not be available to researchers otherwise.

CONCLUSION 6-1: Patient groups can be helpful partners in all aspects of Patient-Centered Outcomes Research, including engaging patients in order to improve research participation and the impact of results.

CONCLUSION 6-2: Patient-directed disease registries can be a source of in-depth, longitudinal, prospective clinical and patient-reported data that are not available from other data sources.

__________________

1

Introduction

The Office of the Assistant Secretary for Planning and Evaluation (ASPE), in partnership with other agencies and divisions of the U.S. Department of Health and Human Services (HHS), coordinates a portfolio of projects that build data capacity for conducting patient-centered outcomes research (PCOR). The PCOR data infrastructure provides decision makers with objective, scientific evidence on the effectiveness of treatments, services, and other interventions used in health care. This research is frequently focused on analyzing existing data to address questions and provide objective information for the purpose of informing real-world health care decisions.

BACKGROUND

The legal framework that established funding for research on the outcomes and effectiveness of treatments and health care interventions dates back to the 2003 Medicare Prescription Drug, Improvement, and Modernization Act. This act provided authorization for the Agency for Healthcare Research and Quality (AHRQ) to support research comparing the outcomes and effectiveness of treatments and clinical approaches and to disseminate the findings from this research. In 2009, the American Recovery and Reinvestment Act provided additional funding to AHRQ, the National Institutes of Health, and HHS for research that compares the effectiveness of medical options. In 2010, the Patient Protection and Affordable Care Act provided further authorization for research that assists patients, clinicians, purchasers, and policy makers in making informed health decisions.

To facilitate PCOR, in 2010 Congress established the Patient-Centered Outcomes Research Trust Fund (PCOR Trust Fund) within the Department of the Treasury. The goals of the PCOR Trust Fund are to fund PCOR, disseminate research findings, and develop a data infrastructure for PCOR. The PCOR Trust Fund has been reauthorized through 2029, through H.R.1865 of the Further Consolidated Appropriations Act of 2020. The most recent statute specified intellectual and developmental disabilities, as well as maternal mortality, as research priorities. The statute also called for PCOR studies to include consideration of the full range of outcomes data. Specifically, the law states that

The bulk of the PCOR Trust Fund funding (80 percent) is allocated for research and is made available through the Patient-Centered Outcomes Research Institute, a nongovernmental organization established by Congress for this purpose. Approximately 16 percent of the PCOR Trust Fund is set aside for disseminating research findings, incorporating findings into clinical practice, and training researchers in PCOR. The agency overseeing this work is AHRQ.

The remaining funding, which constitutes 4 percent of the PCOR Trust Fund, is allocated for building data capacity for PCOR and is overseen by ASPE. Specifically, Section 937(f) of the Public Health Service Act instructed the Secretary of HHS to

___________________

¹https://www.ssa.gov/OP_Home/ssact/title11/1181.htm.

Figure 1-1 shows how the PCOR funding and work are allocated across the three entities. This National Academies of Sciences, Engineering, and Medicine (National Academies) study is focused on issues relevant to ASPE’s continued work on the PCOR data infrastructure, in other words, on the priorities for the use of the 4 percent of the funding that is allocated to HHS for work related to the data infrastructure for PCOR.

As the coordinating agency for the data infrastructure investment portfolio across HHS agencies, ASPE guides the PCOR data infrastructure’s strategic framework and vision, sets funding priorities, and coordinates interagency workgroups. ASPE’s work is assisted by a Leadership Council for the PCOR Trust Fund, which includes representatives from other HHS agencies, including the Administration for Children and Families; the Administration for Community Living; AHRQ; the Assistant Secretary for Preparedness and Response; the Centers for Disease Control and Prevention (CDC); the Centers for Medicare & Medicaid Services; the Food and Drug Administration (FDA); the Health Resources and Services Administration; the Indian Health Service; the National Institutes of Health; the Office of

___________________

²https://aspe.hhs.gov/collaborations-committees-advisory-groups/os-pcortf/about-os-pcortf.

the Chief Technology Officer; the Office of the National Coordinator for Health Information Technology; and the Substance Abuse and Mental Health Services Administration.

The Leadership Council provides input on priorities for the portfolio, including projects to fund. During the period from 2010 to 2019, the PCOR Trust Fund funded 53 projects, which translated to 76 agency awards, totaling approximately $131 million.

Figure 1-2 is a visual representation of ASPE’s current framework for the PCOR data infrastructure. The bottom row shows the main data sources feeding into the PCOR infrastructure. Data collected as part of clinical care include data collected for health care delivery and for billing purposes. Examples of primary data collected as part of research studies include data from clinical trials and data from national health surveys. Other examples of data sources include

• Medicare or Medicaid claims data,
• quality or outcomes data collected by health care providers for the purposes of improving health care value,
• FDA data on the safety of medications and medical devices, and
• CDC data on births and deaths provided by state public health authorities.

The framework describes the relationship between the data sources and the current key functionalities and focus areas (middle row) that support the research. The key functionalities are described in further detail in Box 1-1. Major building blocks are the services, standards, policies, and governance that enable the use of the data for research, described in further detail in Box 1-2. The top row shows the key data users and contributors of data. A more detailed overview of ASPE’s work and the projects funded to date will be included in the final report, at the conclusion of the committee’s review.

ISSUES FOR THE COMMITTEE

ASPE asked the National Academies to appoint a consensus study committee and identify issues critical to building data capacity for PCOR and for generating new evidence to inform health care decisions. The input provided by the committee will contribute to ASPE’s strategic planning for its work related to the data infrastructure over the next decade. The study is part of a broader initiative by ASPE intended to update the strategic plan in light of the reauthorization of the PCOR Trust Fund and advances in health information technology and interoperability tools in recent years.

The study is a collaboration of three units of the National Academies: the Committee on National Statistics, the Board on Health Care Services, and the Computer Science and Telecommunications Board. The 15-person consensus study committee has a diverse membership, including experts with decades of experience, as well as emerging leaders, in the broad fields of (1) PCOR; (2) research methods, statistics, and demography; (3) computer science and data infrastructure; and (4) patient engagement and patient perspectives. Appendix A contains the biographical sketches of the committee members.

As part of its information-gathering activities, the committee was asked to organize three workshops to collect input from stakeholders on aspects of the charge developed in consultation with ASPE. The workshops focused on key topics that the committee believed would particularly benefit from broad input from a variety of data users and other stakeholders. The committee’s conclusions from each workshop are summarized in a series of interim reports, the first of which centered on emerging data needs. That first interim report summarized the discussion and committee conclusions from the first workshop, which focused on looking ahead at data user needs over the next decade.³ The second interim report in the series centered on data standards, methods, and policies that could make the PCOR data infrastructure more useful.⁴ This report summarizes the discussion and committee conclusions from the third workshop, which focused on ways of enhancing collaborations, data linkages, and the interoperability of electronic databases to make the PCOR data infrastructure more useful in the years ahead.

As an interim report focused on one in a series of information-gathering activities, the scope of this report is limited to a subset of the topics relevant to the committee’s charge, and the conclusions reached by the committee are, at this stage, fairly high-level. Some aspects of the topics discussed are

___________________

³https://www.nap.edu/catalog/26297/building-data-capacity-for-patient-centered-outcomes-research-interim-report.

⁴https://www.nap.edu/catalog/26298/building-data-capacity-for-patient-centered-outcomes-research-interim-report.

examined in further detail in other workshops. After completing all of its information-gathering activities, the committee will issue a final report, which will integrate and examine these topics in further detail.

Box 1-3 shows the committee’s Statement of Task for the overall study. The committee will address this charge in its final report, integrating what was learned from the workshops and from all other forms of input, including public meetings with HHS staff and background documentation available on the history and operations of the PCOR Trust Fund. The final report will contain overall findings and conclusions from the study, on the

basis of the committee’s further deliberations and integrated judgment on the input received and materials reviewed.

Appendix B shows the agenda for the workshop, which was held on June 14, 2021. The committee’s goal for this event was to bring together researchers and policy experts to

• discuss how research and data collaborations can evolve to meet PCOR and data capacity challenges, and how HHS can support effective research and data collaborations;
• identify barriers and potential solutions to the access and use of linked public data, as well as to the access and use of linked public and private/proprietary data; and
• discuss the feasibility and utility of developing a phased-in approach to building the interoperable data capacity for PCOR with existing databases in HHS, other federal departments, and the private sector.

Invited speakers in each of the sessions were asked to reflect on the general topics above. An obvious limitation of an activity of this type is that only a small number of stakeholders can be invited to speak. To compensate for this limitation, the invited participants included diverse experts working in a variety of areas and on a range of types of projects, including both early career researchers and experts with decades of experience. A recording of the workshop as well as the presentation slides used by the speakers are available on the National Academies website at www.nationalacademies.org/PCORData.

Prior to the workshop, information about the event was disseminated through National Academies mailing lists and on the project website. To collect additional stakeholder input, members of the public were invited to provide comments on topics related to the workshop (or any other topic related to the committee’s charge), using a public input form available on the National Academies website.

OVERVIEW OF THE REPORT

This report is organized around the main sessions of the workshop and discusses the following topics: federal partnerships and collaborations with other stakeholders (Chapter 2); state-level data and collaborations (Chapter 3); clinical trial networks and collaborations (Chapter 4); public-private partnerships (Chapter 5); and collaborations with patient groups (Chapter 6). The points conveyed by the workshop participants do not necessarily reflect the views of the committee. In each chapter, a summary of the input received is followed by the committee’s conclusions.

The conclusions are based primarily on the input collected as part of the workshop, background documentation received from ASPE and other public sources, and the committee members’ synthesis and expert judgment. Because this is an interim report, the committee’s conclusions at this stage are big-picture conclusions, which will be integrated with additional input over the course of the study.

This page intentionally left blank.

2

Federal Partnerships

This chapter summarizes workshop presentations and discussion focused on federal partnerships and collaborations with other stakeholder groups. The brief overview of the input received from the presenters is followed by the committee’s conclusions.

Micky Tripathi, National Coordinator for Health Information Technology, discussed the work that the Office of the National Coordinator for Health Information Technology (ONC) is doing to support activities related to patient-centered outcomes research (PCOR). He noted that more than a decade after the passage of the Health Information Technology for Economic and Clinical Health Act of 2009, the United States has several well-functioning electronic health records (EHR) systems that are widely used by health care providers and represent a common foundation for capturing and sharing clinical data, even if not all providers are satisfied with their EHRs. Tripathi argued that this is a good time to consider the opportunities offered by EHRs, beyond their core functions of supporting medical records and payment processing, stating that EHRs are still a barely tapped source of information for research.

One area of focus for ONC in recent years has been the advancement and harmonization of data standards, including supporting the use of common data models, looking at research-data models as well as clinical data-models, and thinking about data integration. ONC has supported work on patient matching, aggregating, and linking through the development of tools and advancement of data standards. Tripathi also mentioned structured data capture as another area of past work aimed at facilitating the reuse of these data for clinical research.

Another area of work at ONC focuses on real-world data. Projects included testing a standards-based approach to establishing a coordinated registry network for data regarding women’s health technologies, collecting patient-reported outcomes (PROs) through health IT and leveraging Fast Healthcare Interoperability Resources (FHIR), and using privacy preserving machine learning techniques to enable health information exchanges to support COVID-19-focused PCOR.

Tripathi said that ONC has done work related to creating a privacy and security framework for PCOR by developing resources that support the protection of privacy and the security of electronic health data. The agency also worked on identifying limitations in developing machine learning training datasets when pursuing complex health related research questions. ONC also collaborated with MITRE Corporation to increase the variety of reliable and robust synthetic data, particularly for opioid, pediatric, and complex care use cases. The project focuses on enhancing an open-source synthetic data engine that uses publicly available data to generate synthetic health records. Such tools can safeguard patient privacy and support appropriate stewardship practices in which real patient data are only accessed and used when necessary. These and other mentioned projects have been funded through the Patient-Centered Outcomes Research Trust Fund (PCORTF).¹

Looking ahead, Tripathi said that ONC has several initiatives that will directly support PCOR and patient engagement in PCOR. One initiative is focused on standards for EHR data. ONC will be releasing a new version of the U.S. Core Data for Interoperability, which is a standardized set of health data classes and constituent data elements for EHRs and for nationwide, interoperable health information exchanges. The data are supposed to be available not only for provider-to-provider and provider-to-payer exchanges but also for patients.

Allison Oelschlaeger, Office of Enterprise Data & Analytics, Centers for Medicare & Medicaid Services (CMS), discussed CMS data and data linkages. For context, Oelschlaeger noted that more than 130 million Americans receive health coverage through programs administered by CMS—including Medicare, Medicaid, and the Children’s Health Insurance Program—and the health insurance marketplace. As a result, CMS collects large amounts of data, which are an invaluable resource for comparative clinical effectiveness research. The data include information on enrollment and patient characteristics; utilization and cost of health care services, such as treatments and therapies; and diagnoses.

___________________

¹ For more information on some of the ONC PCOR data infrastructure projects, see https://www.healthit.gov/topic/scientific-initiatives/building-data-infrastructuresupport-patient-centered-outcomes-research.

Despite the wealth of data collected by CMS, Oelschlaeger said this information is often insufficient on its own for research on clinical effectiveness. Much information is missing, including

• cause of death data;
• clinical data (for example, lab results);
• certain patient demographic data (for example, income and high-quality race/ethnicity data);
• data related to social determinants of health;
• health behaviors data;
• patient-generated health information; and
• human services data.

Oelschlaeger said that options for enhancing the CMS data with the missing information could include undertaking new data collection, imputation, or linkages to other data sources.

The PCOR Trust Fund funded several CMS projects throughout the years. One of these projects was Blue Button 2.0,² which is an application programming interface (API) that allows beneficiaries to connect their data to applications and services they trust and enables them to contribute their data to research projects. Blue Button leverages FHIR standards, also discussed by Tripathi. CMS data linkage projects that have been funded by the PCOR Trust Fund include³

• Augmenting the National Hospital Care Survey Data through Linkages with Administrative Records;
• National COVID-19 Longitudinal Research Database, which is linked to CMS data; and
• National Death Index—Medicare Enrollment Data Linkage.

Oelschlaeger said that CMS also has other projects funded through other agencies to link CMS data and data from other sources, including

• Surveillance, Epidemiology, and End Results;
• U.S. Renal Data System;
• Health and Retirement Survey; and
• National Health and Aging Trends Study.

___________________

²https://bluebutton.cms.gov/.

³ For more information on specific projects, see https://aspe.hhs.gov/collaborations-committeesadvisory-groups/os-pcortf/explore-portfolio.

Oelschlaeger discussed several barriers to data linkages that involve CMS data. She said that for privacy reasons, personal information is collected less often, and datasets are increasingly lacking patient identifiers that are sufficient for linkages. Private and proprietary datasets, such as registries, have additional limitations for sharing identifiers. She also noted that the volume of patient records necessary to access in order to perform the linkages is often larger than what is needed once the linkage is complete.

According to Oelschlaeger, researchers increasingly want to use tokenization solutions. These are methods that assign unique keys to the datasets being linked, enabling the linkage to happen without the sharing of patient identifiers. The challenge CMS encountered is that there are a variety of tokenization solutions, and researchers have a variety of preferences about what to use. Oelschlaeger noted that the lack of consistency (the differences in degree of difficulty across different populations) is also a challenge in probabilistic matching, specifically the question of how good the match needs to be in order for the linkage to happen.

Oelschlaeger also discussed the CMS Virtual Research Data Center (VRDC), which is meant to provide a secure and efficient mechanism for researchers to virtually access and analyze CMS data and potentially address some of the challenges. The VRDC allows researchers to access CMS data and perform their own analyses and manipulation of those data virtually, from their independent workstations, and it allows them to download aggregate results from the analyses they perform. The VRDC enables faster access at a lower cost to the CMS data that are already linked by a unique beneficiary ID. Researchers also have the ability to upload other data to the VRDC to perform additional linkages.

Meagan Khau, CMS Office of Minority Health, described the mission of her office as serving as the principal advisor to the agency on the needs of minority populations, including racial and ethnic minorities; people with disabilities; the LGBTQ+ community; individuals with limited English proficiency; rural populations; and persons otherwise adversely affected by persistent poverty or inequality. She highlighted two sections of Executive Order 13985, which are focused on the role of data in meeting the needs of these populations. Specifically, Section 4(a) focuses on methods for assessing equity, which underscores the need to collect demographic data in order to fully assess the extent of existing health disparities and the impact of health equity responses. Section 9(a) establishes a workgroup to gather necessary data to measure and advance equity.

Khau noted that federal datasets often are not disaggregated by key demographic variables, such as race, ethnicity, gender, disability, income, and veteran status. CMS is working on gaining a better understanding of what datasets contain data elements of interest, and what standards are being used. As an example, for race, the Office of Management and Budget

1997 standards use five categories, whereas the 2011 Department of Health and Human Services (HHS) standards⁴ use 14 categories. Various standards and approaches are used in various EHRs.

Detailed, disaggregated data are needed to target interventions, in Khau’s view. As an example (Figure 2-1), she noted that in 2018 the prevalence of diabetes in the population as a whole was around 11 percent. Using five race categories, the prevalence was highest among Native Hawaiian and other Pacific Islanders, at 15 percent. However, when the data are further disaggregated, it becomes clear that within this group the prevalence of diabetes is particularly high among Samoans, at 22 percent.

In terms of the ability to assess equity with respect to demographic data elements identified in Executive Order 13985, such as race, ethnicity, religion, income, geography, gender identity, sexual orientation, and disability, Khau pointed out that not all these data elements are collected across the different HHS programs. This means that in some cases new data collections may be necessary, and it will be important to apply the right standards to each of the data elements, in light of how the data are anticipated to be used.

Khau highlighted several projects that the CMS Office of Minority Health is working on, using CMS data. The Mapping Medicare Disparities Tool⁵ is an interactive map that allows users to identify areas of disparities between subgroups of Medicare beneficiaries (for example, by race or ethnicity), chronic disease prevalence, health outcomes, spending, and utilization. The tool uses Medicare fee-for-service data. The office has also released two reports that use stratified data to look at health disparities.⁶

Khau highlighted several collaborations between the CMS Office of Minority Health and other federal partners. These projects are summarized in Box 2-1. She also called attention to the office’s Minority Research Grant Program, which provides funding for principal investigators at minority-serving institutions to conduct research focused on opportunities to embed health equity into CMS programs.

Mitra Rocca, Center for Drug Evaluation and Research of the Food and Drug Administration (FDA), said that the FDA has 13 projects that are funded by the PCOR Trust Fund.⁷ The projects are as follows:

___________________

⁴https://aspe.hhs.gov/basic-report/hhs-implementation-guidance-data-collection-standards-race-ethnicity-sex-primary-language-and-disability-status.

⁵https://data.cms.gov/mapping-medicare-disparities.

⁶ See https://www.cms.gov/files/document/racial-ethnic-gender-disparities-health-care-medicareadvantage.pdf and https://www.cms.gov/files/document/omh-rural-urban-report-2020.pdf.

⁷https://aspe.hhs.gov/collaborations-committees-advisory-groups/os-pcortf/explore-portfolio.

• Collection of Patient-Provided Information through a Mobile Device Application for Use in Comparative Effectiveness and Drug Safety Research;
• Common Data Model Harmonization (CDMH) and Open Standards for Evidence Generation;
• Cross-Network Directory Service;
• CURE ID: Aggregating and Analyzing COVID-19 Treatments from EHRs & Registries Globally;
• Developing a Strategically Coordinated Registry Network (CRN) for Women’s Health Technologies;

• Development of a Natural Language Processing (NLP) Web Service for Public Health Use;
• Enhancing Data Resources for Studying Patterns and Correlates of Mortality in Patient-Centered Outcomes Research: Linking National Death Index (NDI) and Commercial Claims;
• Making Medicaid Data More Accessible Through Common Data Models and FHIR APIs;
• SHIELD - Standardization of Lab Data to Enhance Patient-Centered Outcomes and Value-Based Care;
• Source Data Capture from Electronic Health Records (EHRs): Using Standardized Clinical Research Data;
• Standardization and Querying of Data Quality Metrics and Characteristics for Electronic Health Data;
• Utilizing Data from Various Data Partners in a Distributed Manner; and
• WHT-CRN Project: Bridging the PCOR Infrastructure and Innovation through Coordinated Registry Network (CRN) Community of Practice.

One of the projects that Rocca leads is the Source Data Capture from EHRs. The goal of the project, which is a collaboration with the University of California, San Francisco (discussed in additional detail by Laura Esserman, Chapter 4), is to develop methods and tools to automate the flow of structured EHR data into external systems. The second project, co-led by Rocca in collaboration with the National Institutes of Health (NIH), is Common Data Model Harmonization and Open Standards for Evidence Generation. This project is a collaboration among five agencies, focusing on harmonizing across multiple common data models and generating real-world evidence from real-world data. An FDA project that is focused on data linkages in particular is the Enhancing Data Resources for Studying Patterns and Correlates of Mortality in PCOR, a project that links commercial claims data to data from the National Death Index, which is a centralized database of death records from state vital statistics offices. A collaboration among the Centers for Disease Control and Prevention (CDC), CMS, and FDA, the goal of this project is to increase the availability of information on the cause of death.

With input from the leads of the various PCOR Trust Fund projects at FDA, Rocca identified several ways collaborations could evolve to meet PCOR and data challenges going forward:

• Develop an infrastructure to support research.
• Adopt Findable, Accessible, Interoperable, and Reusable (FAIR) principles as a goal and use metrics to measure the progress.

• Use distributed data models to conduct analyses across multiple institutions, with data remaining behind data-partner firewalls.
• Build trust and validation so they are engineered into the system, which would enable collaborators to run analytic software for one another.
• Develop open-source tools to support sharing, discovering, and reusing research data.
• Convene workshops with internal and external stakeholders around particular problems.
• Improve the quality and completeness of EHR data.

In terms of the barriers and potential solutions to the access and use of linked public data, Rocca said that there is a need for a strategy and a set of standards at the HHS level that address the challenges associated with lacking a master identifier to help link several sources of data. Related to that, there is a need for standards that address the issue of re-identification. Rocca said there is also a need for a systematic review of HHS data sources with an eye toward transparency and a need to develop informed consent guidelines that enable the sharing of both public- and private-sector data.

Rocca also mentioned the need to establish a formal ontology at the HHS level to make it easier to find data, and for establishing a metadata registry and repository for data elements, controlled terminologies, and mapping for controlled terminologies. She also highlighted the need for interoperability, and challenges associated with the lack of standardization for the data that are collected. She argued that the integration of health care and clinical research will require a change in culture that begins at the point of care, where data are generated.

Regarding suggestions for building an interoperable data capacity for PCOR, Rocca highlighted the following:

• Linking existing databases within HHS and other federal government agencies and the private sector;
• Developing a universal data use agreement;
• Applying tools, standards, and services developed as part of PCOR Trust Fund projects to other types of HHS data;
• Encouraging the development of common architectures and integration frameworks to enable interoperability, rather than developing single solutions; and
• Focusing the PCOR Trust Fund investments on cutting-edge solutions that may result in technical leaps.

Adi Gundlapalli, Public Health Informatics Office, CDC, said that patient-level data with sufficient granularity are essential for improving

health outcomes and that these data need to be made available in a way that preserves privacy. He argued that current laws and policies around the use of patient-level data are nuanced, and sometimes conflicting, which creates confusion for researchers, providers, and patients. This issue needs to be addressed in a way that balances individual privacy considerations, the risk of re-identification, and the utility of the datasets.

Making disease-specific datasets available for public use is an area where balancing the considerations discussed by Gundlapalli would be particularly important. He noted that recently CDC made available three COVID-19 datasets for public use, and this was accomplished by working closely with HHS to apply privacy preserving measures, including, in some cases, suppression algorithms, due to the small cell sizes. Gundlapalli said that the work done at CDC over the past year to acquire, store, provide secure access to, and analyze large datasets with high dimensionality will be an enduring resource for data capacity.

Regarding barriers and potential solutions for increased access to linked public data, Gundlapalli reiterated the risks of privacy breaches and re-identification, which, he said, have to be addressed. He noted that CDC receives only de-identified data. However, there are use cases where linking individuals within a dataset or across datasets, such as vaccination and case records for COVID-19, has tangible benefits for public health action. Because of this, CDC, in collaboration with the HHS Office of the Chief Information Officer, has been evaluating the feasibility of implementing privacy preserving record linkage (PPRL) techniques for public health data at the state, tribal, local, and territorial levels before the data are sent to CDC, with the COVID-19 vaccination data as a use case. These techniques can ensure that personally identifiable information remains within the jurisdictions’ firewalls.

PPRL solutions are now available through commercial vendors, and PPRL algorithms have been applied to large, commercially available health datasets, such as laboratory data, pharmacy data, claims information, and EHRs. There are also many published examples of real-world applications of PPRL and their associated benefits, and Gundlapalli argued that these benefits have to be balanced with the risk to privacy and the efforts required to implement PPRL solutions. He added that potential linkages of publicly available data with private, proprietary data are especially interesting, and these opportunities merit detailed consideration.

As far as building interoperable data capacity, Gundlapalli said that CDC also has ongoing projects that address this issue, and it may be useful to consider what can be learned from that work. CDC has been actively working on public health data modernization, an effort that was given a boost with recent COVID-19-related funding. Through a set of targeted investments across three priority thematic areas, CDC aims to promote the

reporting of clinical and laboratory data to ensure that core public health surveillance systems are interoperable, and the agency supports crosscutting upgrades such as the migration of the data to the cloud and access to new data sources. CDC is hoping that these efforts will also support the interoperable data capacity for PCOR. Gundlapalli summarized the three priority areas as (1) data sharing across the public health ecosystem, (2) using CDC systems for ongoing data modernization, and (3) adopting new standards and approaches for public health reporting.

Alison Cernich, Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD), said that NICHD, as well as NIH more broadly, are heavily invested in ensuring that research incorporates PCOR to the greatest extent possible, and that data gathered through their research are available to other researchers for further analyses. She pointed out that NIH issued a new policy on data sharing for NIH-funded research.⁸ The policy will go into effect on January 25, 2023, and it is based on an understanding that sharing scientific data accelerates biomedical research discovery by enabling validation of research results, providing accessibility to high-value datasets, and promoting data reuse for future studies. The policy is intended to ensure that there is a plan in place for the sharing of data from NIH-funded research. It also specifies allowable costs for data management and sharing, and discusses ways of selecting a repository for the data.

Cernich noted that NIH supports several domain-specific data repositories that are open for both submitting and accessing data. An example is the NICHD Data and Specimen Hub, which allows clinical research data, including PROs, to be posted and shared. NICHD also funds the Data Sharing for Demographic Research infrastructure, which provides curation and archiving services for data relevant to health policy and health systems research as well as broader demographic research.

Concerning barriers, Cernich said that data sharing might be limited by either the parameters of the initial consent provided by the participant or based on NIH policies aimed at protecting the individual. Data sharing has to be balanced with these considerations as a first step. Beyond that, she said, it is important to ensure that data that are shared comply with guiding principles, such as the FAIR data principles and the Transparency, Responsibility, User focus, Sustainability, and Technology (TRUST) data repository principles.

Cernich noted that the COVID-19 pandemic accelerated the need for data that are based on common standards and can quickly be aggregated to produce new knowledge. Tools and resources available for use in public health emergencies and disasters through the Disaster Research Response

___________________

⁸https://grants.nih.gov/grants/guide/notice-files/NOT-OD-21-013.html.

Resource were useful to address this need initially. The PhenX Toolkit was another resource of data standards, clinical report forms, measurement protocols, and survey instruments, some of which included patient-centered outcome variables, such as life changes, household events, and overall impact.

NICHD has also been working on pointing the community toward standard data models, such as the Observational Medical Outcomes Partnership (OMOP) or HL7 FHIR. Cernich said that adopting standard data models can be challenging, because not all of them can accommodate every type of data, but NICHD has been encouraging the use of these standards. The agency has also been developing common data elements where those are lacking. For example, early in the pandemic, NICHD convened a group of stakeholders to specify common data elements for pregnancy as part of a large-scale study to look at the effects of COVID-19 in the context of pregnancy. These data elements were intended to be generalizable to other studies of pregnancy so they could be harmonized and aggregated for more robust analyses.

Cernich also discussed the Gabriella Miller Kids First Data Resource, which is supported by NICHD in collaboration with the National Heart, Lung, and Blood Institute; the National Cancer Institute; and the Common Fund. This resource will use FHIR standards to integrate EHR data, and it is also expanding to include genomic data on conditions in children.

Another NICHD collaboration with ONC, CDC, and the National Center for Health Statistics (NCHS) is focused on developing an HL7 FHIR implementation guide for maternal health, which will specify models for using EHR data standards to identify individual women and their individual pregnancies over time and across health systems. NICHD would also like to be able to link the records of parents and their children. These linkages would make it possible to monitor the impact of emerging public health concerns, such as infectious diseases, to examine the adverse effects of commonly used medications during pregnancy and postpartum on the pregnant person and the infant, and to determine predictive models that can help address various inequities such as inequalities in maternal morbidity and mortality. Cernich noted that NICHD is leading a consortium of projects focused on maternal health to enable coordination and collaboration within this portfolio through the PCOR Trust Fund.

NIH was a pioneer in developing PCOR measures, Cernich said, through projects such as the PROMIS Initiative and the Quality-of-Life Initiative, but she argued that there is a need to continue to refine and implement standard measures of patient perception and evolve with technology. She highlighted the All of Us project, which includes participants in developing some of the project’s modules and is also integrating wearables and other data sources to describe the group of people who are participating.

Cernich pointed to a need for new linkages for data within HHS. For example, NIH and NCHS reached an agreement on the use of data from the National Death Index for research, but other data on vital statistics remain difficult to integrate because of costs and challenges associated with establishing these types of agreements. She also said that access to HHS data needs to be expanded for NIH intramural investigators and that opportunities for doing this exist.

Jacob Kean, Salt Lake City Veterans Affairs Health Care System in the Department of Veterans Affairs (VA), and University of Utah, began by discussing the Department of Defense (DoD) and Department of Veterans Affairs Infrastructure for Clinical Intelligence (DaVINCI) as an example of notable progress on infrastructure for PCOR over the past 5 to 10 years. DaVINCI is a data warehouse and analytic platform that combines DoD and VA health care data, and it serves as an interface between the systems supporting the EHRs at the two agencies. The project began in 2014 and has been building on early successes to achieve scale. To date, DaVINCI has supported more than 60 operations and research projects, and transferring data between the two agencies is a key aspect of this.

Kean discussed lessons learned in several areas, including governance and compliance, data standardization, data quality, education, and partnerships. In the area of governance and regulatory compliance, DoD and VA explored several potential collaborative project governance options and resources for the navigation of regulatory policy, before they settled on an arrangement that was suitable for the DaVINCI project. Kean underscored the time and effort needed to complete this process and noted that establishing partnerships between federal agencies and other entities, such as states or private organizations, can be even more challenging.

In terms of data standardization, DaVINCI uses the OMOP Common Data Model, which Kean described as an enabler and an accelerator. For example, without the OMOP, a data user would need to look at separate data sources for inpatient, outpatient, and surgical encounters, and possibly look at seven or eight different tables for information on medications. OMOP provides researchers with an advantage because they can approach the dataset with an understanding of its structure. While a complete alignment is probably impossible, HHS efforts to harmonize standards are helping remove the barriers posed by the lack of standardization. Kean also underscored the value of HHS support for the development of open-source tools around agreed-upon standards and data models so that the consistency of the work around clinical data improves and also reduces the barrier to entry.

DaVINCI has maintained a focus on data quality throughout project development and execution. This facilitates the use of the data, for example by providing information on fitness for use. Kean said that among the most

effective DaVINCI efforts so far are those related to education. Educational outreach targets current and potential data users and focuses on the structure and provenance of the data. He encouraged HHS to continue to develop and enhance existing educational efforts on the appropriate management and use of data resources.

Kean also discussed challenges and opportunities associated with linked data, including both public and private data. The past few years have witnessed great advances in PPRL technologies and in the scaling of these technologies in a variety of contexts. These advances are happening in parallel with a new set of challenges. One challenge is that there are now thousands of datasets available, and it is difficult to gain a comprehensive understanding of this ecosystem. Kean said that HHS could assist with these efforts by promoting standards, standards education, and tool development. These standards and tools can help with a comparison of similar datasets and the appreciation of strengths and limitations.

Another challenge is related to the costs of linking datasets on a large scale. Kean noted that it is difficult to assign value to the different datasets, but HHS could consider cloud-hosting solutions that enable research and make it possible to shift some of the work from a project level to an agency level. For instance, a PCOR sponsor could sponsor the cost of many data sources and allow others to access the data.

In the future, technology solutions such as blockchain technologies could greatly advance the PCOR data infrastructure. Inherent in every blockchain technology is a distributed ledger, which documents all manipulation and use of data and helps to prevent any unauthorized use. Moreover, these technologies promote federated learning models, which eliminate data transfer. Blockchain technologies could promote the assignment of value to data through combinations of data attributes and data provenance. Furthermore, these technologies could enable data self-sovereignty, which may be a pinnacle of patient centeredness, because it would allow individuals to control access to and use of their data. Kean encouraged HHS to promote regulatory and policy solutions around these transformative technologies.

HHS can play a role in helping to develop and promote use cases, according to Kean. One such use case would be a situation where decentralized models are superior to centralized models for regulatory security and other reasons. Another one could be a situation where trust between entities is a paramount concern. Kean said that the DaVINCI project shows the feasibility of building interoperable data capacity. To date, HHS has played an essential role and could in the future prioritize efforts to advance regulatory guidance, align standards, and foster knowledge of the appropriate use of the growing PCOR ecosystem.

DISCUSSION

During the workshop, the formal presentations were followed by additional discussion among the workshop participants, including the speakers, committee, and audience members. Among the topics that were explored in further detail were the need to harmonize data collections, and data elements in particular, across the federal government. The importance of increasing the adoption of standards and leveraging new opportunities for collaboration made possible by standardized health information technology was also highlighted as part of the discussion. Speakers expressed a desire for coordinating efforts to increase the interoperability of data systems, which would serve as a foundation for the scalability of common approaches in areas such as informed consent.

Speakers acknowledged that increasing the consistency and the use of standards will be a slow and gradual process. In some cases, it might be necessary to accept that heterogeneity exists and consider ways of working within those parameters. It is also important to note that the existing datasets tend to focus on specific populations that are only a subset of the population as a whole and might differ from the overall population on a variety of dimensions.

The discussion also echoed conversations about the challenges resulting from the differing data collection goals in clinical care and research contexts. A related issue is the burden placed on providers and patients who are asked to provide the data. To mitigate these challenges, more clarity is needed about the potential uses and value of the data.

Participants also discussed the usefulness of data that originate from sources other than the context of patient care (such as population surveys) and the potential conceptual limitations that result from focusing on the patient, rather than the individual, regardless of whether the person has a disease, diagnosis, or interaction with a health care provider. This discussion echoed the conversations and committee conclusions that emerged from the first workshop in this series.⁹

Another theme that emerged centered on the challenge of prioritizing projects, given the complexity and broad scope of the PCOR data infrastructure. Priorities are set by legislation or by the agencies, and there are mechanisms for input from committees and workgroups. However, the discussion also made it clear that awareness about the data infrastructure projects is limited among external stakeholders and end users.

___________________

⁹https://www.nap.edu/catalog/26297/building-data-capacity-for-patient-centered-outcomes-research-interim-report.

CONCLUSIONS

The session with federal agency representatives highlighted several key areas where additional work, and in particular collaborative work, is especially needed to continue to build and strengthen the PCOR data infrastructure.

CONCLUSION 2-1: Collaboration among federal agencies and between federal agencies and other partners (such as states, patient groups, and others) is essential for continuing to build the patient-centered outcomes research data infrastructure. The areas where additional collaboration would be particularly useful include the following:

• Increasing consistency in the use of standards for data interoperability and element definitions;
• Addressing barriers that hinder data linkages, such as the limitations associated with health identifiers and mitigating potential selection biases resulting from linkage error;
• Balancing the burden of the data collections and disclosure risks with the value of the datasets;
• Communicating the usefulness of the data collections to those who are asked to provide data about themselves and those who collect the data;
• Promoting discussion and education about fitness for use of the data; and
• Working with stakeholders and patients to promote sharing of data.

While there is frequent collaboration among HHS partners on PCOR data infrastructure work, and the Office of the Assistant Secretary for Planning and Evaluation’s public website contains a comprehensive list of past and current projects funded from the PCOR Trust Fund, additional dissemination efforts focused on external stakeholders could further increase the usefulness of these investments.

CONCLUSION 2-2: There is a need to increase awareness among all stakeholders about new data infrastructure developments funded by the Patient-Centered Outcomes Research Trust Fund. Increased awareness will enhance the efficiency and effectiveness of research, which will increase the impact of the investments made in infrastructure development.

__________________

3

State-Level Data and Collaborations

Lynn Blewett of the State Health Access Data Assistance Center (SHADAC), which is located at the University of Minnesota, noted that SHADAC works with states to leverage state and federal data in the interest of informing policy. Blewett said that large federal data projects using electronic health records (EHRs) and linked data are critical to patient outcomes research, but the time lags in the availability of these data make the results less actionable for state health policy. State analysts are dependent on federal agencies for data access, and congressional objectives may trump state needs. For these reasons and others, local data collaboratives informed by communities of patients, providers, and payers are key to informing state health policy. These data collaboratives make timely and targeted data projects possible, resulting in information that is focused on state needs and priorities and that state policy makers can use.

Blewett discussed several examples of state initiatives. The first example she discussed was the All-Payer Claims Data (APCD) databases. These databases collect and harmonize claims data from public and private payers and include patient demographics and provider codes as well as clinical, financial, and utilization information. The purposes and mechanisms enabling these databases vary across states. The primary objectives are to better understand the financing of health care at the state level, to inform state health reform activities, and to evaluate the outcomes of state reform strategies. The APCD Council provides a forum for states implementing APCDs to share information, expertise, and insight on their development

and use. Blewett said that 23 states have all-payer claims databases, and 6 more are being implemented.¹

Advantages of APCDs include the following:

• Cover the majority of residents in each state;
• Include geographic representation;
• Capture longitudinal information on a wide range of individual patients, providers, and payers;
• Offer comprehensive utilization and spending data at the state level;
• Are mandated by state legislation; and
• Receive federal funding through various initiatives.

Challenges among APCDs include the following:

• Data access for researchers varies by state.
• They provide no data on use of services by the uninsured.
• States cannot require a self-funded insurer and its third-party administrator to share claims data with a state APCD.
• They lack standardization of encounter-level claims from capitated health plans.

Another example discussed by Blewett was a voluntary local health system collaboration, the Minnesota EHR consortium COVID-19 project. There are 11 health systems that are voluntarily participating in this consortium to provide public health surveillance data in close to real time for decision makers. While discussions have been ongoing related to a variety of diseases, the collaboration quickly materialized at the beginning of the pandemic. Blewett said that no patient-level data are shared between systems. Vaccination information is reported by the state and then linked to participating EHR systems; summary data are aggregated at a central site. The project captures about 90 percent of the initial 1.5 million first and second vaccine doses administered in the state.

The following are what has worked in the data sharing consortium:

• Skilled and innovative researchers are embedded in the health systems.
• A distributed data network model is followed (avoids concerns about data privacy and simplifies data use agreements).

___________________

¹https://www.apcdcouncil.org/state/map.

Challenges in the data-sharing consortium include the following:

• There is interest in adding smaller independent clinics and Federally Qualified Health Centers, but that is more difficult and it is costly to build up infrastructure for data submissions.
• Race/ethnicity data need improvement.
• The consortium needs sustainable funding.
• There is a need for improving communications, engagement, and dissemination.

Blewett also discussed the Medicaid Outcomes Distributed Research Network (MODRN), a collaboration to analyze Medicaid data across multiple states to facilitate learning among Medicaid agencies. Participants include AcademyHealth’s State-University Partnership Learning Network and the Medicaid Medical Director Network. This distributed data network allows states to retain their own data and analytic capacity while being able to compare their outcomes data to those from other states. As part of this initiative, 11 university–state partnerships now participate to provide a comprehensive assessment of Medicaid treatment quality in addressing opioid use disorders.

The following are what has worked in MODRN:

• Distributed data network model (avoids concerns about data privacy and simplifies data use agreements);
• Engagement of local universities that have analytic expertise with state Medicaid analysts; and
• Collaboration around policy priorities and closer to real-time analysis.

Challenges with MODRN include the following:

• State participation is limited.
• Data sharing agreements and data use agreements are still required for university-based research access to data files unless all of the analysis is run by the state.
• Financing is needed to support sustainability of network/models.

Blewett offered four overall conclusions based on her experiences: (1) Locally based collaborations that are close to policy makers and decision makers are more feasible and more actionable for state health policy; (2) State regulatory requirements can be leveraged to facilitate data collection and then develop infrastructure for research capacity; (3) Collaborative distributed data networks with motivated and interested researchers

embedded within health systems and public agencies can lead efforts to support targeted data and analytic needs; and (4) Federal financing of local models can be used to inform other activities across the states.

She added that SHADAC collaborates with federal partners to obtain state-level data from the National Health Interview Survey, but it is a “heavy lift.” Among the challenges, she listed the need to have analysts who know how to interact with the National Center for Health Statistics and have special sworn status (from the Census Bureau), and the need for a new proposal every year.

Marsha Lillie-Blanton, George Washington University, focused her remarks related to state-level data and collaborations on broadening the concept of equity to include equity across states. She noted that the policies, practices, and characteristics of geopolitical areas, such as states, matter in the efforts to improve access and quality and to achieve person-centered care.

While the federal role has increased, states continue to be the main drivers of coverage and care for low-income population groups, with Medicaid being the major player in the landscape. Lillie-Blanton pointed out that the states that decided not to expand Medicaid during the early part of the expansion were disproportionately southern states, which have large Black or African American and Hispanic or Latinx populations. This illustrates how states can become drivers of inequities in access to quality care and person-centered care.

Lillie-Blanton discussed the Nationwide Adult Medicaid Consumer Assessment of Healthcare Providers and Systems Survey (NAM CAHPS), a survey that she worked on while she was at the Centers for Medicare & Medicaid Services.² A nationally representative survey of adult Medicaid recipients with state-specific samples, NAM CAHPS is a collaboration among several federal partners as well as 46 states and DC. The data produced include state-specific NAM CAHPS files, which states can get access to on the basis of a data-use agreement. Lillie-Blanton echoed Blewett’s comment about frequent lags in making these types of data available to the states.

Considering both challenges and opportunities associated with collaborations of this type, Lillie-Blanton highlighted the following as some of the areas that need attention:

• Aligning priorities: Both federal and state partners need to identify the data collection effort as a priority.
• Cost issues: Funding needs to be allocated for this type of data collection and analysis.

___________________

²https://www.medicaid.gov/medicaid/quality-of-care/quality-of-care-performance-measure-ment/nationwide-adult-medicaid-consumer-assessment-of-healthcare-providers-and-systems/index.html.

• Longitudinal data: While the baseline data have value, ongoing data collection (even if only every 3 to 5 years) is needed.
• Methodological issues: Comparative analysis across states requires adjustments for state variations in variables that may be unmeasured or not well-measured.
• Data linking: Future federal/state Medicaid surveys will need to include permission in the consumer consent form to link personally identifiable information.

Lillie-Blanton underscored the need for developing partnerships based on trust. Some of the potential partners for federal agencies include Medicaid Agencies, the Medicaid Medical Directors Network, and Public Health and Behavioral Health Agencies. Other stakeholders to consider are professional associations, clinicians and provider groups, advocacy groups, and consumer groups. Academic institutions, policy research organizations, and foundations could also serve as partners.

In terms of building data capacity, two areas emphasized by Lillie-Blanton are (1) supporting the development of state Medicaid infrastructure for data collection, analysis, and reporting; and (2) developing training opportunities and funding for researchers for collecting and analyzing Medicaid data. She noted that some state Medicaid agencies already have strong infrastructures for data collection (e.g., MA, MI, NY, AL) or partnerships with academic institutions (e.g., PA, AR).

Todd Gilmer, University of California, San Diego, made three key points about state-level data and collaborations. First, state-level data are useful for understanding health disparities, because lower-income individuals and families, including those with a significant disability, are underrepresented in many national and commercial datasets, while Medicaid data can provide comprehensive coverage on diverse populations. Second, state-level data can be challenging to work with and to acquire. The learning curves can be steep due to complex and bureaucratic systems, and a path for accessing protected data is not always clear. Furthermore, a high prevalence of Medicaid managed care may result in uneven data quality. Due to these challenges, long-term collaborations can facilitate the interpretation of and access to data. Third, promising opportunities also exist at the county level.

Among the types of data available at the state level, Gilmer highlighted the following:

• Medicaid data:
  • Provide information on health insurance coverage for low-income families and individuals (in expansion states), low-income elderly, and people with disabilities; and

• • Cover a racially/ethnically diverse population with complex health conditions.
• Inpatient and emergency department discharge data:
  • Are available in most states;
  • In California, are provided by the Office of Statewide Health Planning and Development; and
  • Are aggregated by the Agency for Healthcare Research and Quality (AHRQ), which maintains a national database Healthcare Cost and Utilization Project (HCUP) for these data, although with less detail than is available in the state data.
• State-level surveys and indices:
  • Can provide local-area information;
  • Have a greater focus on health disparities compared to national surveys and are potentially more customizable; and
  • In California, are derived from the California Health Interview Survey and the Healthy Places Index.

Gilmer highlighted Medicaid datasets as especially valuable. These datasets have fairly comprehensive coverage, including on medical care and pharmaceuticals. They also have good coverage of mental health and substance use care, particularly for those who are in high need of these services. The datasets also cover home- and community-based services and custodial long-term care. Due to these characteristics, the Medicaid datasets provide a unique platform for studying special populations and topics of interest.

Medicaid programs are uniquely innovative, in Gilmer’s view. Among the innovations, he highlighted the comprehensive, statewide multipayer delivery system and payment reforms that reward value as opposed to volume and support improvements in population health delivery system and payment reform. He also noted that Medicaid provides integrated services for people with complex needs, such as high-risk children and youth; adults eligible for Medicare and Medicaid, including those with long-term care needs; and people with complex physical health, behavioral health, and social service needs. Gilmer also highlighted experimentation with alternative delivery strategies, such as the use of community health workers to build health literacy and peer providers with lived experience to increase engagement in health care.

Gilmer described the Transformed Medicaid Statistical Information System (T-MSIS), which aims to provide Medicaid data on all U.S. states and territories in a more timely way than the previous information system. While there have been some delays in implementing T-MSIS, all states are now reporting data, so it is now starting to become possible to do the types of analyses discussed above. There are also efforts underway to improve data quality, targeting 21 indicators, such as reasonableness of eligible counts, beneficiary demographics, and completeness of key claims service data elements.

Gilmer also highlighted several challenges associated with using state-level data:

• Data access:
  • Each state will have a unique process and Institutional Review Board (IRB) requirements to access data.
  • Data are fragmented, and memoranda of agreement are required for multiple systems.
  • A significant investment is needed for a single study.
• Data quality:
  • Each state dataset will have some unique characteristics.
  • The high prevalence of managed care may affect data quality.
  • There are limitations associated with data on race/ethnicity, language, and sexual orientation.

State data access and quality would benefit from long-term collaborations and investment, Gilmer argued. This would mean maintenance of merged datasets, a standardized process for data access, and a shared understanding of data elements.

Gilmer also briefly discussed examples of data available at the county level. There are many large counties in California, and elsewhere, and they often manage some parts of the health care system. For example, in California mental health and substance use services are managed at the county level. Gilmer said that county-level data may have more detail than data at the state level. For example, San Diego county records include detailed data on race, ethnicity, language, and sexual orientation. When these records are combined at the state level, some of the details are lost due to missing data. It is also important to note that counties provide other social and public safety services, and linking to those datasets presents additional opportunities.

Claudia Steiner, Kaiser Permanente Colorado, discussed datasets produced by AHRQ based on inpatient, emergency department, and ambulatory surgery discharge data. Discharge data are available in all states except Alabama. AHRQ creates five nationwide databases, using a sampling technique that allows estimation to the nation. The nationwide databases are the National (Nationwide) Inpatient Sample; the Kids’ Inpatient Database; the Nationwide Ambulatory Surgery Database; the Nationwide Emergency Department Database; and the Nationwide Readmissions Database. AHRQ also creates three statewide databases, available for some states that allow the distribution of the data: the State Inpatient Databases; the State Ambulatory Surgery Databases; and the State Emergency Department Databases.

The state-level databases include some identifiers that make it possible to link to other databases. Identifiers could include hospital identifiers, encrypted physician identifiers, patient state/county Federal Information Processing Standards codes, and patient zip codes. In some cases, for some states, AHRQ can link additional data at the hospital, physician, or patient level, with permission from the state data organization. State data organizations often have more restricted and fully identified data, and they can perform linkages to additional restricted data available within the state. Examples include linking to birth or death certificates, state-level surveys, patient-reported outcomes, and social determinants of health data collections. Steiner noted that AHRQ is currently actively exploring links to social determinants of health data as well as physician practice variables for Medical Expenditure Panel survey and HCUP data.

Steiner echoed a point made by other speakers, namely that access to state-level data can be challenging because each state has its own process and IRB requirements for accessing the data. She added that the AHRQ HCUP supports a consistent approach to accessing state-level data, and costs are mitigated in many cases. Unique characteristics of the data in some of the states was another challenge discussed by Steiner. AHRQ standardizes the data across all states with a consistently defined set of variables, so state data access and quality have benefited from long-term collaborations with AHRQ. Steiner argued that additional funding and collaboration across federal agencies and within the state partnerships could yield additional value and versatility.

DISCUSSION

The brief discussion that followed the presentations further highlighted the inconsistencies in the availability and quality of the data produced by the states. Comments echoed prior observations that the data collected are not necessarily collected for research purposes and research considerations might not be a priority. For example, while zip-code information is useful, the purpose of zip codes is primarily administrative, which often does not represent the “on the ground” characteristics of an area. In addition, zip codes are sometimes changed. Speakers highlighted the need to develop partnerships built on mutual trust and benefit and to support state data collection systems and analytic efforts.

Participants discussed the additional time needed to aggregate state data at the national level. The delays affect some types of data more than others, and the extent to which having up-to-date data is necessary also varies by the type of data or question, but increased consistency among the states and automation could reduce the time necessary to produce national datasets.

CONCLUSIONS

Many states have robust data collection systems and can produce information that is useful to state and local policy makers. State-generated data are also valuable at the national level, including for answering broader questions about issues that may be influenced by local policy, such as health care access and disparities.

CONCLUSION 3-1: There are opportunities to learn from what states have accomplished in building data capacity.

The data collected, their quality, and ease of access all vary by state. Challenges associated with access, ranging from how the data are stored to the processes involved in accessing them, make the use of state-generated data for research at the national level particularly difficult. The lack of standardization and lag times in data availability present additional challenges.

CONCLUSION 3-2: The usefulness of data available for patient-centered outcomes research could be increased by the sharing and adoption of best practices among the states for the data collected, their quality, and ease of access.

This page intentionally left blank.

4

Clinical Trial Networks and Collaborations

Ruth Carlos, University of Michigan, said that two prime organizations that generate large volumes of clinical trials data are those within the National Cancer Institute (NCI), National Clinical Trials Network (NCTN), and the NCI Community Oncology Research Program (NCORP). Figure 4-1 shows the structure of the NCTN. The five core NCTN research bases conduct therapeutic and cancer control and outcomes research with imaging-based screening and diagnostic trials housed within ECOG-ACRIN (formed as a merger between the Eastern Cooperative Oncology Group [ECOG] for cancer therapy and the American College of Radiology Imaging Network [ACRIN] for cancer imaging). These research bases individually host extensive multidimensional data from their clinical trials. Academic centers and community oncology practices can participate in these clinical trials only through specific base affiliation, and each practice can belong to multiple bases. Carlos noted that no data are routinely shared across research bases.

Two additional research bases conduct only cancer control and outcomes research, such as research on symptom science, patient-reported outcomes, and cancer care delivery, through the NCORP with more than 1,000 practices throughout the United States. Carlos described the NCORP as a valuable setting for cancer clinical trials, because 80 percent of cancer patients receive their treatment in community oncology practices.

Carlos said that while the ECOG-ACRIN datasets are small compared to the national and state datasets discussed, they contain a lot more clinical information, including data on therapeutics, clinical outcomes, potential adverse events, treatment tolerability, treatment adherence, and survival.

Some of the recent studies included the MATCH trial, the National Lung Screening Trial, and the Tomosynthesis and Mammographic Imaging Screening Trial. In addition to these large trials, they also conduct a variety of trials on cancer care delivery, such as an observational trial on financial toxicity (financial distress), an intervention trial on remote delivery of smoking cessation, and a trial on guideline-concordant optimization (de-implementation) of care. Carlos noted that the goal of many of the trials that collect patient-reported outcomes, such as those concerning treatment tolerability, adherence, or quality of life, is to produce information that is actionable and allows clinicians to make decisions on altering, modifying, or otherwise providing supportive care for their patients.

Health equity has also been a particular emphasis within ECOG-ACRIN, with data on ancestry and race, insurance and access, neighborhood deprivation, stress/physiologic dysregulation (allostatic load), and outcome disparities. Carlos and her colleagues conducted retrospective analyses of some of the data from prior trials and noted the absence of information that reflects contemporary thinking about the topics of health equity, structural racism, and discrimination. Addressing an earlier discussion about the collection of genomic data, she argued that it is important to capture both race and ancestry, because race data can provide information on the phenotypic risk of the experience of discrimination, while ancestry may provide information on biological risk.

Carlos noted that there are opportunities for building data capacity within all of these streams of work. In the case of cancer care delivery trials, it becomes important to understand both the clinic- and system-level characteristics and practices, and capture that information in a way that makes it possible to analyze patient outcomes within specific practice types. With patient-reported outcomes, the challenge is to develop ways to relay the information back to the clinician in a manner that ensures that it is received and is actionable. In the area of health equity, capacity could be enhanced by capturing evolving patient-specific insurance design features, capturing ZIP+4 as part of the address information and translating that into measures of structural inequity, and decreasing barriers to data extraction from electronic medical records to be able to obtain data such as allostatic load.

Carlos said that the complexity of the data types and location, as well as the need for equity, transparency, and regulatory compliance, underpinned by strong ethical principles in collection, access, and use, can rapidly seem daunting. This highlights the need to choose actionable potential targets for phased implementation that will ultimately expand data capacity for patient-centered outcomes research (PCOR).

The potential and the challenge of building data capacity for PCOR, according to Carlos, are highlighted by the “four Vs” of big data: volume, variety, velocity, and veracity. She argued that a fifth V worth adding is

value, to characterize data in the context of priorities, such as that of providing actionable information.

Laura Esserman, University of California, San Francisco, said that electronic health records (EHRs) can play a role in accomplishing the goals laid out by previous speakers. EHRs help with organizing information in one place, bill for services, and keep orders and messages collated. Esserman said that learning health systems require another layer of functionality. She explained that EHRs, as they are usually designed and used, do not facilitate the reuse of data for multiple purposes in real time; rather, their unstructured nature makes it difficult to share data, tools, and processes across institutions. Additionally, she said the prevalence of unstructured data in EHRs makes it difficult to use those data for decision support and quality improvement in clinical settings. She noted that current versions of EHRs do not support registries or trials, although they could.

Esserman argued that to realize the vision of shared data, it is necessary to reimagine the process of generating clinical data. As an example, she discussed her work on the OneSource project in collaboration with the UCSF-Stanford Center of Excellence in Regulatory Science and Innovation.

Esserman noted that one challenge associated with how data are captured in EHRs is that the notes that are produced are unstructured and can be contradictory. However, it is possible to imagine a more integrated approach, she said, one where each clinician is responsible for capturing the pieces of data that were important to them in a structured format. Such an approach could contribute high-quality information to “a single source of truth” that could be consistently used for secondary purposes.

After examining source data from EHRs, Esserman and colleagues concluded that there is a disconnect between the data needed for clinical research and what clinicians record in their notes. She explained that EHR data could be more useful to health care providers, patients, and clinical investigators if a system was developed that focused on what data are already captured by clinicians, what is needed beyond that, and a way to integrate that into the clinical workflow. These considerations led to the idea of the OneSource checklist. Esserman noted that the goals of the checklist are to focus on data that are truly essential, determine when they need to be collected, and facilitate creation of a workflow that allows teams to work together to collect high-quality data. She said that the checklist could result in structured data that could be entered once over the course of care but used for multiple purposes.

As an example, Esserman discussed the I-SPY COVID-19 trial. For this study, Esserman and her colleagues began to think about workflow requirements and streamlining data capture before they opened the trial for enrollment. They developed a daily standardized checklist, which is integrated with the EHR, and includes items that clinicians wanted to

capture routinely. The system automates the capture of demographics, medications, and laboratory results. It also supports decisions for both clinical care and research and sends daily checklist and trial reports back to the EHR system. Working groups, such as the safety working group, can easily access information on adverse events that have been reported. Researchers involved with the trial can track what is happening at every site, which makes it possible not only to troubleshoot but to keep the data clean as well. Esserman said the approach is generalizable across sites. It can be built once and then easily integrated into an existing system without an additional major investment.

Esserman noted that in the case of platform trials that can run several studies, centralized agreements can increase quality and efficiency, as well as facilitate collaboration around common approaches to data collection. A system such as OneSource can greatly simplify the workflow and processes and can generate data with the power to change practice. She argued that clinical research is just a special case of clinical care. In systems like OneSource, Esserman said, clinical care teams assemble essential data that support decisions, and by making the clinical trial summaries visible to clinicians, they make it possible to create more disciplined data collection in the clinic setting. That, in turn, improves the process for all patients.

Lesley Curtis, Duke University, began by describing two organizations that she has worked with, the National Institutes of Health (NIH) Healthcare Systems Collaboratory and the Patient-Centered Outcomes Research Institute’s Patient-Centered Outcomes Research Network (PCORnet). The NIH Healthcare Systems Collaboratory’s goal is to strengthen national capacity to implement large-scale cost-effective studies that engage health care delivery systems as research partners for clinical trials. Curtis noted that the coordinating center that she co-leads is involved with pragmatic trial demonstration projects designed to identify best practices and develop general knowledge and resources that are then made available to the research community. She said that the demonstration projects are required to make use of routinely collected EHR data. Curtis described PCORnet as a network of eight large clinical research networks that work together to answer clinical questions by using EHR data that are routinely refreshed, curated, and made accessible through a distributed research network.

Curtis next discussed some challenges and opportunities for improving the PCOR data ecosystem. She pointed out that complete outcomes data are essential for randomized trials. She also noted that the process of acquiring complete outcomes data requires negotiating several individual project-specific and site-specific data-sharing agreements and licenses. Obtaining complete data is often a cumbersome process, she said, because most potential research participants do not receive their care in a closed integrated delivery system.

Advances in privacy preserving record linkage (PPRL) solutions, Curtis said, have offered opportunities to access rich private data resources. However, in order to use those solutions researchers have to negotiate multiple network licenses, pay annual project fees, and access federal sources separately. Curtis also noted that those network license negotiations are time consuming, and she suggested that it would be beneficial to create a standard license for PPRL for federally funded projects to save time and money.

Echoing other speakers, Curtis said that Centers for Medicare & Medicaid Services (CMS) claims data are very useful for outcomes data and clinical trials. She noted that a significant challenge to using those data in clinical trials is that most clinical trials do not collect a Medicare beneficiary ID or social security number, which makes it difficult to link CMS claims data. She said that an additional challenge to using CMS claims data is that beneficiary IDs or social security numbers are usually stored in areas of EHRs that are separate from clinical data records and are difficult to access even with patient consent. She noted that a PPRL solution that could resolve these issues would be very helpful. Curtis also noted that Medicaid data are useful for PCOR and clinical trials but require researchers to negotiate with each individual state to access those data. She agreed with prior speakers that multistate coalitions of those states willing to share their Medicaid data for research would be beneficial to facilitate improved access.

Curtis argued that she would like to see an efficient and comprehensive PCOR data ecosystem that allowed participants to consent to their health data being used for research and allowed researchers to access all of those data without having to rely on the patient providing consent at multiple sites of care. She said that ideally this could be accomplished by the creation of a national identifier system. She noted that a barrier to such an idea is the current climate of misinformation and disinformation that has impacted patient trust in science and research. She underlined the need to develop strategies for disseminating information that emphasizes the value of research and science and combats misinformation.

Curtis concluded by highlighting the importance of standardized and structured data collection. She noted that increased access to raw U.S. Census data has the potential to be a source for social determinants of health (SDOH) data. However, she said use of those data for PCOR requires SDOH data expertise as well as specific skills for downloading and compiling raw Census data. She suggested that one solution could be to create a common set of important SDOH measures that are available as an extractable data package for researchers. She noted that currently researchers frequently must harmonize, clean, and integrate routinely collected data from multiple institutions to create high-quality research datasets. She

emphasized that research and clinical care would benefit from structured and standardized data capture.

DISCUSSION

A key theme that emerged from the discussion also echoed the conversations from previous sessions about the disconnect between the data collected as part of clinical care and the data needed for research. The discussion surfaced concerns about the burden associated with capturing these data and the lack of incentives. Some of the presentations (discussed above) offered ideas for simplifying and streamlining the process of collecting clinical data, which could potentially make it easier to accommodate the need to collect additional data for research, if carefully designed to consider the implications for the resulting information.

Participants discussed ways of integrating data from sources other than medical records into clinical research studies. This could greatly expand research on SDOH, among other subjects. The need to establish data-use agreements was highlighted as a major challenge for sharing and linking data, especially in the case of collaborations that involve several institutions.

CONCLUSION

The session on clinical trial networks and collaborations illustrated the need for better integration between clinical care and research in ways that align differing interests and are mutually beneficial. Better integration can improve both the data available for patient care and the data needed for research.

CONCLUSION 4-1: Infrastructure investments could enhance the utility of data routinely generated in the course of care for clinical trials.

This page intentionally left blank.

5

Public-Private Partnerships

Atul Butte, University of California, San Francisco (UCSF), discussed his experience within the University of California Health System (UC Health), a health enterprise that aims to become the single accountable care organization for the entire University of California system. UC Health combines 20 health professional schools, including six medical schools, 12 hospitals, and 1,000 care delivery sites.

UC Health has a centralized electronic health record (EHR) database that uses the Observational Medical Outcomes Partnership Common Data Model with data elements that are continuously being harmonized. The database includes EHRs from the six academic health centers: UCSF, UCLA, UC Irvine, UC Davis, UC San Diego, and UC Riverside. The database also includes additional information, such as California regulatory data, pathology and radiology text elements, and death index data.

Butte noted that initially the UC Health data are identifiable, and include geographic location, such as home address, as this information is needed for ensuring and improving the quality of medical care delivered. The address information can be linked to indices such as the Area Deprivation Index, Social Vulnerability Index, and the California Healthy Places Index, and contributes to an emerging understanding of social determinants of health. Butte also pointed out that all University of California academic medical centers make health data accessible to patients following federal standards such as Fast Healthcare Interoperability Resources.

Butte said that the database “pays for itself” because it benefits the health system’s operations by improving the quality of care, decreasing specific unnecessary inpatient drug use, helping with managing costs in the

self-funded health plans, and assisting with centralized population health management. The same database can also be used for research, once the data are de-identified. For example, UC Health recently conducted several COVID-19 related projects in collaboration with the Food and Drug Administration.¹

The challenge highlighted by Butte to undertaking collaborations and data sharing is the competitive health care environment. While the benefits of data sharing are evident, health systems are cautious because the data can also potentially be used against them, leading to revenue loss. As an example, he mentioned Stanford University Medical Center health care workers’ use of data on infection rates as a contract negotiation tactic. Butte argued that despite these concerns, leveraging scale for value (as in the case of UC Health) has clear advantages for health systems. Once a database is developed for operational purposes, the additional cost of making it available for research purposes can be relatively small.

Vincent Mor, Brown University, discussed his experiences with building a public-private data sharing cooperative focused on long-term care. The project was part of the National Institute on Aging (NIA) Imbedded Pragmatic Alzheimer’s Disease and AD-Related Dementias Clinical Trials Collaboratory (IMPACT Collaboratory), which builds capacity to conduct pragmatic clinical trials of interventions embedded within health care systems for people living with dementia and their care partners.

Mor noted that COVID-19 has disproportionately affected long-term care residents, and the need for rich real-time data on nursing homes quickly became evident at the beginning of the pandemic. To respond to this need, Mor and his colleagues partnered with Genesis Health Care to obtain real-time data, and later expanded collaborations with the American Healthcare Association, Acumen LLC, and Exponent to build a data repository involving a broader range of nursing homes that have electronic medical records. Mor said that the database can become useful in monitoring future pandemics and policy changes. The database can also make it possible to selectively recruit facilities to participate in embedded randomized controlled trials of interventions.

Mor said that the initial Brown-Genesis COVID-19 partnership was possible because of prior collaborations that laid the foundation for this work. He echoed Butte’s comments that trust between the partners is essential, particularly when there is a sharing of large volumes of real-time data. In the case of this project, there was also a need to access identifiable data

___________________

¹ R. Vashisht, A. Patel, B.O. Crews, O.B. Garner, L. Dahm, C. Wilson, and A.J. Butte. (2021). Age- and sex-associated variations in the sensitivity of serological tests among individuals infected with SARS-CoV-2, JAMA Network Open 4(2), e210337. doi:10.1001/jamanetworkopen.2021.0337.

to enable linkages to information that was not available from the electronic medical records. The researchers were sensitive to the risks for Genesis and have agreed to provide input focused on solving operational challenges as a priority over publications. However, the project also produced important research that contributed to knowledge on the COVID-19 outbreak.

Ultimately, the success of the initial collaboration led to the expansion of the project to include additional nursing home companies and partners. The resulting nursing home data-sharing collaborative now serves as the basis for several new studies, including randomized controlled trials, an analysis of vaccine effectiveness, and tracking breakthrough infections. Mor noted that the American Health Care Association, an industry association, played an important role in recruiting providers and negotiating agreements.

While researchers at Brown University and others who are part of the IMPACT Collaboratory will be the first users of the data, the data will later become available for other NIA investigators. The data from the electronic medical records can be particularly useful for recruiting facilities for trials and clinical research, while the electronic medical records data linked to claims data can further expand the possibilities to areas such as pharmacoepidemiological research, public health surveillance research, and studies of the impact of treatments or policies.

Marc Overhage, Anthem, shared his perspectives on what makes public-private partnerships work based on his experience in a variety of settings, including academia, an academic medical center, an electronic medical records vendor, and currently a payer (Anthem). He emphasized that it is important to think of opportunities for collaboration as more than just data sharing, and consider the sharing of analytic techniques, computational resources, knowledge, and opportunities to commercialize and create value in other ways.

Overhage said that a typical public-private partnership might involve a collaboration between a government entity and a private organization, with the objective of building infrastructure or other services. The partners share the investment, the risks, and the rewards. He underscored the importance but also the challenge associated with establishing the mutual benefit in a partnership.

Overhage echoed previous comments about the particular concerns for private entities, including the potential disclosure of private, sensitive, or proprietary information; risks to brand reputation; and financial risks. He also highlighted the opportunity cost associated with the time invested in the collaboration. The fragmented nature of regulations, including state and local regulations that might apply to data sharing, also represents a risk. There are also security risks, both real and perceived.

Ethical risks were another area of concern Overhage discussed, specifically those associated with the loss of control over how the data might be used in the future. He said it is important to consider potential public

perception, even if regulations permit data sharing. The public might have particular concerns about sharing data that involves both public and private entities, because their trust in the data being used appropriately has to extend beyond the original interaction. Overhage reiterated that the alignment of interests is especially important in these types of collaborations. He noted that while there is value in data, it is important to keep in mind the roles that domain knowledge, analytical expertise, and computational capabilities can play in partnerships.

The fact that data can be reused in ways that many other types of property cannot be raises additional ownership considerations, Overhage said. For example, analyzing the data in ways that someone else might not have analyzed them does not diminish the data’s usefulness to someone else. This is important to consider, particularly for data that resulted from an effort that was publicly funded.

In summary, Overhage listed five key enablers that he believes can make public-private partnerships work:

• Achieving clear alignment of stakeholder interests at the outset of the partnership;
• Establishing responsible data governance;
• Putting processes in place to ensure that the insights that are created are accurate, unbiased, and, where appropriate, explainable;
• Providing decision makers with the tools, processes, and support to act on the insights resulting from the work; and
• Ensuring the long-term economic sustainability of the partnership.

DISCUSSION

A central theme of the discussions that followed the presentations was the risks associated with data sharing. Speakers noted that resistance to transparency is common among all types of entities. For example, despite recent regulations focused on price transparency, it is still not possible to compare costs across health care providers.

Participants also discussed ways of overcoming resistance to data sharing. A potentially compelling argument could be that there is a wealth of information in the various health datasets, and the organizations that own them do not have the capability to take full advantage of the data. Even in organizations with thousands of analysts, the demand for using the data to answer questions far exceeds the capacity to produce answers. Collaborations with academic groups, other commercial entities, government organizations, and others are necessary to take full advantage of the data, despite short-term risks, but the conversation further highlighted the important role trust plays in establishing collaborations that involve data sharing.

Finally, the challenges associated with navigating the process of establishing agreements were discussed, along with the challenges of coordinating with Institutional Review Boards across multiple institutions. At the same time, participants underscored how collaborations, once they are established, can lower data access barriers for early career researchers.

CONCLUSION

Discussions in this session echoed some of the themes that emerged in prior sessions, including barriers to accessing data. While the benefits of data sharing are clear, the workshop also highlighted some of the reasons behind reluctance to share and underscored the risks involved for the organizations providing the data. Successful data sharing agreements can be established when these factors are taken into consideration.

CONCLUSION 5-1: Successful partnerships across health care systems require participant trust, clear evidence of mutual benefit, and the ability to control risk.

This page intentionally left blank.

6

Collaborations with Patient Groups

Pat Furlong of Parent Project Muscular Dystrophy (PPMD) highlighted the importance of data sharing and dataset linkage for rare disease research. She discussed PPMD’s work to develop a clinical trial master protocol and network for research on Duchenne muscular dystrophy, a rare progressive neuromuscular disease. She noted that other rare diseases face similar challenges related to research, so PPMD’s goal is to create a framework that could be replicated.

Furlong said that several organizations have conducted research studies and clinical trials to investigate treatments for Duchenne. She also noted that several past clinical trials for drugs to treat the disease have failed. Each of those clinical trials, she said, set up a significant data infrastructure, but when the trial failed, the infrastructure was dismantled, and frequently the data collected during the trial were lost. Furlong also noted that the data from failed clinical trials were not integrated into any other datasets. PPMD determined that this was a major barrier for progress and has been working on finding a mechanism to improve data sharing and linking for research.

Another challenge related to Duchenne research highlighted by Furlong was trial participation. Furlong noted that participation in a clinical trial requires time, money, and travel that are not always accessible to the patients and their families. She explained that any or all of those factors play a role in the decision to participate in such a trial. If a clinical trial master protocol were deployed at multiple sites, she suggested, this could increase the number of patients who could participate.

Furlong noted that PPMD’s first action to improve data sharing for research was to work with the Critical Path Institute. In this work they

developed a disease progression model for Duchenne muscular dystrophy, one that offers a central source for those data to be accessed by those involved in research. She explained that one of the data needs in research on this disease concerns its natural history and the effects of delayed treatment, data that could be culled from the placebo arms of clinical trials. PPMD is seeking to address this by creating a centralized database as part of its clinical trial master protocol. A key part of that infrastructure would be a mechanism for linking non-proprietary data from clinical trials that are using the master protocol. PPMD expects that this would result in larger datasets for analysis than are typically available for rare disease research. PPMD is also working on creating systems for standardizing data collection as part of its master protocol. Furlong concluded by emphasizing the importance of data sharing and improving stakeholder access to data for the benefit of patients.

James Lewis, University of Pennsylvania, spoke about his experiences with patient-centered outcomes research (PCOR) data in the context of his research on inflammatory bowel disease (IBD). He described several barriers related to the patient-centered component of PCOR, including the following:

• High expenses and time commitment needed,
• Low patient engagement,
• Reluctance among patients to be active research subjects, and
• Patient concerns about the safety of their data.

Lewis noted that in regard to the outcomes research component of PCOR, a barrier of note is that while outcomes research routinely measures processes or clinical outcomes, patients may have different priorities for information.

Lewis also discussed his experiences with the IBD Plexus program, which he described as a collection of ongoing and historical cohort studies from which they are working to build a centralized data warehouse. This data warehouse will be designed to link patients’ electronic health records (EHR) data, biosample-derived data, and claims data to assist researchers in developing a full picture of the patient’s experience with their disease. Lewis noted that an important component of this process was developing a SmartForm, one that could be integrated into Epic EHRs, to standardize how clinicians record patient data in the EHR. They have also engaged another tool in Epic EHRs that allows patients to report their symptom data for integration into the EHR in a standardized format. IBD Plexus is in the early stages of a study that involves a partnership with the Food and Drug Administration (FDA) to use the FDA MyStudies app to collect patient-reported data.

Lewis cited low data quality as one of the challenges with using data from EHRs and patient-reported data. He said these sources of data can be inaccurate or incomplete due to patients’ misinterpreting or skipping questions or because of clinician time constraints. He noted that recently implemented Centers for Medicare & Medicaid Services rules around charting requirements may impact the amount of EHR data available for PCOR.

Lewis described possible solutions for filling data gaps. One of these solutions is to link EHR data, claims data, and patient-reported data, which allows researchers to develop a more holistic dataset for a patient, including information beyond what is captured by a single health care system. However, Lewis noted, concerns about confidentiality and data security among stakeholders—including patients, insurance companies, and researchers—are a challenge for data linkage. He also noted that some patients are concerned about whether their data are being used appropriately.

Finally, Lewis discussed natural language processing to convert free text that is entered in EHRs as another solution for filling in data gaps. However, he noted that given the large number of variables, that process often requires a significant time investment. Another potential solution he proposed was the creation of a neutral national organization to standardize, de-identify, and link data from major EHRs and major sources of claims data for use in research. Lewis concluded by highlighting the need to recognize the multidisciplinary nature of PCOR and the need for collaboration.

Marc Natter, Harvard University, said that early in his career he encountered difficulties finding longitudinal data for PCOR focused on childhood arthritis due to inadequate registries and data infrastructure. He noted that the Childhood Arthritis & Rheumatology Research Alliance (CARRA) and the CARRA Registry were developed to address this challenge. He explained that in the beginning, CARRA developed a registry that served as a federal data warehouse for data derived from studies of pediatric rheumatology diseases.

Natter said that over time the CARRA Registry evolved into a common data collection platform in addition to a research network. The platform includes a master data-use agreement and protocol that applies to all participating institutions, and to all identified and de-identified data, which flow bi-directionally between research sites, CARRA’s Data Coordinating Center, and clinical centers. Natter explained that CARRA continues to build on its data infrastructure, which has facilitated a variety of pediatric rheumatology PCOR studies. He noted that he and his colleagues have also incorporated data infrastructure to facilitate collection of patient-reported outcome (PRO) data, including an application-based tool. He noted that they continue to research how best to collect and integrate PRO data.

Natter also highlighted some lessons that were learned as the CARRA registry and data platform have evolved. He noted that at some institutions

they have faced challenges obtaining Institutional Review Board (IRB) approval for studies that include a patient-facing application platform. He suggested that the Department of Health and Human Services work to develop a set of universal guidelines for these patient-facing applications. He said that this would provide IRBs with clear standards to apply when evaluating proposals that include such applications. He also echoed points made by others about the need for researchers to communicate with patient and physician stakeholders to ascertain what information matters to them and to use that to guide how PCOR analytic results are presented.

Natter also noted that methods research needs more attention and the outcomes of that research should be incorporated into PCOR data standards. He said that validated disease metrics are needed to facilitate more effective use of advanced machine learning algorithms. He concluded by highlighting that as other speakers had noted, researchers need better access to large aggregated datasets that include claims data in order to conduct more robust PCOR data analysis.

DISCUSSION

The brief discussion after the presentations focused on the role patient organizations are best positioned to play in advancing PCOR. Participants highlighted patient organizations’ ability to facilitate connections between researchers and people (patients and others) who are invested in the patient-centered aspect of PCOR. Participants also noted a potential role for patient organizations in facilitating patients’ access to their own health data, including claims data. This would enable patients to contribute their information to research studies. Patient groups can also play an important role in providing input on outcome measures that matter most to people. Many of these ideas echoed the committee’s conclusions from the first workshop.¹

CONCLUSIONS

Collaborations with patient organizations can help in addressing patient concerns about participating in research studies and in building patient engagement, which are both important for achieving a patient-centered approach. Disease registries directed by patient groups can be a particularly useful additional source of data, providing information that would not be available to researchers otherwise.

___________________

¹https://www.nap.edu/catalog/26297/building-data-capacity-for-patient-centered-outcomes-research-interim-report.

CONCLUSION 6-1: Patient groups can be helpful partners in all aspects of patient-centered outcomes research, including engaging patients in order to improve research participation and the impact of results.

CONCLUSION 6-2: Patient-directed disease registries can be a source of in-depth, longitudinal, prospective clinical and patient-reported data that are not available from other data sources.

This page intentionally left blank.

Appendix A

Biographical Sketches of Committee Members

GEORGE ISHAM (NAM) (Chair) is a senior fellow at the HealthPartners Institute and a senior advisor for the Alliance of Community Health Plans. Previously, he served as a senior advisor to the board of directors and the senior management team of HealthPartners, and prior to that, he was HealthPartners’ medical director and chief health officer, responsible for quality of care and health and health care improvement. He has been active in health policy, serving as a member of the Centers for Disease Control and Prevention’s Task Force on Community Preventive Services, a member of the Agency for Healthcare Research and Quality’s United States Preventive Services Task Force, a founding co-chair of the National Committee for Quality Assurance’s committee on performance measurement, as well as founding co-chair of the National Quality Forum’s Measurement Application Partnership. He has an M.D. from the University of Illinois at Chicago and an M.S. in preventive medicine and administrative medicine from the University of Wisconsin–Madison.

JOHN F.P. BRIDGES is professor and vice chair of academic affairs in the Department of Biomedical Informatics at The Ohio State University (OSU) College of Medicine. He is also a professor in the Department of Surgery and an adjunct professor in both the Division of Epidemiology at the OSU College of Public Health and Department of Health Behavior and Society at the Johns Hopkins Bloomberg School of Public Health. Prior to joining OSU he was on the faculty of the Johns Hopkins Bloomberg School of Public Health, the Department of Tropical Hygiene and Public Health within the University of Heidelberg School of Medicine, and the Department of

Epidemiology and Biostatistics within the Case Western Reserve University School of Medicine. He has previously held positions in the Department of Economics at the Weatherhead School of Management at Case Western Reserve University; the National Bureau of Economic Research; Center for Medicine in the Public Interest; and the Center for Health Economics, Research and Evaluation in Australia. He has a Ph.D. in economics from the City University of New York.

JULIE BYNUM is the Margaret Terpenning Professor of Medicine in the Division of Geriatric Medicine and vice chair for faculty affairs in the Department of Internal Medicine at the University of Michigan. She is also a research professor in the Institute of Gerontology, Geriatric Center Associate Director for Health Policy and Research, and a member of the Institute for Healthcare Policy and Innovation. She currently leads a portfolio of National Institutes of Health–funded research that examines the quality of care, diagnosis, and treatment of people with Alzheimer’s disease and related dementia in the community, nursing homes, and assisted living and is the director of the Center to Accelerate Population Research in Alzhiemer’s. She is currently a member of the National Academies of Sciences, Engineering, and Medicine’s Forum on Aging, Disability, and Independence and was a member of a National Academies committee that authored Vital Signs: Core Metrics for Health and Health Care Progress. She has an M.P.H. from the Johns Hopkins University School of Hygiene & Public Health and an M.D. from the Johns Hopkins University School of Medicine.

ANGELA DOBES is vice president of the Crohn’s & Colitis Foundation’s IBD Plexus Program, a research-information exchange platform designed to centralize data and biosamples from diverse research initiatives to advance science, accelerate precision medicine, and transform the care of inflammatory bowel disease (IBD) patients. She has previously worked for clinical technology and pharmaceutical organizations, where she has led implementation of various technology solutions focused on business optimization and accelerating the delivery of new therapies to patients safely. She is currently serving as principal investigator on a study to enhance engagement, research participation, and collaboration through the IBD Partners Patient Powered Research Network. She has an M.A. in public health from the Icahn School of Medicine at Mount Sinai.

DEBORAH ESTRIN (NAE/NAM) is a professor of computer science at Cornell Tech where she holds the Robert V. Tishman founder’s chair, serves as the associate dean for impact, and is an affiliate faculty at Weill Cornell Medicine. Her research activities include technologies for caregiving, immersive health, small data, participatory sensing, and public

interest technology. Estrin was an Amazon Scholar, and before joining Cornell University she was founding director of the National Science Foundation’s Center for Embedded Networked Sensing at the University of California, Los Angeles, pioneering the development of mobile and wireless systems to collect and analyze real-time data about the physical world. Estrin cofounded the nonprofit startup Open mHealth and has served on several scientific advisory boards for early-stage mobile health startups. She has a Ph.D. in electrical engineering and computer science from the Massachusetts Institute of Technology.

OLUWADAMILOLA FAYANJU is the Helen O. Dickens Presidential Associate Professor of Surgery at the Perelman School of Medicine at the University of Pennsylvania. She is also chief of breast surgery at Penn Medicine. Previously, she was associate professor of surgery and population health sciences in the Duke University School of Medicine and director of the Durham VA Breast Clinic. She was also associate director for Disparities & Value in Healthcare with Duke Forge, Duke University’s center for actionable data science. In 2019, she was recognized by the National Academy of Medicine as an Emerging Leader in Health and Medicine Scholar. She received an M.A. in comparative literature from Harvard University and her M.D. and M.P.H.S. from the Washington University in St. Louis.

CONSTANTINE GATSONIS is the Henry Ledyard Goddard University Professor of Statistical Sciences, director of statistical sciences, and professor of biostatistics at Brown University. He was founding director of the Center for Statistical Sciences and founding chair of the Department of Biostatistics at Brown University. He is a leading authority on the evaluation of diagnostic and screening tests and has made major contributions to the development of methods for medical technology assessment and health services and outcomes research. He is a world leader in methods for applying and synthesizing evidence on diagnostic tests in medicine and is currently developing methods for comparative effectiveness research in diagnosis and prediction and radiomics. Since 2016 he has served as a statistical consultant for the New England Journal of Medicine and was the founding editor-in-chief of Health Services and Outcomes Research Methods. He has a Ph.D. in mathematical statistics from Cornell University.

ROBERT GOERGE is a senior research fellow at Chapin Hall at the University of Chicago. He is also a senior fellow and founder of the Master’s Degree in Computational Analysis in Public Policy at the University of Chicago Harris School of Public Policy. His research is focused on improving the available data and information on children and families, particularly those who require specialized services related to maltreatment, disability,

poverty, or violence. At Chapin Hall, he is principal investigator for the Family Self-Sufficiency Data Center, the Linking Federal Data to Local Data project, and the National Survey for Early Care and Education. He currently serves on the National Academies of Sciences, Engineering, and Medicine’s Committee on National Statistics. He has a Ph.D. in social policy from the University of Chicago.

GEORGE HRIPCSAK (NAM) is the Vivian Beaumont Allen Professor and chair of the Department of Biomedical Informatics at Columbia University. He is also the director of medical informatics services for New York Presbyterian Hospital. He is also a board-certified internist. He led the effort to create the Arden Syntax, a language for representing health knowledge that has become a national standard. As chair of the American Medical Informatics Association Standards Committee, he coordinated the medical informatics community response to the Department of Health and Human Services for the health informatics standards rules under the Health Insurance Portability and Accountability Act of 1996. His current research is on the clinical information stored in electronic health records. Using data mining techniques, he is developing the methods necessary to support clinical research and patient safety initiatives. He has an M.D. and an M.S. in biostatistics from Columbia University.

LISA IEZZONI (NAM) is professor of medicine at Harvard Medical School and the Health Policy Research Center at Massachusetts General Hospital, where she served as director in the past. She was previously co-director of research in the Division of General Medicine and Primary Care at Beth Israel Deaconess Medical Center in Boston. Her research focuses on risk adjustment methods for predicting cost and clinical outcomes of care, and on health care experiences and outcomes of persons with disabilities. She has served on the editorial boards of the Annals of Internal Medicine, the Journal of General Internal Medicine, Health Affairs, Medical Care, Health Services Research, and the Disability and Health Journal, among others. She has an M.D. from Harvard Medical School and an M.Sc. from the Harvard T.H. Chan School of Public Health.

S. CLAIBORNE JOHNSTON (NAM) is the inaugural dean of Dell Medical School, vice president for medical affairs, and the Frank and Charmaine Denius Distinguished Dean’s Chair in medical leadership at The University of Texas at Austin. Previously, Johnston was associate vice chancellor for research at the University of California, San Francisco (UCSF). He also directed the Clinical and Translational Science Institute and founded the UCSF Center for Healthcare Value. His research is focused on clinical trials and health services research in stroke. He is also an expert in medical

education, research administration, health care value, and population health. He has led several large-cohort studies of cerebrovascular disease and three international multicenter randomized trials. He has an M.D. from Harvard Medical School and a Ph.D. in epidemiology from the University of California, Berkeley.

MIGUEL MARINO is an associate professor with joint appointments in the School of Public Health Division of Biostatistics and the Department of Family Medicine at Oregon Health & Science University. His research focuses on the development and implementation of novel statistical methodology to address complexities associated with the use of electronic health records (EHRs) to study changes in policy; using EHRs to study health disparities; validation of EHRs as a reliable source for observational studies; pragmatic randomized trials; and preventive health maintenance. He was selected by the National Academy of Medicine as an Emerging Leader in Health and Medicine Scholar. He has a Ph.D. in biostatistics from Harvard University.

ELIZABETH McGLYNN (NAM) is vice president for Kaiser Permanente Research and executive director for the Center for Effectiveness & Safety Research at Kaiser Permanente. She is also interim senior associate dean for research and scholarships at the Kaiser Permanente Bernard J. Tyson School of Medicine. She is an internationally known expert on methods for evaluating the appropriateness and quality of health care delivery. She has led major initiatives to evaluate health reform options under consideration at the federal and state levels. She is the lead of Kaiser Permanente & Strategic Partners Patient Outcomes Research To Advance Learning (PORTAL) Network. She was a member of the Strategic Framework Board, which provided a blueprint for the National Quality Forum on the development of a national quality measurement and reporting system. She chaired the board of AcademyHealth, served on the board of the American Board of Internal Medicine Foundation, and served on the Board of Providence-Little Company of Mary Hospital Service Area in Southern California. She has a Ph.D. in public policy from RAND Graduate School.

DAVID MELTZER (NAM) is the Fanny L. Pritzker Professor in the Department of Medicine, chief of the section of Hospital Medicine and faculty in the Department of Economics and Harris School of Public Policy at the University of Chicago. He is also director of the Center for Health and the Social Sciences and of the Urban Health Lab at the University of Chicago. His research explores problems in health economics and public policy with a focus on the theoretical foundations of medical cost-effectiveness analysis and the cost and quality of hospital care. Since 1997,

he has developed the inpatient general medicine services at the University of Chicago as a Learning Health Care System to produce knowledge on how to improve the care of hospitalized patients, mobilizing the clinical care process to generate and learn from diverse data from electronic health records, claims data, patient interviews, and bio-specimens on more than 100,000 patients. He is the lead of the University of Chicago network site as part of the Chicago Area Patient-Centered Outcomes Research Network. He has an M.D. and a Ph.D. in economics from the University of Chicago.

PAUL C. TANG (NAM) is an adjunct professor in the Clinical Excellence Research Center at Stanford University and an internist at the Palo Alto Medical Foundation. He was formerly chief innovation and technology officer at the Palo Alto Medical Foundation and vice president, chief health transformation officer at IBM Watson Health. He has more than 25 years of executive leadership experience in health information technology within medical groups, health systems, and corporate settings. He has directed innovation and technology teams in provider organizations, academic institutions, corporate research organizations, and product development organizations. Most recently, he led the creation, development, deployment, and evaluation of the application of artificial intelligence to physician point-of-care solutions integrated within an electronic health record system. He also led a corporate enterprise-wide design team. He has chaired numerous federal and private sector advisory and professional association groups related to health information technology and policy. He received an M.S. in electrical engineering from Stanford University and his M.D. from the University of California, San Francisco.

Appendix B

Workshop Agenda

Building Data Capacity for Patient-Centered Outcomes Research:
An Agenda for 2021 to 2030

Virtual Workshop 3: A Comprehensive Data Ecosystem for
Patient-Centered Outcomes Research

June 14, 2021, 11 am – 5 pm EDT

OBJECTIVES FOR THE WORKSHOP

• Discuss how research and data collaborations can evolve to meet PCOR and data capacity challenges, and how HHS can support effective research and data collaborations
• Identify barriers and potential solutions to the access and use of linked public data, and to the access and use of linked public and private/proprietary data
• Discuss the feasibility and utility of developing a phased-in approach to building the interoperable data capacity for patient-centered outcomes research with existing databases in HHS, other federal departments, and the private sector

This page intentionally left blank.

Appendix C

Biographical Sketches of Workshop Speakers

LYNN BLEWETT is the director of the State Health Access Data Assistance Center (SHADAC), a research and policy center that supports state efforts to monitor and evaluate programs and reforms to increase access to needed health care. She is also a professor of health policy in the School of Public Health, University of Minnesota. Her research includes Medicaid payment reform, the evolving health care safety net, and measures to monitor population health outcomes. She brings expertise in state and federal health data resources, including federal surveys such as the Current Population Survey, the American Community Survey, the National Health Interview Survey (NHIS), and the Behavior Risk Factor Surveillance System, which are all accessible through SHADAC’s interactive online Data Center. She also heads up a project funded by the Robert Wood Johnson Foundation to provide expertise in the use of data analytics to inform and monitor implementation of the Patient Protection and Affordable Care Act and was instrumental in establishing the University of Minnesota-based Census Research Data Center, which focuses on health services research and policy. She is also principal investigator of the Integrated Health Interview Series, a project funded by the Eunice Kennedy Shriver National Institute of Child Health and Human Development to harmonize and integrate more than 50 years of the NHIS and make it accessible through a web portal for academic and policy research. She earned an M.A. degree in public affairs and a doctorate degree in health services research from the University of Minnesota.

ATUL BUTTE (NAM), M.D., Ph.D., is the Priscilla Chan and Mark Zuckerberg Distinguished Professor and inaugural director of the Bakar

Computational Health Sciences Institute at the University of California, San Francisco. He is also the chief data scientist for the entire University of California Health System, the 10th largest health system by revenue in the United States, with 20 health professional schools, six medical schools, six academic health centers, 10 hospitals, and more than 1,000 care delivery sites. He has been continually funded by the National Institutes of Health for 20 years, is an inventor on 24 patents, and has authored more than 200 publications. His research has repeatedly been featured as well in the New York Times, Wall Street Journal, and Wired Magazine. He was elected into the National Academy of Medicine in 2015, and in 2013 he was recognized by the Obama Administration as a White House Champion of Change in Open Science for promoting science through publicly available data. Butte is also a founder of three investor-backed data-driven companies: Personalis (IPO, 2019), which provides medical genome sequencing services; Carmenta (acquired by Progenity in 2015), which discovered diagnostics for pregnancy complications; and NuMedii, which finds new uses for drugs through open molecular data. He trained in computer science at Brown University, worked as a software engineer at Apple and Microsoft, received his medical degree at Brown University, trained in pediatrics and pediatric endocrinology at Children’s Hospital Boston, and then received his Ph.D. from Harvard Medical School and the Massachusetts Institute of Technology.

RUTH CARLOS is a professor of radiology and serves as the assistant chair for clinical research at the University of Michigan. Her work encompasses cost-effectiveness analysis, patient preference measurement, and meta-analysis and systematic reviews in diagnostic imaging. Her research also seeks to understand the effectiveness of maternally directed interventions to improve vaccine uptake in adolescent daughters. She brings her specific expertise in evaluating cultural barriers to adolescent HPV vaccination in African-American mothers and developing and pilot-testing tailored interventions directed at these cultural barriers. She also co-directs the Program on Women’s Health Care Effectiveness Research in the Department of Obstetrics and Gynecology at the University of Michigan Medical School and currently chairs the GE Association of University Radiologists Research Radiology Academic Fellowship, a national program supporting early-stage investigators in health services research and care delivery. She received her medical degree from and completed her diagnostic radiology residency at the University of Chicago, and did a fellowship at the University of Michigan in Ann Arbor, joining the faculty at the University of Michigan in 1998. She also holds a master’s degree from the School of Public Health at the University of Michigan.

ALISON CERNICH is the deputy director of the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD). In

this role, she assists the NICHD director in overseeing the institute’s programs supporting research on child development, developmental biology, nutrition, HIV/AIDS, intellectual and developmental disabilities, population dynamics, reproductive biology, contraception, pregnancy, and medical rehabilitation. Prior to this position, she was the director of NICHD’s National Center for Medical Rehabilitation Research (NCMRR), where she managed a $72 million research portfolio aimed at improving the health and well-being of people with disabilities. As NCMRR director, she led the development and revision of the congressionally mandated National Institutes of Health Research Plan on Rehabilitation, an effort that included coordination with 17 institutes and centers and multiple external stakeholders. Before joining NICHD, she served as deputy director of the Defense Centers of Excellence for Psychological Health and Traumatic Brain Injury at the U.S. Department of Veterans Affairs (VA), where she coordinated prevention, education, research, and clinical care efforts for service members and veterans diagnosed with traumatic brain injury. Prior to her 10 years with the VA, she was the traumatic brain injury liaison to the Department of Defense, the chief of neuropsychology and director of the Polytrauma Support Clinic at the VA Maryland Health Care System, and a funded investigator through the VA Rehabilitation Research and Development Service. She received her doctoral degree in clinical psychology from Fairleigh Dickinson University and completed postdoctoral training in cognitive neuroscience at the National Rehabilitation Hospital in Washington, D.C.

LESLEY CURTIS is professor and chair of the Department of Population Health Sciences at the Duke University School of Medicine. A health services researcher by training, she is an expert in the use of Medicare claims data for health services and clinical outcomes research and a leader in national data quality efforts. She serves as co-principal investigator of the Food and Drug Administration’s (FDA’s) Sentinel Innovation Center, co-investigator of the Data Core for the FDA’s Sentinel Initiative to monitor the safety of FDA-regulated medical products, and chair of the Data Quality Subcommittee for the National Evaluation System for health Technology Coordinating Center, which generates real-world evidence for health technology and medical devices. She also serves as co-investigator of the coordinating center for the Patient-Centered Outcomes Research Institute’s National Clinical Research Network (PCORnet), working with health systems and patient networks to develop a harmonized network infrastructure that leverages health systems and electronic health records data for robust observational and interventional research. She received her Ph.D. from the University of Rochester.

LAURA ESSERMAN is a surgeon and breast cancer oncology specialist practicing at the University of California, San Francisco (UCSF) Breast Care

Center. She directs the UCSF Clinical Breast Cancer Program and co-leads the Breast Oncology Program. She is a professor of surgery and radiology at UCSF as well as a faculty member at the UCSF Helen Diller Family Comprehensive Cancer Center. Her research has focused on tailoring treatment to biology, which requires the integration of translational science, bioinformatics, medical and clinical informatics, systems integration, and clinical care delivery. She has led the I-SPY Trials, a collaboration among the National Cancer Institute, the Food and Drug Administration, more than 28 cancer research centers, major pharmaceutical and biotech companies, and the not-for-profit sponsor, Quantum Leap Healthcare Collaborative. Additionally, she led the creation of the University of California-wide Athena Breast Health Network, a learning system designed to integrate clinical care and research as it follows 150,000 women from screening through treatment and outcomes. As part of the network, she has spearheaded the development of the WISDOM study to learn how to improve breast cancer screening by examining the effectiveness of a personalized screening strategy informed by each woman’s breast cancer risk and preferences, in comparison with the standard of annual screening. She recently collaborated with her pulmonary and critical care colleagues to launch the I-SPY COVID-19 Trial, to rapidly screen agents to improve outcomes for critically ill COVID-19 patients. She earned her undergraduate degree at Harvard University and completed her medical and surgical training at Stanford University. She completed a postdoctoral fellowship in breast oncology at Stanford and later earned an M.A. degree from the Stanford Graduate School of Business.

PAT FURLONG is the founding president and chief executive officer of Parent Project Muscular Dystrophy, the largest nonprofit organization in the United States solely focused on Duchenne muscular dystrophy (Duchenne). Its mission is to improve the treatment, quality of life, and long-term outlook for all individuals affected by Duchenne through research, advocacy, education, and compassion. She is the mother of two sons who lost their battle with Duchenne in their teenage years. She has served on the boards of the Genetic Alliance and the Muscular Dystrophy Coordinating Committee of the U.S. Department of Health and Human Services. She has also served on the data safety monitoring board for both the Rare Diseases Clinical Research Network and the Cooperative International Neuromuscular Research Group. She was a member of the Institute of Medicine’s Committee on Accelerating Research and Development for Rare Diseases and Orphan Products. She graduated from Mt. St. Joseph College in Cincinnati, Ohio, with a B.S. in nursing, and attended graduate school at The Ohio State University.

TODD GILMER is professor and chief, Division of Health Policy, in the School of Public Health at the University of California, San Diego. His

research has focused on four areas: health insurance, risk adjustment in Medicaid, cost-effectiveness of diabetes care, and mental health services. He specializes in research design and data analysis; the use of large datasets including those from Medicare, Medicaid, and commercial health plans, national surveys and census data; mixed methods that combine analysis of health insurance claims with qualitative interviews and focus groups; and the evaluation of community-based interventions to improve the health of vulnerable populations. His recent work has examined the comparative effectiveness of supporting housing programs for persons with serious mental illness who are homeless and the importance of the fidelity of these programs to the Housing First model of permanent supported housing; the effectiveness of behavioral health integration and complex care management in Medicaid managed care; the use of peer providers in mental health programs designed for transitional-age youth; and service use after first episode of psychosis. He leads teams of health services researchers to provide data analysis and performance monitoring for San Diego County Behavioral Health Services and in studying innovative service delivery models in San Diego and Los Angeles counties. He received his Ph.D. in economics from the University of Washington.

ADI GUNDLAPALLI is the chief public health informatics officer of the Center for Surveillance, Epidemiology, and Laboratory Services at the Centers for Disease Control and Prevention (CDC). In this role, he leads an interdisciplinary team to meet the evolving data and information needs of public health, thereby enhancing informatics capability. Prior to coming to the CDC, he was the chief health informatics officer for the Salt Lake City Veterans Affairs Health Care System and an infectious diseases staff physician in Utah. He was a tenured professor of internal medicine at the University of Utah School of Medicine and a physician at the University of Utah Hospitals and Clinics. He is board certified in internal medicine, infectious diseases, and clinical informatics. His clinical and research interests include infectious diseases, clinical immunology, bio-surveillance (and biodefense), preparedness for public health emergencies, infection prevention and hospital epidemiology, and health care for vulnerable populations. He received his medical degree from the Madras Medical College in Madras (now called Chennai), India. He received further training at the University of Connecticut Health Center, where he earned a Ph.D. in immunology and completed an internal medicine residency. In Utah he completed a 3-year clinical and research fellowship in infectious diseases at the University of Utah School of Medicine and an M.A. degree (with a thesis) in biomedical informatics.

JACOB KEAN is a research scientist with the Department of Veterans Affairs (VA) Informatics and Computing Infrastructure of the Salt Lake City

VA Health Care System and an associate professor in health system innovation and research in the Department of Population Health Sciences at the University of Utah. He was previously a research scientist at the Regenstrief Institute and on the faculty at Indiana University School of Medicine. Kean served as a visiting scientist at the Boston University Rehabilitation Outcomes Center, a visiting scholar at the Center for Rehabilitation Research Using Large Datasets at the University of Texas Medical Branch, and a VA Career Development Awardee. His expertise lies at the nexus of the creation and operation of research networks and the evaluation of network care practices using patient-centered outcomes. He is the director of the Population Health Science–U Health Learning Health System; principal investigator (PI) of the Cerebral Palsy Research Network Data Coordinating Center; PI of the Department of Defense-funded Optimizing Rehabilitation InterventiONs (ORION) for Cognition Following Complex Traumatic Brain Injury network; and co-PI of the VA-funded Brain Injury Data Sharing Project—all of which are electronic health records–based learning health systems focused primarily on the care of persons with acquired neurological disorders. He has a Ph.D. from Indiana University Bloomington in speech and hearing sciences. He also completed the National Institutes of Health Training Institute for Dissemination and Implementation Research in Health and a post-doctoral master’s degree in measurement, evaluation, statistics, and assessment at the University of Illinois–Chicago.

MEAGAN KHAU is the director of the Data and Policy Analytics Group (DPAG) at the Centers for Medicare & Medicaid Services (CMS) Office of Minority Health (OMH). DPAG conducts research, data collection, and analyses to identify targets to reduce health disparities and improve quality of care, care transitions, access to care, and beneficiary satisfaction for vulnerable populations. DPAG is also involved in developing and implementing initiatives and data analyses to support cross-component/cross-agency collaborations to improve data collection, analysis, and reporting of race and ethnicity, primary language, disability, gender, and other characteristics associated with health disparities. Prior to joining CMS OMH, she was the deputy director of the Division of Pharmacy at the Center for Medicaid and Children’s Health Insurance Program Services, managing the operations of the Medicaid Drug Rebate Program, supporting system developments, ensuring program compliance, and implementing new policies and regulations. She received her master’s degree in health administration from the University of Southern California and her B.A. in sociology from the University of California, Irvine.

JAMES LEWIS is a professor of medicine and epidemiology, a senior scholar at the Center for Clinical Epidemiology and Biostatistics, and an

associate director of the Inflammatory Bowel Disease (IBD) Program at the University of Pennsylvania’s Perelman School of Medicine in Philadelphia, Pennsylvania. He has been actively involved in clinical research related to IBDs, medication safety, and optimizing medical therapies for more than 20 years. More recently, he has focused on the impact of diet on the gut microbiome and the course of IBD. His work has been funded by the National Institutes of Health, the Department of Health and Human Services Agency for Healthcare Research and Quality, the Patient-Centered Outcomes Research Institute, the Centers for Disease Control and Prevention, and numerous foundations and corporate sponsors. He previously served as the chair of the National Scientific Advisory Committee and as a member of the National Board of the Crohn’s & Colitis Foundation. Lewis is currently the lead scientist for the Foundation’s IBD Plexus Research Collaborative and co-principal investigator of SPARC-IBD, a multicenter prospective cohort study of patients with inflammatory bowel disease. He received his M.D. and his M.S. in clinical pharmacoepidemiology from the University of Pennsylvania School of Medicine.

MARSHA LILLIE-BLANTON is associate research professor in the Milken Institute School of Public Health at George Washington University. She is a public health professional with more than 30 years of experience working on health and health care access issues facing vulnerable populations. Her professional career has woven together opportunities to pursue scholarship and teaching in academia with efforts as a practitioner grounded in the realities that confront marginalized communities. She previously served as the chief quality officer and director of the Division of Quality and Health Outcomes at the Center for Medicaid and Children’s Health Insurance Program (CHIP) Services at the Centers for Medicare & Medicaid Services (CMS). With a budget of $500 million over 6 years, she had responsibility for establishing a health care quality measurement and reporting program for Medicaid and CHIP, oversight of state contracts for annual external quality reviews of Medicaid managed care organizations, developing the state-federal partnership in quality improvement activities, and conducting the first-ever nationwide survey of Medicaid beneficiaries’ experiences of care. Prior to her position with CMS, she held senior-level positions with the Henry J. Kaiser Family Foundation and the U.S. Government Accountability Office. She holds a bachelor’s degree from Howard University and a master of health science and doctorate degrees from the Johns Hopkins University Bloomberg School of Public Health.

VINCENT MOR (NAM) is the Florence Price Grant professor of community health at the Brown University School of Public Health and a research health scientist at the Providence Veterans Administration Medical Center.

He has been principal investigator of more than 40 National Institutes of Health (NIH)-funded grants focusing on the use of health services by and outcomes for frail and chronically ill people. He has evaluated the impact of programs and policies including Medicare funding of hospice, changes in Medicare nursing home payment, and the introduction of nursing home quality measures. He coauthored the congressionally mandated Minimum Data Set (MDS) and was architect of an integrated Medicare claims and clinical assessment data structure used for policy analysis, pharmacoepidemiology, and population outcome measurement. He developed summary measures using MDS data to characterize residents’ physical, cognitive, and psychosocial functioning. These data resources are the heart of Mor’s National Institute on Aging (NIA)–funded program project grant, Changing Long-Term Care in America, which examines the impact of Medicaid and Medicare policies on long-term care. These data are also at the core of a series of large, pragmatic cluster randomized trials of novel nursing home-based interventions led by Mor. He received a MERIT award from NIA at NIH, a Robert Wood Johnson Health Policy Investigator award, and the Distinguished Investigator award from AcademyHealth. He received his Ph.D. at the Florence Heller School for Advanced Studies in social welfare, Brandies University.

MARC NATTER is a pediatric rheumatologist and researcher in bioinformatics at the Boston Children’s Hospital Computational Health Informatics Program whose research is centered on the development and implementation of scalable software platforms that enable new ways of collecting and sharing data for research into chronic diseases. He is the chief informatics architect of the multi-site Childhood Arthritis & Rheumatology Research Alliance Registry for pediatric rheumatic diseases, leads the development of patient-facing technology for the Harvard Medical School-led Scalable Collaborative Infrastructure for Learning Healthcare System Clinical Data Research Network, and coordinates data integration and patient-facing technology for the PARTNERS Patient Powered Research Network and other projects. He received his M.D. from S.U.N.Y. at Stony Brook School of Medicine.

ALLISON OELSCHLAEGER is the chief data officer and director of the Office of Enterprise Data & Analytics at the Centers for Medicare & Medicaid Services (CMS). In this role, she focuses on maximizing the value and impact of CMS data for internal and external users. She oversees CMS’s data and information product portfolio and directs efforts to make CMS datasets available to external organizations. She also manages the development of advanced analytics using CMS data that help inform policy decisions and evaluate programs. Before joining CMS, she worked at the Lewin

Group, where she specialized in program evaluation and data analysis. She is a graduate of Georgetown University.

MARC OVERHAGE (NAM) is the chief health information officer for Anthem, Inc., and previously served as the chief medical informatics officer for Cerner and Siemens Health Services. He is an internationally recognized expert in health information modeling, standards, and interoperability as well as clinical decision support, health services research, and implementation science. Previously, Overhage was the director of medical informatics and a research scientist at the Regenstrief Institute, Inc., and professor of medicine and Regenstrief professor of medical informatics at the Indiana University School of Medicine. He also is a member of the medical staff of the Wishard Memorial Hospital. His research has focused on the use of informational interventions to modify provider behavior, including computerized provider order entry, clinical decision support systems, and other forms of feedback. These systems require clinical data to drive them and have led him to begin developing approaches to health information exchange. To facilitate this work, he has engaged in developing clinical information standards, advising the federal government on policy-guiding health information technology, and developing sustainable models for providing health information services. He received his B.A. in physics from Wabash College and an M.D. in medicine and a Ph.D. in biophysics from the Indiana University School of Medicine.

MITRA ROCCA is associate director of medical informatics, Office of Translational Science, Center for Drug Evaluation and Research (CDER), at the Food and Drug Administration (FDA). She joined the FDA in 2009 as the senior medical informatician responsible for developing the health information architecture of the Sentinel System. Rocca serves as the lead for the FDA CDER Health Information Technology (IT) Board, focusing on the use of health IT to enhance regulatory decision making. She also serves as the FDA CDER lead to Health Level Seven (HL7), responsible for the review of HL7 draft standards. Prior to joining FDA, she served as the associate director of health care informatics at Novartis Pharmaceuticals Corporation, focusing on the reuse of electronic health records in clinical research. She served as the co-chair of the HL7 Clinical Interoperability Council from 2012 to 2018. She is a fellow of the American Medical Informatics Association and holds an advanced degree in medical informatics from the University of Heidelberg in Germany.

CLAUDIA STEINER is the executive director of the Institute for Health Research at Kaiser Permanente Colorado. She served as director for the Division of Healthcare Delivery Data, Measures and Research, in the

Center for Delivery, Organization, and Markets within the Agency for Healthcare Research and Quality (AHRQ) until February 2017. There she led the division’s development and dissemination of data and software tools for use in research, policy analysis, quality improvement, and public reporting, with a particular focus on the Healthcare Cost and Utilization Project and the AHRQ Quality Indicators. She has conducted research on the influence of ambulatory surgery on standards of care, utilization, and clinical outcomes; the epidemiology of infectious diseases, including health care–associated infections; the prevalence and factors associated with read-missions to the acute care setting; and the use and impact of new medical technologies. She was a practicing internal medicine physician for 25 years with the Johns Hopkins Community Physicians and continues to serve as a practicing internist for adult patients with the Colorado Permanente Medical Group. She earned her medical degree and completed residency training in internal medicine at the University of Colorado Health Sciences Center. Subsequently, she obtained a master’s of public health at the Johns Hopkins School of Hygiene and Public Health while completing a research fellowship through the Department of Medicine at Johns Hopkins University.

MICKY TRIPATHI is the national coordinator for health information technology at the U.S. Department of Health and Human Services, where he leads the formulation of the federal health information technology (IT) strategy and coordinates federal health IT policies, standards, programs, and investments. He has more than 20 years of experience across the health IT landscape. Tripathi most recently served as chief alliance officer for Arcadia, a health care data and software company focused on population health management and value-based care; as project manager of the Argonaut Project, an industry collaboration to accelerate the adoption of Fast Healthcare Interoperability Resources; and as a board member of HL7, the Sequoia Project, the CommonWell Health Alliance, and the CARIN Alliance. He served as the president and chief executive officer of the Massachusetts eHealth Collaborative, a nonprofit health IT advisory and clinical data analytics company. He was also the founding president and chief executive officer of the Indiana Health Information Exchange (HIE), a statewide HIE partnered with the Regenstrief Institute; an executive advisor to the investment firm LRVHealth; and a fellow at the Berkman-Klein Center for Internet and Society at Harvard University. Prior to receiving his Ph.D., he was a presidential management fellow and a senior operations research analyst in the Office of the Secretary of Defense in Washington, DC, for which he received the Secretary of Defense Meritorious Civilian Service Medal. He holds a Ph.D. in political science from the Massachusetts Institute of Technology, a master’s degree in public policy from Harvard University, and an A.B. in political science from Vassar College.