Appendix C

Building Data Capacity for Patient-Centered Outcomes Research: Interim Report 2–Data Standards, Methods, and Policy

(Full text of the committee’s second interim report released on October 27, 2021.)¹

___________________

¹https://www.nap.edu/catalog/26298/building-data-capacity-for-patient-centered-outcomes-research-interim-report.

This page intentionally left blank.

Building Data Capacity
for Patient-Centered
Outcomes Research

INTERIM REPORT 2–
Data Standards, Methods, and Policy

Committee on Building Data Capacity for
Patient-Centered Outcomes Research:
An Agenda for 2021 to 2030

Committee on National Statistics
Division of Behavioral and Social Sciences and Education

Board on Health Care Services
Health and Medicine Division

Computer Science and Telecommunications Board
Division on Engineering and Physical Sciences

A Consensus Study Report of

THE NATIONAL ACADEMIES PRESS
Washington, DC
www.nap.edu

THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001

This activity was supported by a contract between the National Academy of Sciences and the U.S. Department of Health and Human Services (award #HHSP233201400020B/75P00120F37102). Any opinions, findings, conclusions, or recommendations expressed in this publication do not necessarily reflect the views of any organization or agency that provided support for the project.

International Standard Book Number-13: 978-0-309-27262-9
International Standard Book Number-10: 0-309-27262-9
Digital Object Identifier: https://doi.org/10.17226/26298

Additional copies of this publication are available from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.

Copyright 2022 by the National Academy of Sciences. All rights reserved.

Printed in the United States of America

Suggested citation: National Academies of Sciences, Engineering, and Medicine. (2022). Building Data Capacity for Patient-Centered Outcomes Research: Interim Report 2—Data Standards, Methods, and Policy. Washington, DC: The National Academies Press. https://doi.org/10.17226/26298.

The National Academy of Sciences was established in 1863 by an Act of Congress, signed by President Lincoln, as a private, nongovernmental institution to advise the nation on issues related to science and technology. Members are elected by their peers for outstanding contributions to research. Dr. Marcia McNutt is president.

The National Academy of Engineering was established in 1964 under the charter of the National Academy of Sciences to bring the practices of engineering to advising the nation. Members are elected by their peers for extraordinary contributions to engineering. Dr. John L. Anderson is president.

The National Academy of Medicine (formerly the Institute of Medicine) was established in 1970 under the charter of the National Academy of Sciences to advise the nation on medical and health issues. Members are elected by their peers for distinguished contributions to medicine and health. Dr. Victor J. Dzau is president.

The three Academies work together as the National Academies of Sciences, Engineering, and Medicine to provide independent, objective analysis and advice to the nation and conduct other activities to solve complex problems and inform public policy decisions. The National Academies also encourage education and research, recognize outstanding contributions to knowledge, and increase public understanding in matters of science, engineering, and medicine.

Learn more about the National Academies of Sciences, Engineering, and Medicine at www.nationalacademies.org.

Consensus Study Reports published by the National Academies of Sciences, Engineering, and Medicine document the evidence-based consensus on the study’s statement of task by an authoring committee of experts. Reports typically include findings, conclusions, and recommendations based on information gathered by the committee and the committee’s deliberations. Each report has been subjected to a rigorous and independent peer-review process and it represents the position of the National Academies on the statement of task.

Proceedings published by the National Academies of Sciences, Engineering, and Medicine chronicle the presentations and discussions at a workshop, symposium, or other event convened by the National Academies. The statements and opinions contained in proceedings are those of the participants and are not endorsed by other participants, the planning committee, or the National Academies.

For information about other products and activities of the National Academies, please visit www.nationalacademies.org/about/whatwedo.

COMMITTEE ON BUILDING DATA CAPACITY FOR PATIENT-CENTERED OUTCOMES RESEARCH: AN AGENDA FOR 2021 TO 2030

GEORGE ISHAM (Chair), HealthPartners Institute

JOHN F.P. BRIDGES, The Ohio State University

JULIE BYNUM, University of Michigan

ANGELA DOBES, IBD Plexus, Crohn’s & Colitis Foundation

DEBORAH ESTRIN, Cornell Tech

OLUWADAMILOLA FAYANJU, University of Pennsylvania

CONSTANTINE GATSONIS, Brown University

ROBERT GOERGE, Chapin Hall, University of Chicago

GEORGE HRIPCSAK, Columbia University

LISA IEZZONI, Massachusetts General Hospital

S. CLAIBORNE JOHNSTON, The University of Texas at Austin

MIGUEL MARINO, Oregon Health & Science University

ELIZABETH MCGLYNN, Kaiser Permanente

DAVID MELTZER, University of Chicago

PAUL TANG, Stanford University and Palo Alto Medical Foundation

KRISZTINA MARTON, Study Director

CRYSTAL BELL, Associate Program Officer

RUTH COOPER, Associate Program Officer

MARY GHITELMAN, Senior Program Assistant

BRIAN HARRIS-KOJETIN, Director, Committee on National Statistics

SHARYL NASS, Director, Board on Health Care Services

JON EISENBERG, Director, Computer Science and Telecommunications Board

SAUL RIVAS, National Academy of Medicine Fellow, University of Texas Rio Grande Valley

COMMITTEE ON NATIONAL STATISTICS

ROBERT M. GROVES (Chair), Georgetown University

ANNE C. CASE, Princeton University

MICK P. COUPER, University of Michigan

JANET M. CURRIE, Princeton University

DIANA FARRELL, JPMorgan Chase Institute

ROBERT GOERGE, Chapin Hall at the University of Chicago

ERICA L. GROSHEN, Cornell University

HILARY HOYNES, University of California, Berkeley

DANIEL KIFER, The Pennsylvania State University

SHARON LOHR, Arizona State University, emerita

JEROME P. REITER, Duke University

JUDITH A. SELTZER, University of California, Los Angeles

C. MATTHEW SNIPP, Stanford University

ELIZABETH A. STUART, Johns Hopkins University

JEANETTE WING, Columbia University

BRIAN A. HARRIS-KOJETIN, Director

CONSTANCE F. CITRO, Senior Scholar

BOARD ON HEALTH CARE SERVICES

DAVID BLUMENTHAL (Chair), The Commonwealth Fund

ANDREW BINDMAN, Kaiser Foundation Health Plan, Inc.

NIRANJAN BOSE, Gates Ventures

MELINDA J. BEEUWKES BUNTIN, Vanderbilt University School of Medicine

NEIL S. CALMAN, The Institute for Family Health

PAUL CHUNG, Kaiser Permanente School of Medicine

PATRICIA M. DAVIDSON, Johns Hopkins University School of Nursing

MARTHA DAVIGLUS, University of Illinois at Chicago

JENNIFER E. DEVOE, Oregon Health & Science University

R. ADAMS DUDLEY, University of Minnesota

RICHARD G. FRANK, Harvard Medical School

TERRY FULMER, John A. Hartford Foundation

CINDY GILLESPIE, Arkansas Department of Human Services

ELMER HUERTA, The George Washington University Cancer Center

SHARON INOUYE, Harvard Medical School

JOHN LUMPKIN, BlueCross BlueShield of North Carolina Foundation

FAITH MITCHELL, The Urban Institute

DAVID B. PRYOR, Ascension Health

TRISH RILEY, National Academy for State Health Policy

WILLIAM SAGE, The University of Texas at Austin

HARDEEP SINGH, Baylor College of Medicine

SHARYL NASS, Director

COMPUTER SCIENCE AND TELECOMMUNICATIONS BOARD

LAURA HAAS (Chair), University of Massachusetts Amherst

DAVID CULLER, University of California, Berkeley

ERIC HORVITZ, Microsoft Corporation

CHARLES ISBELL, Georgia Institute of Technology

BETH MYNATT, Georgia Institute of Technology

CRAIG PARTRIDGE, Colorado State University

DANIELA RUS, Massachusetts Institute of Technology

FRED B. SCHNEIDER, Cornell University

MARGO SELTZER, The University of British Columbia

NAMBIRAJAN SESHADRI, University of California, San Diego

MOSHE VARDI, Rice University

JON EISENBERG, Senior Board Director

Acknowledgments

This Consensus Study Report was reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise. The purpose of this independent review is to provide candid and critical comments that will assist the National Academies of Sciences, Engineering, and Medicine in making each published report as sound as possible and to ensure that it meets the institutional standards for quality, objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process.

We thank the following individuals for their review of this report: Rebecca A. Hubbard, Department of Biostatistics, Epidemiology and Informatics, Perelman School of Medicine, University of Pennsylvania; Sue Jinks-Robertson, Department of Molecular Genetics and Microbiology, Duke University; Harold Lehman, Division of General Internal Medicine, Department of Medicine, Johns Hopkins University School of Medicine; Vincent X. Liu, Division of Research, Kaiser Permanente; Keith Marsolo, Population Health Sciences, Duke University School of Medicine; Emily O’Brien, Department of Population Health Sciences, Duke Clinical Research Institute, Duke University School of Medicine; and Jerome Reiter, Department of Statistical Science, Duke University.

Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the conclusions of this report, nor did they see the final draft before its release. The review of this report was overseen by Andrew B. Bindman, Chief Medical Officer, Kaiser Foundation Health Plan and Hospitals, and Alicia L. Carriquiry, Department of Statistics, Iowa State University. They were responsible for

making certain that an independent examination of this report was carried out in accordance with the standards of the National Academies and that all review comments were carefully considered. Responsibility for the final content rests entirely with the authoring committee and the National Academies.

Contents

Summary

1 Introduction

2 Patient-Centered Outcomes Research Data Standards

3 Methods for Patient-Centered Outcomes Research

4 Data Policies and Other Data Infrastructure Considerations

Appendixes

A Biographical Sketches of Committee Members

B Workshop Agenda

C Biographical Sketches of Workshop Speakers

This page intentionally left blank.

Boxes and Figures

BOXES

1-1 Key Data Infrastructure Functionalities in the Existing Strategic Framework for Patient-Centered Outcomes Research

1-2 Building Blocks of the Patient-Centered Outcomes Research Data Infrastructure

1-3 Statement of Task for the Overall Study

2-1 A Perspective on Data Required for Patient-Centered Outcomes Research and Standards to Consider

FIGURES

1-1 Patient-Centered Outcomes Research Trust Fund: Three streams of work and funding

1-2 ASPE’s strategic framework for the patient-centered outcomes research data infrastructure

2-1 Clinical language engineering workbench: Key functionalities in the natural language processing machine learning process

3-1 A patient timeline view of data

This page intentionally left blank.

Summary

The Office of the Assistant Secretary for Planning and Evaluation (ASPE), in partnership with other agencies and divisions of the U.S. Department of Health and Human Services (HHS), coordinates a portfolio of projects that build data capacity for conducting patient-centered outcomes research (PCOR). PCOR focuses on producing scientific evidence on the effectiveness of prevention and treatment options to inform the health care decisions of patients, families, and health care providers, taking into consideration the preferences, values, and questions patients face when making health care choices. The data infrastructure includes data sources and functionalities that support the research. Major building blocks are the services, standards, policies, and governance that enable the use of the data.

ASPE asked the National Academies of Sciences, Engineering, and Medicine to appoint a consensus study committee to identify issues critical to the continued development of the data infrastructure for PCOR. The committee’s work will contribute to ASPE’s development of a strategic plan that will guide their work related to PCOR data capacity over the next decade.

As part of its information-gathering activities, the committee organized three workshops to collect input from stakeholders on the PCOR data infrastructure, which includes a variety of types of data, such as clinical data, research data, administrative data from payer records, and patient-provided data. This report, the second in a series of three interim reports, summarizes the discussion and committee conclusions from the second workshop, which focused on data standards, methods, and policies that could make the PCOR data infrastructure more useful in the years ahead.

Participants in the workshop included researchers and policy experts working in these areas. The first report in the series centered on emerging data needs.¹ The third report will discuss research and data collaborations.

The conclusions included in this interim report are based primarily on the input collected as part of the workshop, background documentation received from ASPE and other public sources, and the committee members’ synthesis and expert judgment regarding the input received. As an interim report based on one in a series of information-gathering activities, the scope of this report is narrowly focused on a subset of key topics relevant to the committee’s charge. The conclusions reached by the committee are, at this stage, fairly high level. After completing all of its information-gathering activities, which include but are not limited to the three workshops, the committee will also issue a final report, containing the study’s overall findings and conclusions.

DATA STANDARDS

Part of the workshop discussed in this report focused on standards for PCOR. Standards are increasingly widely used for a variety of purposes, including collecting, storing, analyzing, and exchanging data. One theme that emerged from the workshop was that these standards are most useful when they are focused on addressing a specific problem or are driven by the specific value they can contribute. The needs and norms evolve over time, and because of this, standards need to evolve too. The workshop also identified some key areas where ASPE’s role is particularly important.

CONCLUSION 2-1: Standards are most useful when their development is driven by their potential uses and a clear concept of the value they can contribute.

CONCLUSION 2-2: The Office of the Assistant Secretary for Planning and Evaluation could add significant value in the area of standards for patient-centered outcomes research by

• continuing to promote the development of a data infrastructure and an implementation strategy that facilitates the use of standards and access to the data;
• convening stakeholder meetings to enhance communication and work toward developing a common language for standards;

___________________

¹https://www.nap.edu/catalog/26297/building-data-capacity-for-patient-centered-outcomes-research-interim-report.

• facilitating accessibility to the data and collaborations with existing organizations working in this area; and
• leading efforts to catalogue and exemplify data standards and analytic standards.

CONCLUSION 2-3: While data standards are important to conducting patient-centered outcomes research, applying standards to the analytic methods as well is important to facilitate the reliability and reproducibility of study results.

Learning from the work on standards happening across the globe would further advance PCOR.

CONCLUSION 2-4: An international perspective is an important consideration for the patient-centered outcomes research data infrastructure, and the infrastructure focused on standards specifically would benefit from building on work that happens internationally.

METHODS

A promising area of research focuses on better understanding the longitudinal, holistic experiences of people across time and different settings, which requires matching records across databases. To balance these opportunities and concerns, it would be useful to develop a carefully thought-out strategy for linking data from a variety of sources, and to focus on strengthening the data infrastructure in additional ways that would enable longitudinal research that provides a comprehensive understanding of people’s experiences over their life course.

CONCLUSION 3-1: The ability to adopt a longitudinal, comprehensive perspective of an individual’s journey could open new opportunities for patient-centered outcomes research. The shift could be facilitated by focusing on efforts to

• simplify integration of data across the research data ecosystem;
• address challenges posed by the limitations associated with health identifiers;
• incorporate person-generated data into health data systems; and
• leverage real-world data to expand the timeline view of a person’s health-related experiences.

There is also a need for transparency and the continued refinement of best practices related to how data and methods are used. This is especially

important for emerging data sources and methods that capture more lifestyle and behavioral information than traditional clinical measurements, as well as the application of artificial intelligence methods, which could have biases confounded by patterns associated with new technologies. These considerations are also important for the use of observational data.

CONCLUSION 3-2: Observing scientific best practices, including those of transparency and ethical use of data, is essential to generate trust in patient-centered outcomes research among all stakeholders, including the public and researchers. This is important both for observational data and for emerging data sources and methods.

The workshop highlighted the importance of interpreting best practices in the dissemination of research broadly, to include not only sharing results but also making available other resources and components associated with the research process, such as the software developed for the analyses.

CONCLUSION 3-3: The results of patient-centered outcomes research (and research in general) are only replicable and are most useful when the underlying data and comprehensive research documentation (such as analytic code) are made available for use by others.

DATA POLICY AND OTHER DATA INFRASTRUCTURE CONSIDERATIONS

A theme that emerged from the workshop was the need to involve the people and communities whose data are being used in decisions about the data collection and data use throughout the entire research lifecycle. This is essential for building trust, which increases willingness to participate and, in turn, the likelihood that the data that are obtained will be complete, reliable, representative, and relevant to diverse stakeholders. This is particularly important for data on social determinants of health.

CONCLUSION 4-1: Building and maintaining trust among the people and communities whose data are being sought for research is essential for high-quality data. Including representatives of consumers and patients in the research process to understand how to measure health impacts that matter to individuals is an important component in building trust.

The existing laws and regulations that govern the use of data for research, including the Health Insurance Portability and Accountability Act of

1996, are outdated and would benefit from a critical review and updating to facilitate PCOR while preventing misuses of the data.

CONCLUSION 4-2: This is an opportune time to revisit and update the legislation and rules governing data privacy and the sharing of data for research.

This page intentionally left blank.

1

Introduction

The Office of the Assistant Secretary for Planning and Evaluation (ASPE), in partnership with other agencies and divisions of the U.S. Department of Health and Human Services (HHS), coordinates a portfolio of projects that build data capacity for conducting patient-centered outcomes research (PCOR). The PCOR data infrastructure provides decision makers with objective, scientific evidence on the effectiveness of treatments, services, and other interventions used in health care. This research is frequently focused on analyzing existing data to address questions and provide objective information for the purpose of informing real-world health care decisions.

BACKGROUND

The legal framework that established funding for research on the outcomes and effectiveness of treatments and health care interventions dates back to the 2003 Medicare Prescription Drug, Improvement, and Modernization Act. This act provided authorization for the Agency for Healthcare Research and Quality (AHRQ) to support research comparing the outcomes and effectiveness of treatments and clinical approaches and to disseminate the findings from this research. In 2009, the American Recovery and Reinvestment Act provided additional funding to AHRQ, the National Institutes of Health, and HHS for research that compares the effectiveness of medical options. In 2010, the Patient Protection and Affordable Care Act provided further authorization for research that assists patients, clinicians, purchasers, and policy makers in making informed health decisions.

To facilitate PCOR, in 2010 Congress established the Patient-Centered Outcomes Research Trust Fund (PCOR Trust Fund) with the U.S. Department of the Treasury. The goals of the PCOR Trust Fund are to fund PCOR research, disseminate research findings, and develop a data infrastructure for PCOR. The PCOR Trust Fund has been reauthorized through 2029, through H.R.1865 of the Further Consolidated Appropriations Act of 2020. The most recent statute specified intellectual and developmental disabilities, as well as maternal mortality, as research priorities. The statute also called for PCOR studies to include consideration of the full range of outcomes data. Specifically, the law states that:

The bulk of the PCOR Trust Fund funding (80%) is allocated for research and is made available through the Patient-Centered Outcomes Research Institute (PCORI), a nongovernmental organization established by Congress for this purpose. Approximately 16 percent of the PCOR Trust Fund funding is set aside for disseminating research findings, incorporating findings into clinical practice, and training researchers in PCOR. The agency overseeing this work is AHRQ.

The remaining funding, which constitutes 4 percent of the PCOR Trust Fund, is allocated for building data capacity for PCOR and is overseen by ASPE. Specifically, Section 937(f) of the Public Health Service Act instructed the Secretary of HHS to:

___________________

¹https://www.ssa.gov/OP_Home/ssact/title11/1181.htm.

²https://aspe.hhs.gov/collaborations-committees-advisory-groups/os-pcortf/about-os-pcortf.

Figure 1-1 shows how the PCOR funding and work is allocated across the three entities. This National Academies of Sciences, Engineering, and Medicine study is focused on issues relevant to ASPE’s continued work on the PCOR data infrastructure, in other words, on the priorities for the use of the 4 percent of the funding that is allocated to HHS for work related to the data infrastructure for PCOR.

As the coordinating agency for the data infrastructure investment portfolio across HHS agencies, ASPE guides the PCOR data infrastructure’s strategic framework and vision, sets funding priorities, and coordinates interagency workgroups. ASPE’s work is assisted by a Leadership Council for the PCOR Trust Fund, which includes representatives from other HHS agencies, including the Administration for Children and Families, the Administration for Community Living, the Assistant Secretary for Preparedness and Response, AHRQ, the Centers for Disease Control and Prevention (CDC), the Centers for Medicare & Medicaid Services, the U.S. Food and Drug Administration (FDA), the Health Resources and Services Administration, the Indian Health Service, the National Institutes of Health, the Office of the Chief Technology Officer, the Office of the National Coordinator for Health Information Technology, and the Substance Abuse and Mental Health Services Administration. The Leadership Council provides input on priorities for the portfolio, including projects to fund. During the period

2010 to 2019, the PCOR Trust Fund funded 53 projects, which translated to 76 agency awards, totaling approximately $131 million.

Figure 1-2 is a visual representation of ASPE’s current framework for the PCOR data infrastructure. The bottom row shows the main data sources feeding into the PCOR infrastructure. Data collected as part of clinical care include data collected for health care delivery and for billing purposes. Examples of primary data collected as part of research studies include data from clinical trials and national health surveys. Other examples of data sources include Medicare or Medicaid claims data; quality or outcomes data collected by health care providers for the purposes of improving health care value; FDA data on the safety of medications and medical devices; and CDC data on births and deaths provided by state public health authorities.

The framework describes the relationship between the data sources and the current key functionalities and focus areas (middle row) that support the research. The key functionalities are described in further detail in Box 1-1. Major building blocks are the services, standards, policies, and governance that enable the use of the data for research, described in further detail in Box 1-2. The top row shows the key data users and contributors of data. A more detailed overview of ASPE’s work and the projects funded to date will be included in the final report, at the conclusion of the committee’s review.

ISSUES FOR THE COMMITTEE

ASPE asked the National Academies of Sciences, Engineering, and Medicine to appoint a consensus study committee and identify issues critical to building data capacity for PCOR and for generating new evidence to inform health care decisions. The input provided by the committee will contribute to ASPE’s strategic planning for their work related to the data infrastructure over the next decade. The study is part of a broader initiative by ASPE intended to update the strategic plan in light of the reauthorization

of the PCOR Trust Fund and advances in health information technology and interoperability tools in recent years.

The study is a collaboration of three units of the National Academies: the Committee on National Statistics, the Board on Health Care Services, and the Computer Science and Telecommunications Board. The consensus study committee has a diverse membership; its 15 members include experts with decades of experience, as well as emerging leaders, in the broad fields of (1) PCOR; (2) research methods, statistics, and demography; (3) computer science and data infrastructure; and (4) patient engagement and patient perspectives. Appendix A contains the biographical sketches of the committee members.

As part of its information-gathering activities, the committee was asked to organize three workshops to collect input from stakeholders on aspects of the charge developed in consultation with ASPE. The workshops focused on key topics that the committee believed would particularly benefit from broad input from a variety of data users and other stakeholders. The committee’s conclusions from each workshop are summarized in a series of interim reports, of which the first centered on emerging data needs. This first interim report summarizes the discussion and committee conclusions from the first workshop, which focused on looking ahead at data user needs over the next decade. The second workshop in the series centered on data standards, methods, and policies that could make the PCOR data infrastructure more useful. The third workshop discussed research and data collaborations. This report summarizes the discussion and committee conclusions from the second workshop, which focused on data standards, methods, and policies that could make the PCOR data infrastructure more useful. The third report will discuss research and data collaborations.

As an interim report focused on one in a series of information-gathering activities, the scope of this report is limited to a subset of the topics relevant to the committee’s charge and the conclusions reached by the committee are, at this stage, fairly high level. Some aspects of the topics discussed are examined in further detail in other workshops. After completing all of its information-gathering activities, the committee will issue a final report, which will integrate and examine these topics in further detail.

Box 1-3 shows the committee’s Statement of Task for the overall study. The committee will address this charge in its final report, integrating what was learned from the workshops and from all other forms of input, including public meetings with HHS staff and background documentation available on the history and operations of the PCOR Trust Fund. The final report will contain overall findings and conclusions from the study, on the basis of the committee’s further deliberations and integrated judgment on the input received and materials reviewed.

Appendix B shows the agenda for the workshop, which was held on May 24, 2021. The committee’s goal for this event was to bring together researchers and policy experts to

• Identify data standards and methods that can make the PCOR data infrastructure more useful for research and other data needs.
• Identify data policies that are needed to facilitate the continued development and operation of the PCOR data infrastructure.
• Discuss what HHS is best positioned to address and support, and how the agency could maximize resources available for the PCOR

• data infrastructure (representing 4% of the PCOR Trust Fund), in the context of the HHS public mission, authorities, programs, and data resources.

Invited speakers in each of the sessions were asked to reflect on the general topics above. The specific questions for each session are described in Chapters 2 through 4. An obvious limitation of an activity of this type is that only a small number of stakeholders can be invited to speak. To compensate for this limitation, the invited participants included diverse experts working in a variety of areas and on a range of types of projects, including both early career researchers and experts with decades of experience. A recording of the workshop as well as the presentation slides used by the speakers are available on the National Academies website at www.nationalacademies.org/PCORData.

Prior to the workshop, information about the event was disseminated through National Academies mailing lists and on the project website. To collect additional stakeholder input, members of the public were invited to provide comments on topics related to the workshop (or any other topic related to the committee’s charge), using a public input form available on the National Academies website.

OVERVIEW OF THE REPORT

This report is organized around the three main sessions of the workshop: Chapter 2 discuses data standards, Chapter 3 is centered on research methods, and Chapter 4 describes discussions focused on data policies and related infrastructure considerations. The points conveyed by the workshop participants do not necessarily reflect the views of the committee. In each chapter, a summary of the input received is followed by the committee’s conclusions. The conclusions are based primarily on the input collected as part of the workshop, background documentation received from ASPE and other public sources, and the committee members’ synthesis and expert judgment. Because this is an interim report, the committee’s conclusions at this stage are big-picture conclusions, which will be integrated with additional input over the course of the study.

2

Patient-Centered Outcomes Research Data Standards

This chapter summarizes presentations and discussion focused on data standards. As discussed in Chapter 1, the Office of the Assistant Secretary for Planning and Evaluation (ASPE) considers standards to be one of the building blocks of the patient-centered outcomes research (PCOR) infrastructure. Specifically

Speakers participating in this session were asked to focus on the questions below. The brief overview of the input received from the presenters is followed by the committee’s conclusions.

• What data standards could make the PCOR data infrastructure more useful for research and other data needs? What data standards are likely to become more relevant looking forward? What needs to be prioritized?

___________________

¹https://aspe.hhs.gov/patient-centered-outcomes-research-trust-fund-faqs.

• What role can ASPE play in supporting effective standards to build data capacity that supports PCOR studies? What characteristics of the U.S. Department of Health and Human Services’ (HHS’) public mission, programs, or authorities could be leveraged?

John Halamka, Mayo Clinic, provided some context for the session by describing three ways of thinking of data standards. First, there is a need for standards for presenting content from data sources such as electronic health records or other administrative records. Second, there is a need for vocabulary standards that would provide semantic interoperability for the content (e.g., SNOMED CT, RxNorm, and LOINC). Third, standards are needed for transporting the data from one place to another in a secure way (such as HL7 Version 2, EDIFACT, X12, and XML standards of various kinds). Ultimately, this led to the development of the Fast Healthcare Interoperability Resources (FHIR), which enable provider-to-provider, provider-to-patient, and provider-to-payer workflows to be supported by FHIR/JavaScript object notation and rest parameters. Halamka said that this is a good development, though it is not enough.

Looking ahead at the next 10 years, the Mayo Clinic has an initiative to move to a digital-first approach in its operations, undertake clinical research and clinical trials with less friction, and enable global access to new kinds of ideas. To accomplish this, the Mayo Clinic needs access to more types of data, and expanding the FHIR standard is one way to enable that. Two interesting examples are the Minimal Common Oncology Data Elements (mCODE) and the Mobile Health Augmented Cardiac Rehabilitation (MCard) data sets. Taking the basic FHIR construct and adding domain-specific data elements will be useful for research, but thousands of data elements may be needed to deal with various use cases.

Halamka also discussed an example of data standards applicable to research on COVID-19 treatment. There is a need to define “ventilator days” for research on when medications versus other interventions would make a difference to COVID-19 patients, he noted; at present, ventilator days are not defined in electronic health records in a standard way. Ultimately, a working group of about 100 experts might be involved in deciding on the definitions of data elements that are needed to answer process questions.

The challenges associated with data standards are heightened, Halamka added, when the research goes beyond structured and unstructured clinical data to incorporate other forms of data, such as research data emerging from various “omics” fields (e.g., genomics). Data from wearable devices, or what he called “high-velocity continuous data,” comprise another area where standards are lacking.

Halamka also discussed the topic of data de-identification. He said that the Mayo Clinic has de-identified its data for use in clinical research and

scientific discovery, but de-identification is itself a science that is evolving. For example, when using information from computed tomography (CT) on a patient’s head, the patient’s name and medical number would be removed, but it remains possible to reconstruct the person’s face from the CT with 3D reconstruction software. Halamka said that he and his colleagues found that they have been able to re-identify a person this way 27 percent of the time using publicly available images and photo matching. Because of this, they have been working to develop technologies that would prevent head CT data from being used to reconstruct a face. This area needs attention, because de-identified information that cannot lead to unique identification today could nevertheless potentially become identifiable in the future as technologies evolve. Halamka also highlighted the related consideration of ways of obtaining consent for the future use of data, and specifically the reuse of de-identified, aggregate data.

Shaun Grannis, Regenstrief Institute and Indiana University, focused on standards for data linkage, particularly for participant-provided information. Within this area, he discussed the topics of (1) patient identity strategy, (2) digital identity and federation, (3) privacy-preserving record linkage, and (4) linking of social determinants of health (SDOH) data.

Grannis noted that the patient identity strategy in the United States is evolving based on a recognition that matching patient records from different sources is one of the few remaining large holes in the electronic health data infrastructure. For this reason, Congress charged the Office of the National Coordinator for Health Information Technology with writing a report focused on effective matching methods. While the report had not been released at the time of this workshop, Grannis underscored the importance of monitoring developments in this area to understand how PCOR can benefit from any changes.

Within the digital identity framework, Grannis noted, not everyone is willing to wait for the U.S. government to develop a national strategy. He pointed out that “matching on devices” is an area where a lot of work is taking place. Organizations such as the CARIN Alliance are working to develop a digital identity and a federated trust agreement to increase and federate trust in digital identity credentials. They are also considering digital identity frameworks that work well within FHIR. Grannis noted that there are also new developments in the more technical areas of identity certification or proofing. Examples of this include the work of FIDO and DirectTrust, as well as identity assigners such as ID.me, AllClearID, and Okta. These organizations are beginning to work on identity approaches at scale in health care, so it is important to observe these developments and learn from these experiences.

Concerning privacy issues, Grannis mentioned that he is working on two projects to advance privacy-preserving record linkage. This is a

maturing field, with some maturing technologies that already work well. There is a need to establish best practices, use cases, and evidence-based guidance on how to conduct this work to ensure that this field does not become too fragmented and those participating in it do not find themselves doing everything differently. Specifically, it would be helpful to have widely shared technical descriptions of what types of data, what types of tokens, and what types of information combinations work well.

Grannis also discussed his work on linking nontraditional, nonclinical SDOH data in Indiana. He noted that there are various methods for linking these types of data, including linking by person, place, or time—or some combination of those—but work remains to identify the best approaches. Grannis argued that for the SDOH data, the main considerations are granularity, standardization, and linkage. For example, in some cases using zip code-level data is valuable, but in other cases more granular information is needed. A variety of data are available on SDOH, and Grannis said that coordinating work on PCOR with the work carried out by the Office of the National Coordinator for Health Information Technology in this area would be important.

Concerning the challenge of matching data, Grannis noted that there have been sustained efforts over the years to find new approaches to matching, and many researchers have been advocating for similar things. What is needed is alignment in this area. As an example of building on evidence-based research to develop standards, Grannis mentioned a 2019 paper that showed that standardizing address and last name significantly improves matching accuracy.² This research led to a bipartisan Senate bill calling to address standardization, and work is now in progress on developing a universal standard.

Evelyn Gallego, EMI Advisors, discussed her work on the Gravity Project, which focuses on developing consensus-driven data standards to support use and exchange of SDOH within the health care sectors and between the health care sector and other sectors, including research. She said that even before the onset of COVID-19, there was growing awareness that SDOH information improves whole-person care and lowers health care costs, and that unmet social needs negatively impact health outcomes.

Gallego discussed several uses of social risk data, identified by the Social Interventions Research and Evaluation Network (SIREN). These areas include medical care, population health management, community health improvement, social risk interventions, risk adjustment, and

___________________

² S.J. Grannis, H. Xu, J.R. Vest, S. Kasthurirathne, N. Bo, B. Moscovitch, R. Torkzadeh, and J. Rising. (2019). Evaluating the effect of data standardization and validation on patient matching accuracy. Journal of the American Medical Informatics Association, 26(5), 447–456. https://doi.org/10.1093/jamia/ocy191.

research. Despite a clear business case, clinical systems face challenges in capturing and exchanging this type of data. Gallego cited a 2020 paper from the National Association of Social Determinants of Health that identified the key challenges as follows:

• Consent management.
• Standardization of SDOH data collection and storage.
• Data sharing between ecosystem parties.
• Access and comfort with digital solutions.
• Concerns about information collection and sharing.
• Social-care sector capacity and capability.
• Unnecessary medicalization of SDOH.³

Gallego discussed two of these areas in detail: standardization and data sharing.

The Gravity Project was launched in 2019 with the goal of developing data standards for domains that Gallego described as grounded in a 2014 National Academies report.⁴ The domains include items such as education, elder abuse, environment, financial insecurity, food deserts, food insecurity, homelessness, housing instability, inadequate housing, interpersonal violence, material hardship, neighborhood safety, racism, social isolation, stress, transportation insecurity, unemployment, and veteran status.

The Gravity Project develops data standards to represent patient-level SDOH data documented across four clinical activities: screening, assessment/diagnosis, goal setting, and treatment/interventions. Described as a “public collaborative,” the project convenes participants from across the health and human services ecosystem, including clinical provider groups, community-based organizations, standards development organizations, federal and state government, payers, technology vendors, and others.

Gallego described the Gravity Project as having two workstreams: work on terminology, focused on SDOH domains; and technical work, focused on specifications for Health Level Seven International (HL7) FHIRs. The terminology workstream focuses on defining data elements for each SDOH domain by asking What concepts need to be documented across the four activities of screening, diagnosis, goal setting, and interventions? What codes reflecting these concepts are currently available? and, What codes are missing? On the technical side, the HL7 SDOH clinical care FHIR implementation guide provides guidance on how to do assessment screening,

___________________

³https://www.nasdoh.org/wp-content/uploads/2020/08/NASDOH-Data-Interoperability_FINAL.pdf.

⁴https://www.nap.edu/catalog/18709/capturing-social-and-behavioral-domains-in-electronic-health-records-phase.

how to capture health concerns or problems that inform the diagnosis, goal setting, interventions, capturing consent, and aggregation for data exchange and reporting.

When the Gravity Project was launched, according to Gallego, its leaders developed a conceptual framework that accounts for various entry points for the data (e.g., a digital application used by a patient or the health care providers’ electronic health records). They defined SDOH data concepts that can be documented and shared across the four activities discussed above, regardless of the initial input system. This framework emphasizes the value of these data for secondary use by public and private payers, social service providers, public health entities, and researchers.

Rachel Richesson, University of Michigan, discussed the concept of a learning health care system, where research influences practice and practice influences research. Standards could provide the infrastructure for turning real-world data into real-world evidence, and thereby be the foundation for enabling real-world evidence to influence practice.

Richesson shared her perspective on the key data needed for PCOR and the associated standards to consider. Box 2-1 summarizes these data types and relevant standards. She argued that there is a need for robust data that describe patients and patient populations, including patient problems, in a standard way, with up-to-date problem lists. There is also a need to capture treatments and interventions, broadly defined. Richesson said that patient goals and preferences are increasingly important. Standards exist for some clinical domains, but they are not widely used. Richesson argued that the outcomes and endpoints most useful for PCOR are those that are condition-specific.

In terms of standards (the right-hand column in Box 2-1), Richesson said that it is important to have terminology that can represent related concepts at different levels of granularity while at the same time being suitable for being combined analytically. She highlighted the Gravity Project and the Gender Harmony Project as approaches that start with the high-level question of what concepts need to be measured and then look at what data are available and how those data can be pulled together in a useful way. The information is then shared with those who need to implement the standards. She cited SNOMED CT as the recognized standard for nursing data, such as nursing goals, nursing diagnosis, and nursing-related outcomes, and also noted that a set of standards is quickly emerging from the work of BPM+ Health group, an organization that is modeling clinical pathways and clinical workflows.

Richesson summarized her perspective on the key steps for developing standards as the following:

1. Leverage processes of existing standards organizations;
2. Encourage patient engagement in standards developing organizations;

3. Support tools for use of standards in real-world settings;
4. Promote semantic interoperability and intrinsic value sets or “groupers”;
5. Use concept maps/information models (e.g., recent nursing work⁵);
6. Show value of standards for application development and dissemination;
7. Create tools to make it easy for application developers of all types;
8. Develop standards roadmaps;
9. Make use of feedback loops on standards (e.g., to understand whether they are useful or whether they are granular enough); and
10. Repeat the steps above, as needed.

Patrick Ryan, of both Janssen Research and Development and Columbia University, discussed data standards based on his experiences with the Observational Health Data Sciences and Informatics (OHDSI) collaborative. OHDSI is an open, multistakeholder, interdisciplinary collaborative whose mission is to improve health by empowering communities to collaboratively generate the evidence that promotes better health decisions and better care. OHDSI is driving development and adoption of open community data standards, open-source analysis software, and open-science best practices among regulators, academia, industry, payors, and health systems. While OHDSI is not a data standards organization, it is a heavy user of those standards and a steward of them simply because they advance the collaborative’s goal of generating reliable evidence to improve health.

OHDSI has a data network that includes organizations with patient-level data in more than 150 databases. Standardization, Ryan said, is about structure, content, and learning from the differences among data sources, rather than trying to create something that is homogeneous. The data in this network are not centralized, and organizations can decide whether and how to participate in the network. Researchers can conduct “network studies” by identifying a research question and then reaching out to the data network to generate standardized aggregate results.

Ryan argued that data standards are a means to an end, not an end in itself. In this context, the key questions to ask would be (1) What evidence would be useful to improve health policy and health care, which could be reliably generated by the PCOR data infrastructure? (2) How can data standards enable real-world analytics to meet the relevant evidence needs moving forward? and (3) What needs to be prioritized?

___________________

⁵ B.L. Westra et al. (2020). A refined methodology for validation of information models derived from flowsheet data and applied to a genitourinary case. Journal of American Medical Informatics Association, 27(11), 1732–1740. doi: 10.1093/jamia/ocaa166. PMID: 32940673.

Ryan said that the OHDSI community has given a lot of thought to the topic of reliable evidence. Data standards are necessary, he pointed out, to enable replicability, generalizability, and robustness, because answering questions requires that disparate data be brought together, and there is a need for a mechanism that enables examining the same question in different data sets in some reliable way. It is also clear that data standards without standardized analytics are not sufficient to ensure reliable evidence. Ryan also argued that public health questions require global data to generate global evidence, and standards that are not limited to data in the United States are more useful, even for research specific to the population in the United States.

Thinking about the specific role of standards in generating evidence, and working backward from the question of what evidence would be helpful to improve health policy and health care, Ryan discussed three analytical use cases on which the OHDSI community focuses: (1) characterization, or producing descriptive statistics to understand what is happening in the world; (2) estimation, or causal inference, to understand the effects of medical interventions and the comparative effectiveness of interventions; and (3) prediction of risks. Data standards need to be based on the evidence needed and the type of use case it needs to support.

The ecosystem of complementary standards discussed by Ryan includes data standards to enable data exchange (e.g., the FHIR from HL7), data standards to harmonize data structure and enable analytics (e.g., the Observational Medical Outcomes Partnership Common Data Model or OMOP CDM⁶), and analytics standards to generate and disseminate evidence (e.g., the Health Analytics Data-to-Evidence Suite or HADES). Underlying all of this, there are vocabulary standards that harmonize data content and also enable analytics (e.g., ATHENA). The OHDSI community works on bringing these standards together. For example, they developed a suite of open-source analytic tools that sit on top of the OMOP CDM as their open community data standard, but they are also collaborating with HL7 on the interoperability of FHIR standards and the OMOP CDM standards for enabling analytics.

Ryan said that the initial focus of the OMOP CDM was on health systems data, clinical data, and health economics data, because those are the analytical use cases and the evidence needs that OHDSI is trying to fill within their community. Underlying the data model are the infrastructure of vocabulary standards and the mappings from source codes onto standards, which enable the adoption of standards. He added that the use of vocabularies is not just an obligation, but an opportunity to expand the

___________________

⁶ For an overview of the CDM, see https://ohdsi.github.io/TheBookOfOhdsi/CommonDataModel.html#fn20.

value of the data. The OMOP data partners have different data structures and different data contents. Each partner goes through its own journey to standardize its data under a common data model, and OMOP enables those data to be analyzed through standardized analytic routines.

Ryan said that the OHDSI community has done a lot of work to try to help policy makers, regulators, and clinicians with questions related to COVID-19. One of the first efforts was related to examining the safety of hydroxychloroquine. Starting with the need, they reached out to their data network across the world to find out who had standardized data that could meaningfully contribute to this question. They then worked on developing the right analytic approach to apply to the question. While this work ultimately resulted in academic publications,⁷ the primary goal was to generate evidence that could inform policy decisions and the work of the European Medicines Agency and the U.S. Food and Drug Administration (FDA).

VG Vinod Vydiswaran, University of Michigan, focused on the role of natural language processing (NLP) in the use of PCOR data. He argued that three of the areas deserving attention to make the PCOR data infrastructure more useful are (1) the informatics infrastructure that includes clinical notes, (2) computable phenotypes as knowledge objects, and (3) looking beyond electronic health records for health data.

To illustrate the use of clinical notes, Vydiswaran discussed his work with the Patient-Centered Network of Learning Health Systems (LHSNet), a Clinical Data Research Network funded by the Patient-Centered Outcomes Research Institute. The common data model for LHSNet focuses on structured data typically available in clinical settings, such as demographic information, laboratory values, and ICD-9 and -10 codes. Vydiswaran worked on extending the common data model to include textual components and extract clinically relevant information from free text. This work built on a prior study, the Clinical Language Engineering Workbench (CLEW), developed as part of the National Program of Cancer Registries of the Centers for Disease Control and Prevention.

Figure 2-1 shows the CLEW NLP machine learning process functionalities. Vydiswaran noted that it is important to expand the component that creates a pipeline for extracting features, not only to look at individual attributes but also to look at concepts and relationships between concepts.

Vydiswaran said that the second area of work where advances could make the PCOR data infrastructure more useful is the use of computable phenotypes as knowledge objects, specifically standardized definitions for analysis across multiple sites. By knowledge objects, he meant the

___________________

⁷https://www.medrxiv.org/content/10.1101/2020.04.08.20054551v2; https://www.thelancet.com/journals/lanrhe/article/PIIS2665-9913(20)30276-9/fulltext; https://academic.oup.com/rheumatology/article/60/7/3222/6048420.

computational components of research that can be used and expanded on by others. This includes standardized clinical natural language tools for processing text so that it is interpreted the same way across multiple sites.

The typical data elements in computable phenotypes, Vydiswaran said, are structured components such as ICD-9 and -10 codes, Current Procedural Terminology (CPT) codes, information about medications, and sometimes key terms and phrases, along with the frequency of their mentions. Novel areas for consideration include patient-reported outcomes such as symptoms, medication response, and adverse events in telephone notes, medication refill requests through web portal requests, and care provider information, especially for patients unable to independently manage their

___________________

⁸https://aspe.hhs.gov/system/files/pdf/259016/NLP-CLEW-UserGuidanceDocument-508.pdf.

health care needs. A lot of information about patient-reported outcomes is available in electronic health records in text form.

Vydiswaran mentioned his prior work on self-reporting behavior concerning the toxicity of oral anticancer agents in clinical notes.⁹ In that work, he found that 23.5 percent of the clinical oral anticancer agent toxicity notes were based on telephone encounters. In another study, Vydiswaran and his colleagues are working on extracting patient-provided information on Crohn’s disease symptoms, medication response, and adverse events using email and telephone notes stored in electronic medical records.

Vydiswaran also encouraged looking beyond electronic health records for health data. For example, information on the adverse effects of drugs can increasingly be found on social media. However, Vydiswaran noted that processing consumer-generated text is even more challenging than processing clinical text, due to the prevalence of grammatical errors, typos, new acronyms, and abbreviations.

Text found on social media can be useful for a variety of purposes, beyond collecting data on drugs’ adverse effects. For example, geo-located social media can be useful for exploring community health information. Social media can be analyzed through the lens of communities (e.g., affluent versus disadvantaged neighborhoods), demographics (e.g., “BlackTwitter”), or patient cohorts (e.g., smoking cessation patient groups). Spatio-temporal factors that can be linked to patients include air pollution, neighborhood walkability, rurality, and “food deserts.”

Vydiswaran mentioned that his current work includes the use of social media to augment information in the FDA Adverse Event Reporting System and the Vaccine Adverse Event Reporting System. NLP can be helpful for parsing text from both social media and the federal adverse event reporting databases.

As a summary of his key points, Vydiswaran emphasized (1) the need for an enhanced informatics infrastructure for processing textual clinical notes; (2) treating computable phenotypes as knowledge objects, and incorporating patient-reported outcomes derived using NLP; and (3) taking advantage of health-related social media to augment existing data.

After the presentations, participants in this session were asked to comment on what the federal government could do to accelerate work on standards for patient-centered outcomes data and research. Halamka said that one of the strengths of the federal government is that of a convener of meetings. As an example, he mentioned the role of the Office of the

___________________

⁹ Y. Jiang, V.G.V. Vydiswaran, Y.L. Eun, H. Joo, A. Zheng, and M.R. Harris. (2018). Feasibility of Identifying Oral Anticancer Agent Toxicity Self-Reporting and Management Advice from Clinical Notes. AMIA Annual Symposium, November 3–7, San Francisco, CA.

National Coordinator for Health Information Technology in convening meetings to attempt to harmonize U.S. and international standards on vaccine credentials. He added that he does not believe that attempting to regulate standards would be advisable. If the federal government facilitates the harmonization of standards, those standards will be adopted because they will bring value. Grannis agreed that convening and building consensus is what the government does well. He argued that this would be especially useful for accelerating work in areas that are “hotspots” of nonstandardized data, such as SDOH.

Gallego highlighted creating incentives for testing the standards as a need that could be filled by federal or state governments, through mechanisms such as grants. Richesson argued for a library of computable phenotypes. She noted that there is a need for better organizing information on existing standards. ASPE could facilitate the sharing of tools and metadata standards and incentivize the reporting of results. She said that currently it is almost easier to develop new standards than to find something that has already been done and that would work in a particular situation. Ryan agreed with the idea of a phenotype library, and argued that the scope could be broadened to other types of information about what was learned from a study. He also agreed with Gallego on the need for testing, and with Halamka and Grannis on ASPE’s potential role in community building. Vydiswaran highlighted the need for maintaining analytic tools over the years, by updating documentation to facilitate their use and progress toward eventual standardization.

DISCUSSION

During the workshop, the formal presentations were followed by additional discussion among the workshop participants, including the speakers, committee, and audience members. Among the topics that were explored in further detail were arguments for and against additional standards. One of the themes that emerged is the inherent difficulty associated with the process of agreeing on standards. It could be argued that an adequate variety of standards already exists for most situations, and the challenge is to converge around a single standard. Despite the challenges, past experience in a variety of domains indicates that convergence can be accomplished by bringing together communities and increasing communication around these topics. Participants also acknowledged that the context around standards is continuously evolving because the use cases, workflows, and information flows are always changing. This means that standards will also have to evolve over time.

Another topic that was discussed by workshop participants was the role of the federal government, and particularly HHS, in the context of

standards for PCOR. Convening stakeholders emerged as a role that is especially well suited for HHS to handle. Participants cautioned against the blunt instrument of regulation, arguing that standards are most likely to be adopted when they bring value, because without a clear purpose and value for the standards, clinician frustration with electronic health records could increase. HHS could play a role in facilitating discussions to prioritize areas where standardization could be most useful and convening activities around topics such as SDOH, where there is a notable lack of standards and a common language.

Speakers also highlighted the need for a support infrastructure that would facilitate activities such as testing and enable the adoption of standards. Examples of the support infrastructure focused on building community and collaboration around standards. At the same time, views differed on a potential role for HHS in incentivizing the use of standards, and what those incentives could look like.

Workshop participants noted that there was a need for cataloging existing standards, because currently it is almost easier to develop a new approach than to figure out what already exists and whether and how it applies to a particular situation. This need extends to the cataloging of analytic tools, which are easier to use when documentation from earlier studies is maintained and updated.

Another theme addressed by the speakers and revisited as part of the discussion was that of the relationship between U.S. and international data standards. Participants noted that there are a variety of differences among countries, ranging from population differences to differences in health care systems. The use of standards also differs, but common international standards would facilitate a better understanding of the heterogeneity and could inform policy decisions everywhere, including in the United States.

CONCLUSIONS

The workshop session demonstrated that, within the context of PCOR, standardization is increasingly and ever more widely applied to the processes of collecting, storing, analyzing, and exchanging data. These standards are most useful when they are focused on addressing a specific problem or are driven by a specific use case. The needs and norms evolve over time, and because of this, standards need to evolve too. Lessons might be learned from best practices that emerge for the development of standards.

The workshop made it clear that in some areas there is a fair amount of agreement around what standards are needed and what useful standards look like. In other areas, for example, for data on SDOH, the work is just beginning, so these areas might not be ready for wide agreement on standards. In all cases, extensive testing of the potential standards is necessary.

CONCLUSION 2-1: Standards are most useful when their development is driven by their potential uses and a clear concept of the value they can contribute.

Participants in this session did not see a large role for ASPE in developing standards or deciding what the standards should be. ASPE’s most valuable contributions could be in developing an architecture and an implementation strategy that facilitate common language and interoperability across data sets, as well as accessibility of the data. Other areas where ASPE could play an important role include convening stakeholder meetings to discuss and develop standards, as well as taking the lead in cataloguing existing standards.

CONCLUSION 2-2: The Office of the Assistant Secretary for Planning and Evaluation could add significant value in the area of standards for patient-centered outcomes research by

• continuing to promote the development of a data infrastructure and an implementation strategy that facilitates the use of standards and access to the data;
• convening stakeholder meetings to enhance communication and work toward developing a common language for standards;
• facilitating accessibility to the data and collaborations with existing organizations working in this area; and
• leading efforts to catalogue and exemplify data standards and analytic standards.

The speakers touched on the need for a broad interpretation of standards, to include not only the data but also the methods used to analyze PCOR data.

CONCLUSION 2-3: While data standards are important to conducting patient-centered outcomes research, applying standards to the analytic methods as well is important to facilitate the reliability and reproducibility of study results.

Speakers also highlighted the potential benefits of staying abreast of the standards development that happens not only in the United States but also internationally. Learning from experiences across the globe would further advance PCOR and benefit patients in the United States.

CONCLUSION 2-4: An international perspective is an important consideration for the patient-centered outcomes research data infrastructure, and the infrastructure focused on standards specifically would benefit from building on work that happens internationally.

3

Methods for Patient-Centered Outcomes Research

The workshop discussions summarized in this chapter cover methods that could advance patient-centered outcomes research (PCOR) and make PCOR data more useful going forward. The brief overview of the input received from the presenters is followed by the committee’s conclusions. Speakers in this session were asked to focus on the following questions:

• What emerging methods are likely to be most relevant for the PCOR data infrastructure looking forward? What are the most important research and data challenges?
• What computing advances, innovative health information technologies, and methodologies might present opportunities going forward?
• What role can the Office of the Assistant Secretary for Planning and Evaluation (ASPE) play in supporting effective methods for PCOR studies? What characteristics of HHS’s public mission, programs, or authorities could be leveraged?

Nigam H. Shah, Stanford University, argued that to make better use of patient-centered outcomes data, it would be useful to adopt a patient timeline view of the data. Typically, data are thought of as residing in tables, text files, images, and so on, but health care happens over time, and it is useful to think of the data in those terms, as events occurring over time.

Figure 3-1 illustrates Shah’s patient timeline perspective. The figure shows a patient’s journey, with the red dots denoting events with some health care relevance. Depending on which access point into medical data

they use, researchers only get a partial view of the event that really occurred. For example, they might have access to claims filing information corresponding to a medical visit, which might have International Classification of Disease (ICD) codes, medication codes, Common Procedural Terminology (CPT) codes, and laboratory test orders. From the electronic health records, researchers might be able to access test results, the clinician’s notes, and perhaps the signal streams from bedside monitors. Outside of the health system, we might get data from wearables, such as a Fitbit or an Apple watch. In a research setting, we might find gene expression data, genomic data, and perhaps more kinds of molecular measurements. Outside of the context of medical care, researchers might have access to information about online activities, such as phone usage, browsing, social media postings, audio clips, and so on.

Shah said that artificial intelligence (AI) or machine learning may make it easier to automate the processing of the types of data included in the timeline view. A more innovative use of AI is to combine multiple data modalities to trigger some proactive action.

Shah pointed out that few data source systems in routine use have native constructs for handling a task such as, for example, finding patients with a history of myocardial infarction who have pneumonia. This query has to be programmed in SQL (Structured Query Language) programming language, which is currently the dominant mechanism for interacting with this type of data. To be able to make full use of timeline data, it is also necessary to have tools that can perform interval algebra. The next step after performing interval algebra would be navigating knowledge graphs.

For example, one might want to find patients with disorders of glucose metabolism or patients with disorders of glucose metabolism treated with certain types of medications. The challenge is that current data systems do not handle medical knowledge graphs, and manual coding is necessary.

Shah also discussed the need to be able to state phenotype definitions, in other words, the necessary and sufficient conditions for believing that a particular event of interest occurred. As an example, hypertension as a phenotype might imply a different blood pressure cutoff, depending on the decade. “Ventilator days” was another example, mentioned in the workshop in connection with the COVID-19 pandemic. Shah noted that the complexity of phenotype definitions ranges from the collection of codes to elaborate Word documents that need to be translated into SQL.

Shah summarized his points as the need for technology that allows researchers to go from timelines to data frames in real time, taking timeline objects (with their as-yet unsolved storage challenges) and performing advanced analytics, satisfying the necessary and sufficient statements to conclude that a particular exposure outcome happened, and producing an analysis data frame in real time. He argued that solving these challenges would greatly accelerate PCOR.

Shah also discussed the Advanced Cohort Engine (ACE), a search engine he and his colleagues developed for patient data.¹ The search engine consists of a persistent in-memory database of patient objects and a temporal query language, both optimized for fast search, and a flexible application programming interface to access and retrieve data. Researchers can quickly find patients by searching across diagnosis and procedure codes, concepts extracted from clinical notes, laboratory test results, or vital signs, as well as by visit types and duration of inpatient stays. They can then compare the outcomes of these patients. Shah mentioned that the search engine is available with the Observational Medical Outcomes Partnership Common Data Model (OMOP CDM) version 5.3 and higher, emphasizing the importance of adhering to at least one community standard so that going from data to analyses is reproducible, reliable, and scalable.

Shah underscored the need to upgrade the collective computational infrastructure in the United States to be able to conduct the types of analyses he described in real time. There is also a need for a stronger focus on systems and software, beyond methods development.

Sharon-Lise Normand, Harvard University, highlighted data silos as one of the main challenges for PCOR data and PCOR in general. There are a large number of data sources, the usability and availability of unique IDs

___________________

¹ A. Callahan, V. Polony, J.D. Posada, J.M. Banda, S. Gombar, and N.H. Shah. (2021). ACE: The Advanced Cohort Engine for searching longitudinal patient records. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocab027.

are questionable, and linkages across databases are difficult to accomplish. She agreed with Shah that understanding what works requires longitudinal observation of patients over time.

Normand pointed at missing data as another challenge for PCOR. While missing data has always posed difficulties for statisticians, it is important to understand what this means specifically for electronic health records and to consider solutions for irregularly spaced data. She also noted the added challenges of missing data when more than one source is linked, when dealing with large volumes of data, and when using machine learning approaches.

The large number of data sets and new tools for processing the data also bring new challenges in terms of uncertainty over the precision of the estimates. One issue is selective inference, and the numerous decisions required as part of the analysis, because this ends up making the findings difficult to reproduce. Normand noted that currently no good methods exist for dealing with the propagation of differing errors associated with the use of multiple, complicated data sets. She argued that it is necessary to have an honest reflection concerning uncertainty and that there is a need for transparency regarding the assumptions and decisions that are made as part of the analyses.

Normand also discussed several areas where methodological opportunities exist for PCOR. The first such area is clinical trials. The availability of large volumes of electronic data will make it possible to streamline approaches for adaptive trials, which are difficult to conduct. Developing parallel randomized and prospective observational studies, using the same database and the same cohort receiving treatment at the same time, can also increase ability to learn about how effective certain treatments are. Finally, the usefulness of the data could be improved if adjustments were developed for reporting non-blinded outcomes, such as when participants are asked to complete a questionnaire.

The second area where opportunities exist for advancing PCOR is causal inference and the adoption of experimental thinking. Normand encouraged more emphasis on designing studies, as opposed to simply focusing on the analysis. Integrating causal inference approaches would be particularly useful in pragmatic trials. Opportunities exist to better understand the implications of missing data in sparse data settings and to better understand uncertainty and error propagation for the estimates.

Finally, Normand argued that exploiting the connectedness of information that is available from observational settings represents another methodological opportunity. She described this as a longitudinal multitask approach, sharing information across devices and across patients.

Addressing what ASPE could focus on, Normand said that the new Bridge2AI funding opportunity announcement from the National Institutes

of Health (NIH) made her think that it would be useful to have “on demand” data, for example data that are already linked. Building a trial infrastructure for randomized and observational studies would be helpful. She also argued that to be able to make valid inferences about patient-centered outcomes data, it is necessary to invest in statistical methodology.

Sherri Rose, Stanford University, began by talking about data transformations in cases where it might be necessary to intervene if the information available represents structural biases in the collection of those data. Rose mentioned a recent paper she coauthored with colleagues on the ethical use of machine learning in health care.² The paper discusses the potential for data tools to exacerbate existing health disparities, along the different steps in the process from problem selection, through data collection and outcome definitions, and finally to algorithm development and potential postdeployment activities. She said that the preprocessing steps are where data transformations would typically be considered, but a lot of the work can also happen in the algorithm-building stage, or in the post-processing stage, where one might decide, for example, to adjust thresholds to handle concerns about the data. Rose said that questions to ask include Who decides the research questions? Who is the target population? and, What do the data reflect?

Rose discussed her prior work on using data transformation to bring causal conceptual thinking to the matter of fairness in data infrastructure.³ The work focused on payment systems, aiming to reduce disparities in low-income neighborhoods and underprovision of services for chronic conditions, and the idea was to develop a methodology to set policies at desired levels. Rose added that thinking about how to do these types of data transformations is challenging, but this methodology is underleveraged and could benefit data infrastructure.

In the context of data linking and causality, Rose discussed a collaboration she had undertaken with Normand, in which they linked claims, registry, and vital statistics data to study the comparative effectiveness of cardiac stents.⁴ Using machine learning made it possible to identify heterogeneous effects for the safety outcome in a cohort of patients receiving percutaneous coronary interventions. One piece of information they did not have was a

___________________

² I.Y. Chen, E. Pierson, S. Rose, S. Joshi, K. Ferryman, and M. Ghassemi. (2021). Ethical machine learning in healthcare. Annual Review of Biomedical Data Science, 4(1), 123–144. https://www.annualreviews.org/doi/abs/10.1146/annurev-biodatasci-092820-114757.

³ S.L. Bergquist, T.J. Layton, T.G. McGuire, and S. Rose. (2019). Data transformations to improve the performance of health plan payment methods. Journal of Health Economics, 66, 195–207. https://doi.org/10.1016/j.jhealeco.2019.05.005.

⁴ R.S. and S.-L. Normand. (2018). Double robust estimation for multiple unordered treatments and clustered observations: Evaluating drug-eluting coronary artery stents. Biometrics, 75, 289–296. https://doi.org/10.1111/biom.12927.

reliable estimate of the operator’s skill, so they controlled for the operator in that study. She said that technology now exists to film operators doing surgery and produce an estimate of operator skill based on the footage to augment existing data. This technology might not scale well to large data sets, but it is an area with a lot of potential.

Generalizability was another topic touched on by Rose. While generalizability is typically considered in the context of inference, she argued that it is also useful to think of generalizability in prediction and clustering. The literature on generalizability from disparate data sources is spread across a variety of fields, including computer science, statistics, and health sciences. She mentioned a review she completed with one of her students and noted that there is substantial work yet to be done in this area.⁵

In some of their current work, Rose and her colleagues are focusing on integrating randomized and observational data, where each data source contains individuals who were missed by the other source with respect to the covariate distribution. In this work, they are focused on the area of overlap between randomized and observational data to develop new estimators and leverage the probability of selection into the randomized trial and the probability of receiving the treatment or receiving the intervention.

Rose echoed some of the conclusions highlighted by other speakers regarding opportunities to enhance PCOR. This includes the need to support work on developing new databases and software as well as maintaining existing software. She emphasized as well the importance of supporting the development of creative new methods for building data infrastructure. In closing, she urged researchers to consider whether their algorithms have a social impact statement. In connection with building tools for a data infrastructure, she named several social impact principles, including responsibility, explainability, accuracy, auditability, and fairness.

Nirosha Mahendraratnam Lederer, of Aetion, said that the company’s Aetion Evidence Platform puts real-world data on a patient timeline, and uses transparent and scientifically validated workflows to analyze the data to generate real-world evidence. All the company’s designs and analysis considerations are maintained in an audit trail.

Concerning real-world data, Lederer said that many variables of interest are available from traditional sources, such as medical and pharmacy claims, hospital chargemasters, electronic medical records, and clinical laboratory results. Additionally, newer digital tools, such as mobile health apps, sensors, and wearables enable the collection of additional patient-generated data. Linking traditional and newer digital sources makes it possible to generate a fuller picture of a patient, including the different factors

___________________

⁵ I. Degtiar and S. Rose. (2021). A Review of Generalizability and Transportability. https://arxiv.org/pdf/2102.11904.pdf.

that ultimately impact the person’s well-being, beyond physical health and traditional clinical outcomes. This enables researchers to study questions that patients really care about and provide information that can address issues with health care delivery, as well as structural and societal challenges.

Lederer suggested that the key for successfully using real-world data for research is to start with a well-defined research question that can be studied in the real world, and to ensure that a data source suitable for answering the question is available. However, the right data are not always available. For example, data that would allow researchers to study health disparities are rarely usable. Lederer said that there are two main reasons for challenges of this type.

The first reason why the available real-world data do not always include all the necessary data elements is that they are typically collected for purposes such as claims for billing, and not targeted for research purposes. As a result, these data sets are often missing key variables necessary to generate high-quality evidence. Lederer suggested that identifying a minimum set of core data elements to collect in routine clinical care can enable more meaningful research and facilitate data linkages. She added that building on existing tools and initiatives instead of creating new programs might be most practical. For example, instead of creating bespoke sets of required minimum data elements, perhaps the Office of the National Coordinator for Health Information Technology could collaborate with the research community to augment the United States Core Data for Interoperability (USCDI) to include essential data elements for research. Lederer acknowledged that USCDI applies only to electronic health records, and suggested that perhaps voluntary community standards could work for other types of data. To encourage adoption, it would be important to engage with digital health companies and organizations such as the Digital Medicine Society, because high-quality standardized data entry can make a big difference in the suitability of data for research.

The second reason why real-world data sometimes do not include data elements necessary for research is that privacy laws or commercial interests may be restricting the accessibility of these data, even when the data elements are captured. Lederer said that in an effort to protect privacy, frequently the tradeoffs are between information on race, geographical granularity, and place of service. All three of these data elements are essential for addressing issues with health care quality and disparities. Given technological advances, and the way data are collected, used, and linked today, Lederer said that it is important to revisit these policies in the context of research.

Another challenge highlighted by Lederer is that often only researchers within a health care system have access to certain data, and external researchers do not. This brings up the issue of parity, and the benefits that

would result from more researchers with different perspectives and ideas having the opportunity to test their hypotheses using these restricted data sets.

Lederer also highlighted challenges associated with carrying out analysis using real-world data, even when the data are available. She pointed out that not all real-world data translate to high-quality evidence. However, there are some clear principles for generating high-quality evidence that is patient-centered and suitable for decision making. This includes starting with the concept of designing the target randomized controlled trial that one would conduct to answer the research question, and then emulating that trial through an observational study.

Over the longer term, Lederer said that she supports making research an infrastructure investment for state-of-the-art data curation and analytics such as AI or advanced methods to aim to quantify and adjust time-varying unobservables. However, in the more immediate term, it would be useful to think about how to refine the operationalization of valid research. Lederer suggested organizing, evaluating, and incentivizing the use of PCOR and real-world evidence tools to promote the generation of decision-grade real-world evidence. She added that this could essentially be a real-world PCOR toolbox. While many of the tools that would need to be included in such a toolbox are available today, many researchers do not know that they exist or how to find them. Making a toolbox readily available could also result in the development of a framework for conducting real-world PCOR.

Lederer noted that the development of new tools is also necessary. For example, raw data need to be transformed into variables that can be analyzed. For federal data sets that are available for public use, this could involve creating standardized, validated measures and measure sets to help operationalize and augment the use of the data. It would also be useful to create disease-specific tools, which could include master protocols for real-world evidence that centralize the relevant expertise needed for high-priority research questions. One potential model that could be leveraged is the Reagan-Udall Foundation for the U.S. Food and Drug Administration (FDA) in collaboration with Friends of Cancer Research’s COVID-19 Evidence Accelerator, which convenes health care stakeholders to use a common data shell and protocol to run analyses for high-priority research questions for COVID-19.

The creation of a toolbox would not be easy, Lederer acknowledged. Part of the process would be reviewing all of the available resources, triaging them, harmonizing them, and identifying how they can be used for regulatory decision making, clinical decision making, policy decision making, or personal decision making.

Lederer also emphasized the importance of transparency for building credibility to advance the science of real-world PCOR. This includes data

transparency, protocol transparency (e.g., preregistering studies to address potential concerns over hacking and data dredging), and publishing the results regardless of the outcome.

Lederer argued that science should not be proprietary. She highlighted the benefits of a culture where inferential protocols are made publicly available, not only in the interest of reproducibility and replicability, but also because otherwise one might not have access to the full scope of expertise necessary to conduct a high-quality study. She said that transparency could be accelerated through incentives, such as tying access to federal data sets or federal funding to the registration of the studies and the publication of the research protocols and results.

Lara Mangravite, Sage Bionetworks, focused on the governance component of the data infrastructure, discussing issues related to the governance structures used to enable research that typically involves data from multiple sources. She briefly described the governance structure of the National Center for Advancing Translational Sciences (NCATS) National COVID Cohort Collaborative (N3C), which assembled medical records from approximately 65 medical centers from across the country into one central repository with the purpose of using it for research.

Mangravite noted that, typically, one of the important roles within governance structures is that of a data steward. The data steward is responsible for the technology that allows the data to be managed and used and for the legal agreements between the data donors and the data users. The N3C effort, Mangravite said, illustrated the challenge of integrating data across systems. She argued that data interoperability between health systems and data linkage across sources requires increased investment in data quality standards.

Mangravite also discussed the evaluation of care, which in her view often requires integrating data on lived experience beyond the data captured in the medical system. Person-centered research necessitates data that comes directly from the patient, in part because clinical care is impacted by a variety of factors not captured in the medical record. She noted, however, that obtaining data on individuals’ lived experiences increasingly involves private and sensitive information, and that it is important to consider the value proposition from the perspective of the individual providing the data. Sage Bionetworks works with a lot of sensor data, and Mangravite pointed out that a person’s characteristics impact what researchers see, for example, in the accelerometry data from someone walking down the street. This has implications for how the data are analyzed, but also how the data are managed and what the privacy considerations and value proposition are.

Mangravite said that most discussions of data sharing involve researchers acting as data stewards and exchanging data. These discussions typically do not involve the individuals whose data are shared. The expectation is

that these individuals would simply need to trust the data governance to happen in their interests. This might not be an acceptable situation, given that for many types of data the risks and benefits are not well understood. Mangravite argued that approaches to data governance need to change; they need to go beyond simply making the informed consent processes more dynamic and involving the people being represented in the data in the practice of the research itself, throughout the lifecycle of the studies. This is especially important because the implications of the use of the data are not always clear at the stage when the data are collected. Involving the individuals providing the data in the study would at a minimum ensure transparency about how the data are being used, and ideally would also allow room to impact decisions related to the data use.

One of Sage Bionetwork’s current projects, called the MindKind Study, aims to identify self-management strategies that might work for youth with anxiety or depression. Mangravite said that to observe youths’ mental health states in combination with self-management strategies or other activities they may be engaging in, and to do that dynamically over time, requires a lot of data that are not found in medical records. She and her colleagues are examining whether integrating youth into the data governance and stewardship model impacts their willingness to participate. They asked youth and researchers in several countries a series of questions related to data governance (e.g., their preferences related to who can access the data, who controls the data, what kind of research can be conducted with the data, and so on). They found agreement between the youth and researchers on responses to many of the questions, including, for example, on who can access the data. However, perspectives differed on some questions, such as, for example, on who control the data. Mangravite said that they are now conducting a study that aims to better understand differences between what is preferred and what is acceptable, and looks at how potential changes based on what they learned would impact willingness to participate.

In summary, Mangravite highlighted the need for integration of data across systems, and the integration of participants into the research lifecycle, as two of the areas that need the most attention in terms of the data infrastructure. To integrate participants, her specific suggestions were to focus on enabling richer understanding of lived experiences outside of the medical system, support the alignment of research questions with community needs, and support capacity building for translating research outcomes into action.

DISCUSSION

As with the previous session, the formal presentations on PCOR methods were followed by additional discussion among the participants.

One topic that was explored in further detail and emerged as a key theme was the need for a more holistic “timeline” view of people’s experiences. The main challenges associated with developing longitudinal data sets are the costs and barriers associated with following and identifying people over time and with linking information from different sources. The fragmentation of the health care system and the lack of unique identifiers were reoccurring themes in the discussion of barriers. The limited availability of timestamps associated with the data that are available was also highlighted.

A theme that had been explored in detail in the committee’s first workshop and was revisited by the participants in this one was the need to broaden research perspectives from the patient to the person in a broader sense, bringing in additional data on factors that are outside of the health care provider system. Integrating relevant data that go beyond provider databases represents its own challenges, but a timeline view that expands beyond a person’s experiences within the health care system would greatly increase our ability to understand, for example, chronic diseases.

The topic of data privacy was also discussed, including concerns about the unknowns in the area of potential re-identification. The discussion echoed points made by the speakers in the previous two sessions, highlighting tensions related to different perspectives on whether fully de-identified data is a realistic goal, and whether access to identified data with strong security and penalties for misuse would be an option. This topic, and the committee’s conclusions, are addressed further in Chapter 4.

Another theme that emerged during the discussion was the importance of engaging patients in the research process and being transparent about the methods used to generate findings. Participants also discussed the need to balance the goals of transparency with the interests of stakeholders who would like to keep some of the information proprietary, and there are efforts under way to develop approaches that achieve this balance. However, widespread adoption would be more likely if there were incentives and a central repository in place.

CONCLUSIONS

One theme that emerged from the session on methods for PCOR was the potential usefulness of adopting a timeline or longitudinal perspective on understanding a person’s journey through the health care system, and through life events that have a relevance to health more broadly. Several changes could facilitate this shift, as summarized in Conclusion 3-1.

CONCLUSION 3-1: The ability to adopt a longitudinal, comprehensive perspective on an individual’s journey could open new opportunities for

patient-centered outcomes research. The shift could be facilitated by focusing on efforts to

• simplify the integration of data across the research data ecosystem;
• address challenges posed by the limitations associated with health identifiers;
• incorporate person-generated data into health data systems; and
• leverage real-world data to expand the timeline view of a person’s health-related experiences.

Speakers emphasized the need for transparency and for consideration of related scientific principles, such as reproducibility of the data and methods used for PCOR. These considerations are important for all types of data and analysis, but the increasing use of tools such as machine learning and natural language processing raises the question of whether best practices can ensure that these tools do not have negative social impacts.

CONCLUSION 3-2: Observing scientific best practices, including those of transparency and ethical use of data, is essential to generate trust in patient-centered outcomes research among all stakeholders, including the public and researchers. This is important both for observational data and for emerging data sources and methods.

The session on methods highlighted the importance of interpreting best practices around the dissemination of the research broadly. That is, best practices apply not only to the sharing of results but also to other resources and components associated with the research process, such as the software developed for analyses. Sharing all these resources ensures that the data can be widely used and that the research can be replicated. Ultimately the goal of patient-centered outcomes research is to benefit people, so the question of what happens to the research after it is done, and the sharing of the information with those whose data are being used, also deserves further attention.

CONCLUSION 3-3: The results of patient-centered outcomes research (and research in general) are only replicable and are most useful when the underlying data and comprehensive research documentation (such as analytic code) are made available for use by others.

4

Data Policies and Other Data Infrastructure Considerations

This chapter summarizes the workshop discussions centered on data policies and other data infrastructure considerations. Speakers in this session were asked to focus on the questions below. The chapter concludes with the committee’s conclusions.

• What data policies are likely to be most relevant for the patient-centered outcomes research (PCOR) data infrastructure looking forward?
• What role can the Office of the Assistant Secretary for Planning and Evaluation (ASPE) play in supporting these policies for PCOR studies?
• What characteristics of HHS’ public mission, programs, or authorities could be leveraged?

Pamela Riley, District of Columbia Department of Healthcare Finance, discussed policy considerations to support PCOR on the social determinants of health (SDOH). She focused on ways to address unmet social needs in health care systems and health care settings in order to improve health care delivery. This includes addressing basic resource needs both in the clinical setting and at an individual patient level, as well as using data to address population health issues. Riley noted that the District of Columbia Medicaid agency is currently working to help develop a community-level social needs screening referral and resource inventory and is also working on ways to support data collection, sharing, and use to improve clinical care delivery and population health.

Riley emphasized the importance of being aware of considerations specific to data on SDOH, including sensitivities around data collection, such as the issue of what the data are going to be used for. She noted that collecting data on social needs, such as whether a person has enough food to eat or has unmet financial needs, can be particularly sensitive from the perspective of those who are being asked to provide the data. Because of this, it is important to think about best practices for collecting these types of data to assure that the information is complete and can be reliably used to inform practice. Riley highlighted the related need to involve stakeholders at both the community and patient levels, as well as the need for transparency, noting that lack of trust often hinders efforts of this type and that involving stakeholders could potentially help address this. It is necessary, she said, to engage people, patients, communities, and other stakeholders every step of the way in designing and implementing data collection approaches and research strategies that people are actually onboard with and that will be reliable and useful.

Riley also noted the importance of considering ahead of time how the data will be used. Is it to inform clinical care delivery? Is it to inform population health at the health-system, national, state, or local level? Is it for academic research? The uses of the data need to be considered when thinking about policies around data infrastructures. A related consideration is what specific data are needed depending on the intended purpose. For example, what data does a hospital need in order to evaluate community-level interventions? What does a state need for planning purposes?

Data sharing was another topic discussed by Riley. Approaches to data sharing need to consider the sensitivities, and balance the need to reduce how many times people are being asked to provide the same information with the need to ensure that those who provide the data have confidence in giving permission for the intended uses, which could range from improving their own care to fulfilling a broader purpose.

In terms of interoperability and combining data from multiple data sources, Riley argued that clinical claims data that are being collected from nonclinical entities is one area that represents a challenge.

Concerning consent management, Riley considered this to be an area whose challenges need to be addressed in a systematic way that is broadly applicable. This is particularly important if the goal is to support whole-person studies that include data on physical health, SDOH, and behavioral health. She argued that an infrastructure needs to be put in place for consent management, to facilitate best practices in data sharing for research, and to improve patient care.

Riley pointed to a particular need to identify best practices that work at the local level, because data collection, data sharing, and interventions often have a local focus. She also emphasized the need to develop an

infrastructure for data collection by nonhealth care entities. Community-based behavioral health providers and mental health and substance abuse disorder providers, she said, are especially likely to be lagging behind in their capacity to implement electronic health records. Community-based organizations, in general, would benefit both from being able to participate in health information exchanges and from being able to share data in a standardized way.

Touching on the role of the federal government, Riley said there is a need to build an evidence base for what works. There is room for a federal voice in this, she argued, specifically concerning interventions that work to address SDOH related to health care delivery, what works to address needs and in what settings, and what works among which populations. There is also a need to better understand what data collection approaches are most effective in obtaining complete, reliable data. Related to that, there is a need to understand what gives people confidence and comfort in how the data will be used so that they are willing to share the information. More broadly, there is a need for an infrastructure that facilitates the involvement, at every step of the way, of those who are providing the data.

Abel Kho, Northwestern University, argued for a need to improve the quality of the identifying information being collected, something particularly important because the need for record linkages is becoming increasingly common. As an example, Kho mentioned the use of speeding cameras by the police and how that information is linked to other databases before a speeding ticket is mailed. He also discussed the City of Chicago’s use of Clearview AI for facial recognition, which has then been linked to additional data sets to identify potential criminal activity. Kho said that the use of these technologies raises questions that also have implications for health research. For example, in Chicago, areas with high crime rates also have high rates of chronic disease. Kho echoed Riley’s points related to being thoughtful about why the data are collected, who they are collected from, and what the needs of the communities are. He also emphasized the need for community input on data collection, for example in the case of whether and where to use street cameras.

Kho noted that the way people identify themselves impacts researchers’ ability to perform record linkages. He said that it is important to consider not only the implications of research bias, but also social bias. For example, gender information has historically not been considered useful for record linkages, because this information typically has binary values in electronic health records. However, the concept of gender identity has changed dramatically over the years, and it is now considered to be a much more complex construct. Data systems are not set up for capturing a lot of this information yet, but Kho said that collecting detailed data that reflects how people self-identify is important for a variety of reasons,

including understanding their sense of self-identity and to avoid discriminatory practices.

Those who are interested in data need to be thoughtful about identity concepts, which are evolving and are shaped by the social context, Kho argued. The information available in health records to identify a person for the purposes of record linkages is also constantly changing. While binary gender is captured in virtually all electronic health records, Kho and his colleagues have been noticing an increase in the availability of data on sex assigned at birth and sexual orientation. They also found that social security numbers are less and less likely to be available in electronic health records, while email addresses are increasingly available, and are becoming more useful for record linkage. Other types of information that might be captured and useful for record linkages include driver’s license images and information about the person’s occupation. Kho said that the latter is largely driven by COVID-19. He argued that it is important to think about how to balance considerations such as privacy versus a “big brother” approach, or category labels versus self-identity, in a dynamic, constantly shifting environment.

In Kho’s view, the data available today are subject to historical and social biases, even as the data and identities themselves are constantly changing. Therefore, it is necessary to understand when and how the data were collected. Kho said that policies can help with data hygiene, that is, with standardizing data collections, which is easier to do than addressing data bias. He underscored that data bias is not solely a technical issue. To address it requires engaging stakeholders early and often, particularly among at-risk or hidden populations.

Julia Adler-Milstein, University of California, San Francisco, focused her remarks on the need to use policy levers to advance interoperability, which she described as applicable to many of the topics discussed throughout the day, including data standards, consent management, approaches to identification and patient matching, governance, and incentives to share data. She said that the main lesson from her work in this area is that meaningful progress cannot be made on interoperability if policy efforts, and in particular federal policy efforts, are limited to convening activities. She argued that there is a need for policy actions to address some of the market failures, in particular the lack of incentives to invest in the types of infrastructure discussed throughout the workshop. While recognizing that policy is a blunt instrument, Adler-Milstein said that the types of policies the federal government and states need to focus on are those that create strong incentives to engage in interoperable data sharing.

Although many capabilities have been developed for interoperability supporting PCOR, Adler-Milstein argued, the challenge remains getting those capabilities adopted at scale. She said that to facilitate such adoption,

there is a need to measure and incentivize conformance to existing standards. An example, she mentioned the patient-reported outcomes Fast Healthcare Interoperability Resources implementation guides. These standards are available, but they are not federally required, and there are no test tools that would make it possible to determine whether someone is actually conforming to the standards.

Adler-Milstein also discussed the United States Core Data for Interoperability (USCDI) as a policy vehicle to promote scale. USCDI is a set of data elements that health systems must make available through an application programming interface (via their electronic health records), and that set of data elements will, by definition, expand over time. Thus, Adler-Milstein said that data that are included in USCDI are expected to become available at scale. She added that the data also need to conform to specified standards, but that it is not clear to what extent this will be measured and enforced. Without such measurement and enforcement, there is a risk that a lot of manual work will be required to make use of the data, despite widespread availability.

Adler-Milstein provided several examples of how the framework she suggested could work in several domains. In the case of patients, robust identity data are needed, because when these data exist and they are con-formant, it is possible to perform data matching across sources. Demographic data exist within USCDI, and they have been part of the first wave of required data elements. What is needed now is ongoing conformance assessment and incentives to address poor conformance.

On the patient-centered side, there is a need to support efforts that advance robust identity matching across data sources, in part to overcome the challenges posed by data fragmentation, but also to address the need for longitudinal data. Some examples of projects that are targeting identity matching to support more comprehensive and longitudinal data are Gravity and the Da Vinci Payer Coverage Decision Exchange. Adler-Milstein argued that the first step toward developing policies that support scaling PCOR activities would be to identify which use cases are important to prioritize. In turn, this would facilitate adding patient-centered outcomes data that support those use cases into USCDI.

Deven McGraw, Ciitizen, provided an overview of the existing privacy laws that govern how data are accessed, used, and shared for research. The four laws that are most relevant are

1. The Health Insurance Portability and Accountability Act of 1996 (HIPAA).
2. “Part 2”, which relates to regulations on substance abuse data confidentiality.

3. The Family Educational Rights and Privacy Act (FERPA), which covers educational institutions.
4. The Privacy Act, which covers federal government data resources.

In the case of some entities, more than one of these laws might apply.

McGraw said that HIPAA has the most impact on PCOR data. She noted that HIPAA does not cover all health data, only the data of covered entities and their business associates. This coverage, however, is broad, and it includes most doctors and hospitals, health plans, and contractors. Exceptions might be some doctors practicing concierge medicine and some mental health professionals.

McGraw noted that HIPAA only governs identifiable data, which is known as protected health information (PHI). The disclosure of PHI is permitted for uses that fall in the category that combines treatment, payment, and operations. Public health disclosures are included among the permitted uses. McGraw said that in general, HIPAA was designed to enable data flows within a health care system, and data flows that are usual, expected, and customary, but it might be important to disclose only the minimum information necessary, particularly when the data are not used for treatment.

PHI can also be disclosed for research. This use of identifiable data was not considered to be standard and usual, but rather something that would require the consent of the person whose data would be used. However, regulators have recognized that requiring authorization for all research might not be feasible, so provisions exist that allow for a privacy board or an institutional review board to waive the consent requirement. Recent guidance from HHS enables an entity that is covered by HIPAA to obtain broad consent for research, instead of study-specific consent, for future uses of data. There are also provisions for the use of limited data sets, which involves removing some identifying information but allowing some identifying data elements to be left in.

McGraw said that the Common Rule is not included among the laws that govern data disclosure, because the Common Rule is a research ethics rule, not a privacy rule. However, there are many similarities between how HIPAA and the Common Rule govern research uses of data.

Data that are de-identified are not covered by HIPAA, and McGraw noted that it is also typical for privacy laws globally to only apply to identifiable data. One approach to de-identification is the Safe Harbor method, which involves the removal of identifiers that fall into 18 categories. It is also possible to rely on expert opinion (such as that of a statistician) to determine whether there is a risk of re-identification for a particular data set.

McGraw said that the use of data that are not identifiable falls outside of the realm of privacy regulations and that is why this type of data is

widely used in research, but it is also why there is a robust commercial trade of unidentified data. She noted that data that are not identifiable are not aggregated data, but individual-level data that have been stripped of identifiers or manipulated in some other way (e.g., by noise being introduced) to reduce the risk of re-identification. She emphasized that the standard is very low risk, rather than zero risk, and that there are no penalties to protect those data against re-identification.

Some state laws provide stronger data protections than HIPAA, and McGraw said that HIPAA does not preempt such laws. State laws typically govern access to data on minors, with minors in some states holding the right to consent to the disclosure of certain types of data for research, to third parties, or to their parents. Other states have robust consent laws that are not limited to minors, but as with HIPAA, they typically cover only identifiable data.

McGraw also noted that while HIPAA addresses permitted disclosures of data, it does not require the sharing of data for research purposes. However, the new information blocking rules¹ that went into effect in April 2021 create a presumption for sharing electronic health information for any lawful purpose, including research. These rules apply to health care providers, certified electronic health records vendors, and health information exchanges. Initially they will cover the USCDI data elements, but eventually they will cover all electronic health information. The penalties for “blocking” the sharing are up to $1 million per incident for electronic health records vendors and health information exchanges. The providers are referred to the Centers for Medicare & Medicaid Services (CMS) for “appropriate disincentives.” There are eight Safe Harbor provisions that allow an entity covered by these rules to decline a request for data sharing. These provisions are related to concerns such as privacy, security, harm, and infeasibility.

McGraw said that there has not yet been any enforcement since the rules went into effect (approximately 2 months prior to the workshop), and the rules around enforcement are still under consideration. McGraw noted that providers who want to decline a request for data have to attest to CMS that they are not information blocking, and if that claim does not hold up it could result in a False Claims Act penalty.

Another area where there are new developments concerns patients’ rights to their own data. Patients already have the right to their data under HIPAA, but this is now being more robustly enforced. Information blocking rules prioritize access by patients or apps acting on their behalf. There are also provisions to allow people to send data from their electronic health

___________________

¹ For more on information blocking, see https://www.healthit.gov/topic/information-blocking.

records directly to third parties, such as researchers, but these are still pending implementation.

Don Detmer, University of Virginia, shared his views on policy reforms necessary for robust data sharing for PCOR. He noted that HIPAA was passed into law in 1996, before use of the Internet became widespread. Today the use of data for health research is still defined by the rules that were developed based on “pre-Internet thinking,” with some minor regulatory tweaks taking place over the years. Detmer said that while there are some promising developments, it is time to ask whether this system is working.

The societal context today differs from what it was before the Internet. A large volume of new forms of data is available, and there is growing interest in goals such as equity, engaging patients and citizen-scientists, supporting precision medicine, and supporting precision health. Detmer said that continuing to do things within the current framework is going to be less than optimal. He argued that the basic structure for conducting research needs a reset to allow for informed public policy development that addresses new societal desires, with citizen-scientists, patients, and health providers as primary players in the data system, along with covered entities and business associates.

Detmer argued for reviving and enacting the HIPAA changes included in section 1124 in H.R.6, the initial 21st Century Cures Act. The revisions proposed at the time were to expand the access, use, and sharing of PHI from treatment, payment, and health care operations to also include “data research.” In 2015, H.R.6 passed the House by a vote of 344 to 77, but it did not pass the Senate.

Detmer described several current prevailing options for data sharing, clinical registries, and de-identified data sets, each of which he considers flawed. Registries with individuals donating their data are time-consuming to build and maintain and typically do not contain enough information. The aggregating of data sets that are limited to begin with has limited use when diverse data are sought. Using de-identified data poses additional challenges because authentication is difficult or impossible with diverse data sets.

Looking at potential solutions, Detmer wondered whether regulations could allow the use of text or e-mail for the sharing of PHI for research, and in particular the use of text or e-mail for obtaining approval from individuals. This would address the burden and challenges associated with obtaining written consent. He also wondered whether authorization could be created that would allow specified entities, such as the Patient-Centered Outcomes Research Institute, to securely access PHI in relevant databases, without individual consent. Detmer observed that while this is probably not realistic within the current system, it is allowed in some countries. He also suggested developing a system for unique patient identifiers, which becomes

especially important for the use of longitudinal data. Secure options for handling this type of information exist today.

Detmer argued for the need for a National Academies study that would develop a vision and a plan for a sound functional replacement for HIPAA. He mentioned a prior study that could serve as a model.² His desired goals and capabilities for the new framework would include robust system security for all data and “no-questions-asked opt-in” privacy for sharing personal data. He added that the data sharing would assure (1) system trust, (2) compassionate care, (3) scientific health care practice and evaluation for individuals and populations, (4) support for citizen-science and special populations, (5) secure unique personal identifiers, (6) pandemic data fitness and management, and (7) automation of all business operations and other administrative functions to reduce the time investment required.

DISCUSSION

One topic that was discussed by workshop participants in additional detail related to the role of the federal government in incentivizing the adoption of standards and assuring conformance. Participants cautioned about the burden associated with requirements of this type, and the potentially disproportionate burden on smaller health care providers.

A theme that emerged from the discussion was the need to better understand what type of information is truly important to people. Participants discussed projects such as Pastors 4 PCOR that involved community-based organizations to facilitate community engagement in research, identify specific disease priorities, and build trust.

Another topic that was revisited was the desire to link data from different sources and the consent and privacy challenges associated with this. In many cases complications related to consent prevent data sharing and linking, even when people are interested in making their data available for research. There was debate about the extent of public support for the concept of a unique identifier and whether the potential benefits are becoming more widely recognized. Technical solutions, such as tokenization, are creating new options that did not exist before, and this presents an opportunity to reassess these questions in new light.

Participants discussed the need to revisit HIPAA, which was passed in 1996, before the spread of social media, apps that require broad consent for data sharing, and expansive databases that are publicly available or can be purchased. HIPAA, in its current form, is not focused on privacy, and it

___________________

² Institute of Medicine. (1997). The Computer-Based Patient Record: An Essential Technology for Health Care (revised edition). Washington, DC: National Academy Press. https://doi.org/10.17226/5306.

only covers a small slice of health data. Many aspects of the regulation are outdated. Solutions could range from updating HIPAA “at the margins” to comprehensive privacy legislation. Participants commented that ASPE could have an enormously influential role in bringing stakeholders together on this issue.

CONCLUSIONS

This session echoed discussions from previous sessions about the importance of transparency in how the data will be used. Speakers also echoed the need to involve the people whose data are being used, as well as their communities, in decisions at each stage of the process, from data collection through research and dissemination. Building and maintaining trust with those whose data are being sought is essential to ensure that the data obtained are representative, complete, and reliable. This is especially important when the data could be perceived as sensitive, as is the case with some SDOH information.

CONCLUSION 4-1: Building and maintaining trust among the people and communities whose data are being sought for research is essential for high-quality data. Including representatives of consumers and patients in the research process to understand how to measure health impacts that matter to individuals is an important component in building trust.

The workshop made it clear that there are concerns about the laws and rules governing data access and data sharing. HIPAA, in particular, was developed several decades ago, and its approach to setting thresholds for data disclosures makes it outdated. There is a need for a new framework with guardrails that balance the risk of disclosure with the need for research that improves peoples’ health. This includes a need for a critical review of current privacy legislation, an understanding of public perspectives, and the development of recommendations for revisions or reform that would be applicable to the protection of health data in the post-Internet world, with a focus on preventing misuses of the data.

CONCLUSION 4-2: This is an opportune time to revisit and update the legislation and rules governing data privacy and the sharing of data for research.

Appendix A

Biographical Sketches of Committee Members

GEORGE ISHAM (NAM) (Chair) is a senior fellow at the HealthPartners Institute and a senior advisor for the Alliance of Community Health Plans. Previously, he served as a senior advisor to the board of directors and the senior management team of HealthPartners, and prior to that, he was HealthPartners’ medical director and chief health officer, responsible for quality of care and health and health care improvement. He has been active in health policy, serving as a member of the Centers for Disease Control and Prevention’s Task Force on Community Preventive Services, a member of the Agency for Healthcare Research and Quality’s United States Preventive Services Task Force, as a founding co-chair of the National Committee for Quality Assurance’s committee on performance measurement as well as founding co-chair of the National Quality Forum’s Measurement Application Partnership. He has an M.D. from the University of Illinois at Chicago and an M.S. in preventive medicine and administrative medicine from the University of Wisconsin–Madison.

JOHN F.P. BRIDGES is professor and vice chair of academic affairs in the Department of Biomedical Informatics at The Ohio State University (OSU) College of Medicine. He is also a professor in the Department of Surgery and an adjunct professor in both the Division of Epidemiology at the OSU College of Public Health and Department of Health Behavior and Society at the Johns Hopkins Bloomberg School of Public Health. Prior to joining OSU he was on the faculty of the Johns Hopkins Bloomberg School of Public Health, the Department of Tropical Hygiene and Public Health within University of Heidelberg School of Medicine, and the Department of

Epidemiology and Biostatistics within the Case Western Reserve University School of Medicine. He has previously held positions in the Department of Economics at the Weatherhead School of Management at Case Western Reserve University, the National Bureau of Economic Research, Center for Medicine in the Public Interest, and the Center for Health Economics, Research and Evaluation in Australia. He has a Ph.D. in economics from the City University of New York.

JULIE BYNUM is the Margaret Terpenning Professor of Medicine in the Division of Geriatric Medicine and vice chair for Faculty Affairs in the Department of Internal Medicine at the University of Michigan. She is also a research professor in the Institute of Gerontology, Geriatric Center Associate Director for Health Policy and Research, and a member of the Institute for Healthcare Policy and Innovation. She currently leads a portfolio of National Institutes of Health–funded research that examines the quality of care, diagnosis, and treatment of people with Alzheimer’s disease and related dementia in the community, nursing homes, and assisted living and is the director of the Center to Accelerate Population Research in Alzhiemer’s. She is currently a member of the National Academies of Sciences, Engineering, and Medicine’s Forum on Aging, Disability, and Independence and was a member of a National Academies committee that authored Vital Signs: Core Metrics for Health and Health Care Progress. She has an M.P.H. from the Johns Hopkins University School of Hygiene & Public Health and an M.D. from the Johns Hopkins University School of Medicine.

ANGELA DOBES is vice president of the Crohn’s & Colitis Foundation’s IBD Plexus Program, a research-information exchange platform designed to centralize data and biosamples from diverse research initiatives to advance science, accelerate precision medicine, and transform the care of Inflammatory Bowel Disease (IBD) patients. She has previously worked for clinical technology and pharmaceutical organizations, where she has led implementation of various technology solutions focused on business optimization and accelerating the delivery of new therapies to patients safely. She is currently serving as principal investigator on a study to enhance engagement, research participation, and collaboration through the IBD Partners Patient Powered Research Network. She has an M.A. in public health from the Icahn School of Medicine at Mount Sinai.

OLUWADAMILOLA FAYANJU is the Helen O. Dickens Presidential Associate Professor of Surgery at the Perelman School of Medicine at the University of Pennsylvania. She is also chief of breast surgery at Penn Medicine. Previously, she was associate professor of surgery and population health sciences in the Duke University School of Medicine and director of the

Durham VA Breast Clinic. She was also associate director for Disparities & Value in Healthcare with Duke Forge, Duke University’s center for actionable data science. In 2019, she was recognized by the National Academy of Medicine as an Emerging Leader in Health and Medicine Scholar. She received an M.A. in comparative literature from Harvard University and her M.D. and M.P.H.S. from the Washington University in St. Louis.

DEBORAH ESTRIN (NAE/NAM) is a professor of computer science at Cornell Tech where she holds the Robert V. Tishman founder’s chair, serves as the associate dean for impact, and is an affiliate faculty at Weill Cornell Medicine. Her research activities include technologies for caregiving, immersive health, small data, participatory sensing, and public interest technology. Estrin was an Amazon Scholar, and before joining Cornell University she was founding director of the National Science Foundation’s Center for Embedded Networked Sensing at the University of California, Los Angeles, pioneering the development of mobile and wireless systems to collect and analyze real-time data about the physical world. Estrin cofounded the nonprofit startup, Open mHealth, and has served on several scientific advisory boards for early-stage mobile health startups. She has a Ph.D. in electrical engineering and computer science from the Massachusetts Institute of Technology.

CONSTANTINE GATSONIS is the Henry Ledyard Goddard University Professor of Statistical Sciences, director of statistical sciences, and professor of biostatistics at Brown University. He was founding director of the Center for Statistical Sciences and founding chair of the Department of Biostatistics at Brown University. He is a leading authority on the evaluation of diagnostic and screening tests and has made major contributions to the development of methods for medical technology assessment and health services and outcomes research. He is a world leader in methods for applying and synthesizing evidence on diagnostic tests in medicine and is currently developing methods for comparative effectiveness research in diagnosis and prediction and radiomics. Since 2016 he has served as a statistical consultant for the New England Journal of Medicine and was the founding editor-in-chief of Health Services and Outcomes Research Methods. He has a Ph.D. in mathematical statistics from Cornell University.

ROBERT GOERGE is a senior research fellow at Chapin Hall at the University of Chicago. He is also a senior fellow and founder of the Master’s Degree in Computational Analysis in Public Policy at the University of Chicago Harris School of Public Policy. His research is focused on improving the available data and information on children and families, particularly those who require specialized services related to maltreatment, disability,

poverty, or violence. At Chapin Hall, he is principal investigator for the Family Self-Sufficiency Data Center, the Linking Federal Data to Local Data project, and the National Survey for Early Care and Education. He currently serves on the National Academies of Sciences, Engineering, and Medicine’s Committee on National Statistics. He has a Ph.D. in social policy from the University of Chicago.

GEORGE HRIPCSAK (NAM) is the Vivian Beaumont Allen Professor and chair of the Department of Biomedical Informatics at Columbia University. He is also the director of medical informatics services for New York Presbyterian Hospital. He is also a board-certified internist. He led the effort to create the Arden Syntax, a language for representing health knowledge that has become a national standard. As chair of the American Medical Informatics Association Standards Committee, he coordinated the medical informatics community response to the Department of Health and Human Services for the health informatics standards rules under the Health Insurance Portability and Accountability Act of 1996. His current research is on the clinical information stored in electronic health records. Using data mining techniques, he is developing the methods necessary to support clinical research and patient safety initiatives. He has an M.D. and an M.S. in biostatistics from Columbia University.

LISA IEZZONI (NAM) is professor of medicine at Harvard Medical School and the Health Policy Research Center at Massachusetts General Hospital, where she served as director in the past. She was previously co-director of research in the Division of General Medicine and Primary Care at Beth Israel Deaconess Medical Center in Boston. Her research focuses on risk adjustment methods for predicting cost and clinical outcomes of care, and on health care experiences and outcomes of persons with disabilities. She has served on the editorial boards of the Annals of Internal Medicine, the Journal of General Internal Medicine, Health Affairs, Medical Care, Health Services Research, and the Disability and Health Journal, among others. She has an M.D. from Harvard Medical School and an M.Sc. from the Harvard T.H. Chan School of Public Health.

S. CLAIBORNE JOHNSTON (NAM) is the inaugural dean of Dell Medical School, vice president for medical affairs, and the Frank and Charmaine Denius Distinguished Dean’s Chair in medical leadership at The University of Texas at Austin. Previously, Johnston was associate vice chancellor for research at the University of California, San Francisco (UCSF). He also directed the Clinical and Translational Science Institute and founded the UCSF Center for Healthcare Value. His research is focused on clinical trials and health services research in stroke. He is also an expert in medical

education, research administration, health care value, and population health. He has led several large-cohort studies of cerebrovascular disease and three international multicenter randomized trials. He has an M.D. from Harvard Medical School and a Ph.D. in epidemiology from the University of California, Berkeley.

MIGUEL MARINO is an associate professor with joint appointments in the School of Public Health Division of Biostatistics and the Department of Family Medicine at Oregon Health & Science University. His research focuses on the development and implementation of novel statistical methodology to address complexities associated with the use of electronic health records (EHRs) to study changes in policy, using EHRs to study health disparities, validation of EHRs as a reliable source for observational studies, pragmatic randomized trials, and preventive health maintenance. He was selected by the National Academy of Medicine as an Emerging Leader in Health and Medicine Scholar. He has a Ph.D. in biostatistics from Harvard University.

ELIZABETH McGLYNN (NAM) is vice president for Kaiser Permanente Research and executive director for the Center for Effectiveness & Safety Research at Kaiser Permanente. She is also interim senior associate dean for research and scholarships at the Kaiser Permanente Bernard J. Tyson School of Medicine. She is an internationally known expert on methods for evaluating the appropriateness and quality of health care delivery. She has led major initiatives to evaluate health reform options under consideration at the federal and state levels. She is the lead of Kaiser Permanente & Strategic Partners Patient Outcomes Research To Advance Learning (PORTAL) Network. She was a member of the Strategic Framework Board, which provided a blueprint for the National Quality Forum on the development of a national quality measurement and reporting system. She chaired the board of AcademyHealth, served on the board of the American Board of Internal Medicine Foundation, and served on the Board of Providence-Little Company of Mary Hospital Service Area in Southern California. She has a Ph.D. in public policy from RAND Graduate School.

DAVID MELTZER (NAM) is the Fanny L. Pritzker Professor in the Department of Medicine, chief of the section of Hospital Medicine and faculty in the Department of Economics and Harris School of Public Policy at the University of Chicago. He is also director of the Center for Health and the Social Sciences and of the Urban Health Lab at the University of Chicago. His research explores problems in health economics and public policy with a focus on the theoretical foundations of medical cost-effectiveness analysis and the cost and quality of hospital care. Since 1997, he has developed the

inpatient general medicine services at the University of Chicago as a Learning Health Care System to produce knowledge on how to improve the care of hospitalized patients, mobilizing the clinical care process to generate and learn from diverse data from electronic health records, claims data, patient interviews, and biospecimens on more than 100,000 patients. He is the lead of the University of Chicago network site as part of the Chicago Area Patient-Centered Outcomes Research Network. He has an M.D. and a Ph.D. in economics from the University of Chicago.

PAUL C. TANG (NAM) is an adjunct professor in the Clinical Excellence Research Center at Stanford University and an internist at the Palo Alto Medical Foundation. He was formerly chief innovation and technology officer at the Palo Alto Medical Foundation and vice president, chief health transformation officer at IBM Watson Health. He has more than 25 years of executive leadership experience in health information technology within medical groups, health systems, and corporate settings. He has directed innovation and technology teams in provider organizations, academic institutions, corporate research organizations, and product development organizations. Most recently, he led the creation, development, deployment, and evaluation of the application of artificial intelligence to physician point-of-care solutions integrated within an electronic health record system. He also led a corporate enterprise-wide design team. He has chaired numerous federal and private-sector advisory and professional association groups related to health information technology and policy. He received an M.S. in electrical engineering from Stanford University and his M.D. from the University of California, San Francisco.

Appendix B

Workshop Agenda

Building Data Capacity for Patient-Centered Outcomes Research:
An Agenda for 2021 to 2030

Virtual Workshop 2: Data Standards, Methods, and Policy

May 24, 2021, 11 am–5 pm EDT

OBJECTIVES FOR THE WORKSHOP

• Identify data standards and methods that can make the PCOR data infrastructure more useful for research and other data needs.
• Identify data policies that are needed to facilitate the continued development and operation of the PCOR data infrastructure.
• Discuss what HHS is best positioned to address and support, and how the agency could maximize resources available for the PCOR data infrastructure (representing 4% of the PCOR trust fund), in the context of the HHS public mission, authorities, programs, and data resources.

This page intentionally left blank.

Appendix C

Biographical Sketches of Workshop Speakers

JULIA ADLER-MILSTEIN (NAM) is a professor of medicine and director of the Center for Clinical Informatics and Improvement Research at the University of California, San Francisco (UCSF). She spent 6 years on the faculty at the University of Michigan prior to joining UCSF. She is a leading researcher in health information technology policy, with a specific focus on electronic health records (EHRs) and interoperability. She has examined policies and organizational strategies that enable effective use of EHRs and promote interoperability. She is also an expert in EHR audit log data and their application to studying clinician behavior. Her research—used by researchers, health systems, and policy makers—identifies obstacles to progress and ways to overcome them. She has served on an array of influential committees and boards, including the NHS National Advisory Group on Health Information Technology, the Health Care Advisory Board for Politico, and the Interoperability Committee of the National Quality Forum. Adler-Milstein holds a Ph.D. in health policy from Harvard University.

DON EUGENE DETMER (NAM) is professor of medical education at the University of Virginia. He has served as vice president for health sciences at the University of Virginia and the University of Utah, as the Dennis Gillings professor for health management at Cambridge University, as president/chief executive officer of the American Medical Informatics Association, and as medical director of policy for the American College of Surgeons. Professorial appointments have included university professor of health policy, professor of surgery, business administration, public health sciences,

preventive medicine, as well as visiting professor at University College London. He is past chair of the Board of Regents of the National Library of Medicine; the National Committee on Vital and Health Statistics; the National Academies of Sciences, Engineering, and Medicine’s Board of Health Care Services; and Blue Ridge Academic Health Group, which he founded. Current boards include the Corporation for National Research Initiatives, the American College of Medical Informatics, and the International Academy of Health Sciences Informatics. He helped envision the national health information infrastructures of the United States and Hong Kong, as well as shaped policy for direct electronic communications of health records with patients in the United States and Europe. He earned an M.A. from Cambridge University and an M.D. from the University of Kansas. He completed postgraduate training at Johns Hopkins University, the National Institutes of Health, Duke Medical Center, and Harvard Business School.

EVELYN GALLEGO is the chief executive officer and founder of EMI Advisors LLC, an 8(a) certified Small Minority-Owned Business, founded to deliver value-driven health data management advisory services to government and commercial clients. She helps clients to bridge the gap between health information technology policy and standards and business requirements. She has a strong ability to work across and build consensus with diverse stakeholder groups to include multidisciplinary providers, policy makers, health care payers, researchers, system vendors and implementers, and standard development organizations. Gallego provides specialized expertise in digital health interoperability and health policy with a focus on alignment of regulatory, technical, and process improvement requirements to enable the effective adoption and use of technology. She is a thought leader in the areas of care coordination, social determinants of health, health information technology (IT) policy analysis and development, health information exchange and interoperability, and health IT standards development. She currently serves as the program manager and subject matter expert for three leading interoperability projects including the HL7 Gravity Project, the ONC STARS HIE Technical Assistance Program, and the NIH/AHRQ Multiple Chronic Care Electronic Care Plan Project. Gallego earned her international M.B.A. from the Schulich School of Business in Toronto, Canada, and her M.P.H. in health policy from George Washington University.

SHAUN GRANNIS is the vice president of data analytics and a medical informatics research scientist at the Regenstrief Institute. He is also the Sam Regenstrief professor of medical informatics and professor of family medicine at the Indiana University School of Medicine. In these roles, he

collaborates with national and international health stakeholders seeking to advance health data technical infrastructure and data-sharing capabilities. He has provided identity management consultancy to organizations, including the World Health Organization and the Office of the National Coordinator for Health Information Technology. Grannis also supports health information exchange (HIE) activity among more than 120 hospitals in Indiana for use in clinical research and disease surveillance. His recent research focuses on developing and testing large-scale HIE-based solutions in support of population health and public health informatics; integrating clinical and social determinants of health (SDH) to identify at-risk patients in need of SDH services, which include nutrition counseling, financial planning, and medical-legal partnership assistance; developing and testing novel patient matching methods; and leveraging machine learning–based models to improve discovery and decision support in a variety of contexts. Grannis holds an M.D. from Michigan State University and bachelor’s degree in aerospace engineering from the Massachusetts Institute of Technology.

JOHN HALAMKA (NAM) is the president of Mayo Clinic Platform. Prior to the Mayo Clinic, he served as the executive director of the Health Technology Exploration Center for Beth Israel Lahey Health in Massachusetts. During his tenure at Beth Israel Lahey Health, he oversaw digital health relationships with industry, academia, and government worldwide. Previously, he was chief information officer at Beth Israel Deaconess Medical Center for more than 20 years. In his role at Beth Israel Deaconess Medical Center, Halamka was responsible for all clinical, financial, administrative and academic information technology (IT). As a Harvard Medical School professor, he served the George W. Bush administration, the Obama administration, and governments around the world planning their health care IT strategies. In addition, he was the international healthcare innovation professor at Harvard Medical School. He remains chairman of New England Healthcare Exchange Network Inc. and is a practicing emergency medicine physician. Halamka received his B.S. in medical microbiology and his B.A. in public policy from Stanford University, his M.D. from the University of California, San Francisco, and his M.S. from Harvard University.

ABEL KHO is professor of medicine and preventive medicine in the Feinberg School of Medicine at Northwestern University and founding director of the Center for Health Information Partnerships and the Institute for Augmented Intelligence in Medicine. He has served as principal investigator for several regional or national projects including the Office of the National Coordinator for Health Information Technology–funded Chicago Health IT Regional Extension Center, the Patient-Centered Outcomes Research Institute–funded Chicago Area Patient Centered Outcomes Research Network, and the Agency

for Healthcare Research and Quality–funded Health Hearts in the Heartland consortium within the EvidenceNOW initiative. His research focuses on developing regional electronic health record–enabled data sharing platforms for a range of health applications including high throughput phenotyping, cohort discovery, estimation of population level disease burden, and quality improvement. Kho received his M.D. from the Medical College of Wisconsin and completed a residency and chief residency in internal medicine at the University of Wisconsin–Madison.

NIROSHA MAHENDRARATNAM LEDERER is director of real-world evidence strategy at Aetion, where she leads the engagement of federal accounts and advises clients on generating decision-grade evidence. Previously, she was a managing associate at the Duke-Margolis Center for Health Policy, where she led the Center’s real-world evidence portfolio. Prior to this position, she was a subject matter expert in the Oncology Center of Excellence at the U.S. Food and Drug Administration. While there, Lederer helped to implement patient-focused drug development in cancer products including clinical trial study design and product review, as well as foster consensus across U.S. and ex-U.S. health care stakeholders on best practices for patient-reported outcome capture, analysis, and communication. She has more than 15 years of pharmaceutical policy and health economics and outcomes research experiences, including providing evidence-generation advisory services at Avalere Health, working in commercial and medical roles at Genentech and Bristol-Myers Squibb, respectively, and serving on Capitol Hill during the passage of the Affordable Care Act. Lederer received her Ph.D. in health outcomes and policy from the University of North Carolina at Chapel Hill with a focus on large database analyses and decision-sciences. She received an M.S.P.H. in health policy and management from the Johns Hopkins Bloomberg School of Public Health and a B.A. in public health from the Johns Hopkins University.

LARA MANGRAVITE is president of Sage Bionetworks, an organization focused on the development and implementation of practices for large-scale collaborative biomedical research. Sage Bionetworks’ work is centered on new approaches to scientific process that use open systems to enable community-based research regarding complex biomedical problems. Previously, she served as director of the systems biology research group at Sage Bionetworks where she focused on the application of collaborative approaches to advance understanding of disease biology and treatment outcomes at a systems level with the overriding goal of improving clinical care. Mangravite has a B.S. in physics from the Pennsylvania State University and a Ph.D. in pharmaceutical chemistry from the University of California, San Francisco. She completed a postdoctoral fellowship

in cardiovascular pharmacogenomics at the Children’s Hospital Oakland Research Institute.

DEVEN McGRAW is general counsel and chief regulatory officer for Ciitizen, a consumer health technology start-up. Previously, she directed U.S. health privacy and security as deputy director, Health Information Privacy at the U.S. Department of Health and Human Services’ Office for Civil Rights and chief privacy officer (acting) of the Office of the National Coordinator for Health Information Technology. Widely recognized for her expertise in health privacy, she directed the Health Privacy Project at the Center for Democracy & Technology for 6 years and led the privacy and security policy work for the HITECH Health IT Policy Committee. She also served as the chief operating officer of the National Partnership for Women and Families. She advised health industry clients on Health Insurance Portability and Accountability Act of 1996 (HIPAA) compliance and data governance while a partner at Manatt, Phelps & Phillips, LLP. McGraw graduated magna cum laude from Georgetown University Law Center and has an M.P.H. from Johns Hopkins University.

SHARON-LISE NORMAND is S. James Adelstein professor of health care policy (biostatistics) in the Department of Health Care Policy at Harvard Medical School and professor in the Department of Biostatistics at Harvard School of Public Health. Her research focuses on the development of statistical methods for health services and outcomes research, including the evaluation of medical devices, causal inference, provider profiling, evidence synthesis, item response theory, and latent variables analyses. Her application areas include cardiovascular disease, severe mental illness, medical device safety and effectiveness, and medical technology diffusion. Normand was the 2010 president of the Eastern North American Region of the International Biometrics Society and inaugural vice chair of the Patient-Centered Outcomes Research Institute’s Methodology Committee (2010–2012). She was awarded the ASA 2011 Health Policy Statistics Section’s Long Term Excellence Award, the 2012 American Heart Association’s Distinguished Scientist Award, the 2017 American Heart Association Council on Quality of Care and Outcomes Research Outstanding Lifetime Achievement Award, and the 2018 Mosteller Statistician of the Year. She is a fellow of the American Statistical Association, the American Heart Association, the American College of Cardiology, and the American Association for the Advancement of Science. Normand earned her Ph.D. in biostatistics, and M.Sc. and B.Sc. in statistics, and completed a postdoctoral fellowship in health care policy.

RAcHEL RIcHESSON is a professor in the Department of Learning Health Sciences, School of Medicine at the University of Michigan. She

conducts original research on the quality and usability of data from electronic health records (EHRs) for research and has fostered numerous interdisciplinary research collaborations. She has directed implementation of data standards for a number of multinational multisite clinical research and epidemiological studies, including the National Institutes of Health (NIH) Rare Diseases Clinical Research Network, Type 1 Diabetes TrialNet, and The Environmental Determinants of Diabetes in the Young study, and the national distributed Patient-Centered Outcomes Research Network. Richesson currently leads the EHR Core for the NIH Health Systems Research Collaboratory, which is developing standards and quality metrics for clinical phenotyping using EHR data in pragmatic clinical trials. In addition, she and Department of Learning Health Science chair Charles Friedman colead the multi-stakeholder “Mobilizing Computable Biomedical Knowledge” community charged with establishing the standards, policies, and governance needed for biomedical knowledge to be widely disseminated and applied. Richesson holds a Ph.D. and an M.S. in health informatics and an M.P.H. from the University of Texas.

PAMELA RILEY is medical director of the District of Columbia Department of Health Care Finance, overseeing medical administration and quality of care in the District of Columbia’s Title XIX (Medicaid), CHIP, and Alliance Programs. She previously served as vice president for delivery system reform at The Commonwealth Fund, developing and managing grants focused on transforming health care delivery systems for vulnerable populations, including low-income groups, racial/ethnic minorities, and uninsured populations. She also served as program officer at the New York State Health Foundation, where she developed and managed grant-making programs in the areas of integrating mental health and substance use services, addressing the needs of returning veterans and their families, and diabetes prevention and management. Earlier in her career, Riley served as clinical instructor in the Division of General Pediatrics at the Stanford University School of Medicine. She served as a Duke University Sanford School of Public Policy Global Health Policy fellow at the World Health Organization in Geneva, Switzerland, and has served as a volunteer physician in Peru and Guatemala. Riley has an M.D. from the University of California, Los Angeles (UCLA) David Geffen School of Medicine, and completed her internship and residency in pediatrics at Harbor-UCLA Medical Center in Torrance, California. She received an M.P.H. from the Harvard School of Public Health as a Commonwealth Fund fellow in minority health policy.

SHERRI ROSE is an associate professor at Stanford University in the Center for Health Policy and Center for Primary Care and Outcomes Research. She is also codirector of the Health Policy Data Science Lab.

Her research is centered on developing and integrating innovative statistical machine-learning approaches to improve human health. Within health policy, she works on risk adjustment, comparative effectiveness research, and health program evaluation. She has published interdisciplinary projects across varied outlets, including Biometrics, the Journal of the American Statistical Association, the Journal of Health Economics, Health Affairs, and the New England Journal of Medicine. Rose is the coeditor of Biostatistics and chair of the American Statistical Association’s Biometrics Section. Her honors include a National Institutes of Health Director’s New Innovator Award, the ISPOR Bernie J. O’Brien New Investigator Award, and Mid-Career Awards from the American Statistical Association’s Health Policy Statistics Section and Penn-Rutgers Center for Causal Inference. She was also named a fellow of the American Statistical Association in 2020. In 2011, she coauthored the first book on machine learning for causal inference, with a sequel text released in 2018. Rose has a B.S. in statistics from George Washington University, and a Ph.D. in biostatistics from the University of California, Berkeley.

PATRIcK RYAN is vice president of observational health data analytics at Janssen Research and Development, where he is leading efforts to develop and apply analysis methods to better understand the real-world effects of medical products. He is an original collaborator in Observational Health Data Sciences and Informatics, a multistakeholder, interdisciplinary collaborative to create open-source solutions that bring out the value of observational health data through large-scale analytics. Ryan served as a principal investigator of the Observational Medical Outcomes Partnership, a public-private partnership chaired by the U.S. Food and Drug Administration, where he led methodological research to assess the appropriate use of observational health care data to identify and evaluate drug safety issues. He has worked in various positions within the pharmaceutical industry at Pfizer and GlaxoSmithKline and also in academia at the University of Arizona Arthritis Center. Ryan received his undergraduate degrees in computer science and operations research at Cornell University, his M.Eng. in operations research and industrial engineering at Cornell, and his Ph.D. in pharmaceutical outcomes and policy from University of North Carolina at Chapel Hill.

NIGAM H. SHAH is a professor of medicine (biomedical informatics) at Stanford University, associate chief information officer for data science at Stanford Healthcare, and a member of the Biomedical Informatics Graduate Program as well as the Clinical Informatics Fellowship. His research focuses on combining machine learning and prior knowledge in medical ontologies to enable use cases of the learning health system. He received

the AMIA New Investigator Award for 2013 and the Stanford Biosciences Faculty Teaching Award for outstanding teaching in his graduate class on data-driven medicine. He was elected into the American College of Medical Informatics in 2015 and was inducted into the American Society for Clinical Investigation in 2016. Shaw holds an M.B.B.S. from Baroda Medical College, India, and a Ph.D. from Penn State University, and completed his postdoctoral training at Stanford University.

VG VINOD VYDISWARAN is an assistant professor in the Department of Learning Health Sciences with a secondary appointment in the School of Information at the University of Michigan, Ann Arbor. His research focuses on developing and applying text mining, natural language processing, and machine-learning methodologies for extracting relevant information from health-related text corpora. This includes medically relevant information from clinical notes and biomedical literature, and studying the information quality and credibility of online health communication (via health forums and tweets). His previous work includes developing novel information retrieval models to assist clinical decision making, modeling information trustworthiness, and addressing the vocabulary gap between health professionals and laypersons. Vydiswaran received his Ph.D. from the University of Illinois at Urbana-Champaign and his M.Tech from the Indian Institute of Technology Bombay.