THE LEARNING HEALTH SYSTEM SERIES
CARING FOR THE INDIVIDUAL PATIENT
Understanding Heterogeneous
Treatment Effects
David Kent, Jessica Paulus, Mahnoor Ahmed,
and Danielle Whicher, Editors
WASHINGTON, DC
NAM.EDU
NATIONAL ACADEMY OF MEDICINE 500 Fifth Street, NW Washington, DC 20001
This publication has undergone peer review according to procedures established by the National Academy of Medicine (NAM). Publication by the NAM signifies that it is the product of a carefully considered process and is a contribution worthy of public attention, but does not constitute endorsement of conclusions and recommendations by the NAM. The views presented in this publication are those of individual contributors and do not represent formal consensus positions of the authors’ organizations; the NAM; or the National Academies of Sciences, Engineering, and Medicine.
Support for this publication was provided by the Patient-Centered Outcomes Research Institute® (PCORI®), through two awards: a Patient-Centered Outcomes Research Institute (PCORI) Eugene Washington PCORI Engagement Award (1900-TMC) and the Predictive Analytics Resource Center (SA.Tufts.PARC.OCSCO.2018.01.25); and the Predictive Analytics and Comparative Effectiveness (PACE) Center at the Tufts Medical Center. The views presented in this publication are solely the responsibility of the editors and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute® (PCORI®), its Board of Governors, or its Methodology Committee; or of the PACE Center and/or the Tufts Medical Center.
International Standard Book Number-13: 978-1-947103-16-0
Library of Congress Control Number: 2019948398
Digital Object Identifier: https://doi.org/10.17226/27112
Copyright 2019 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
Suggested citation: Kent, D., J. Paulus, M. Ahmed, and D. Whicher, Editors. 2019. Caring for the Individual Patient: Understanding Heterogeneous Treatment Effects. Washington, DC: National Academy of Medicine.
ABOUT THE NATIONAL ACADEMY OF MEDICINE
The National Academy of Medicine is one of three Academies constituting the National Academies of Sciences, Engineering, and Medicine (the National Academies). The National Academies provide independent, objective analysis and advice to the nation and conduct other activities to solve complex problems and inform public policy decisions. The National Academies also encourage education and research, recognize outstanding contributions to knowledge, and increase public understanding in matters of science, engineering, and medicine.
The National Academy of Sciences was established in 1863 by an Act of Congress, signed by President Lincoln, as a private, nongovernmental institution to advise the nation on issues related to science and technology. Members are elected by their peers for outstanding contributions to research. Dr. Marcia McNutt is president.
The National Academy of Engineering was established in 1964 under the charter of the National Academy of Sciences to bring the practices of engineering to advising the nation. Members are elected by their peers for extraordinary contributions to engineering. Dr. John L. Anderson is president.
The National Academy of Medicine (formerly the Institute of Medicine) was established in 1970 under the charter of the National Academy of Sciences to advise the nation on issues of health, health care, and biomedical science and technology. Members are elected by their peers for distinguished contributions to medicine and health. Dr. Victor J. Dzau is president.
Learn more about the National Academy of Medicine at NAM.edu.
This page intentionally left blank.
WORKSHOP PLANNING COMMITTEE
DAVID M. KENT (Chair), Tufts Medical Center
THOMAS CONCANNON, RAND Corporation
ROBERT GOLUB, Journal of the American Medical Association
SHELDON GREENFIELD, University of California, Irvine
RODNEY HAYWARD, University of Michigan
A. CECILE J. W. JANSSENS, Emory University Rollins School of Public Health
MUIN J. KHOURY, Centers for Disease Control and Prevention
PETER ROTHWELL, University of Oxford
EWOUT STEYERBERG, Leiden University Medical Center
ANDREW J. VICKERS, Memorial Sloan Kettering Cancer Center
NAM Staff
Development of this publication was facilitated by contributions of the following NAM staff, under the guidance of J. Michael McGinnis, Executive Officer and Executive Director of the Leadership Consortium for a Value & Science-Driven Health System:
DANIELLE WHICHER, Senior Program Officer
MAHNOOR AHMED, Research Associate
JESSICA BROWN, Executive Assistant to the Executive Officer
JENNA OGILVIE, Communications Officer
Tufts University Staff
DAVID KENT, Tufts Medical Center
JESSICA PAULUS, Tufts Medical Center
Consultant
ROBERT POOL, Hired Pens LLC
This page intentionally left blank.
REVIEWERS
This Special Publication was reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with review procedures established by the National Academy of Medicine (NAM). These reviewers were asked to consider the accuracy of the content within this Special Publication, the accuracy with which conversations at the workshop on which this Special Publication was based were conveyed, and the strength and balance of this Special Publication’s arguments.
We wish to thank the following individuals for their contributions:
FRANK DAVIDOFF, Annals of Internal Medicine (Emeritus)
SETH MORGAN, National Multiple Sclerosis Society
JODI SEGAL, Johns Hopkins University
CHRISTINE STAKE, Ann & Robert H. Lurie Children’s Hospital of Chicago
The reviewer composition includes individuals with subject-matter expertise, attendees at the workshop, and those who did not attend the workshop. The reviewers listed above provided many constructive comments and suggestions, but they were not asked to endorse the content of the publication, and did not see the final draft before it was published. Review of this publication was overseen by DANIELLE WHICHER, Senior Program Officer, NAM, and J. MICHAEL McGINNIS, Executive Officer, NAM. Responsibility for the final content of this publication rests entirely with the editors and the NAM.
This page intentionally left blank.
PREFACE
“The premise of traditional research is to put a treatment at the center of consideration and decide, Is this treatment helpful for an average patient? Trouble is, there aren’t very many average patients out there, and I, like most people, am not an average patient.”
—Seth Morgan, neurologist, multiple sclerosis patient, and patient advocate
Evidence-based medicine (EBM) arose from a clear need and represents a major advance in the science of clinical decision making. Traditional approaches to decision making based on expert opinion, extrapolations of pathophysiologic reasoning, or personal experience led to extreme variations in practice patterns, which have been well documented, starting in the 1970s (Wennberg and Gittelsohn, 1973). Many routinely accepted clinical practices have been found to be ineffective (or harmful) when subjected to evaluation by randomized trial designs, and large proportions of “effective” procedures were found to be inappropriate when scrutinized by expert review (Chassin et al., 1987). More broadly, it is well understood—not only in medicine, but in many fields—that human decision making is plagued by fundamental cognitive biases, and that statistically driven decision making has general advantages compared with human “expert” judgment (Kahneman et al., 1982; Meehl, 2013).
Despite broad acceptance of EBM, however, a fundamental incongruity remains unresolved: Evidence is derived from groups of people, yet medical decisions are made by and for individuals. Randomization—introduced by R. A. Fischer in the field of agriculture and ported into clinical research by Austin Bradford Hill—ensures the comparability of treatment groups within a clinical trial, which allows for unbiased estimation of average treatment effects. If, like farmers growing crops, we treated groups of patients instead of individuals, or if patients with the same disease were identical to one another in all factors that determined the harms and the benefits of therapy, then these group-level averages would make a perfectly sound foundation for medical decision making. However,
patients differ from one another in many ways that determine the likelihood of an outcome, both with and without a treatment. Nevertheless, despite persistent assertions by clinicians that determining the best therapy for each patient is a more complicated endeavor than simply picking the best treatment on average, popular approaches to EBM have encouraged an over-reliance on the average effects estimated from clinical trials as guides to decision making for individuals.
Shortly after the turn of the 20th century, the decoding of the human genome promised to deliver us from one-size-fits-all medicine. But a decade and a half later, it appears unlikely that genetic information will be leveraged broadly or deeply into clinical decision making. The effects of individual single nucleotide polymorphisms (SNPs) tend to be small (Goldstein, 2009), they typically add little information to easily obtainable clinical or phenotypic information (Ioannidis, 2009), and even in combination they account for only a small proportion of heritability (Manolio et al., 2009). (The limitations of polygenic scores are well reviewed in A. Cecile J.W. Janssens’s presentation; see Chapter 4.) While more than 350 different pharmacogenomic associations are included in pharmaceutical labels, the clinical utility of these tests is generally not established; and despite important efforts (e.g., those described by Josh Peterson; see Chapter 5), pharmacogenomics has not brought us substantially closer to understanding individualized benefit–harm trade-offs for most interventions.
Notwithstanding the challenges of unraveling the genetics of disease states and the disappointments, to date, of gene-based approaches to diagnosis, prognosis, and treatment, the goals of personalized medicine remain deeply compelling. Better population-based outcomes will only be realized when we understand more completely how to treat patients as the unique individuals they are. Our patients surely expect nothing less. The reality of effect modification (i.e., that the same treatment in different patients may have different consequences) is undeniable to any physician. For example, angiotensin inhibitors can both cause and prevent kidney dysfunction, anticoagulation treatments can both cause and prevent strokes (hemorrhagic and embolic, respectively), and antihypertensive medications can both cause and prevent cardiac events. But these patient-level variations are not completely unpredictable. A simple medical history and physical examination can provide abundant information about how patients with the same disease (or those included in the same trial) can differ from one another in many important ways that influence benefit–harm trade-offs.
In May 2018, under the auspices of the National Academy of Medicine (NAM), we gathered a group of experts and stakeholders—physicians, methodologists, patients, payers, and regulators, among others—to discuss the tension between group evidence and decision making for individuals. The group focused on
“predictive” approaches to heterogeneous treatment effects (HTE). That is, for evidence to be more applicable at the individual patient level, we need to combine methods for strong causal inference (e.g., randomization) with methods for prediction that permit inferences about which particular patients are likely to benefit and which are not.
One point of agreement for better patient-centered evidence was that rather than serially examining subgroups defined “one variable at a time” for statistically significant interaction effects, a more relevant approach is to disaggregate patients by fundamental dimensions of risk using models that incorporate the effects of multiple prognostically important clinical variables simultaneously to yield “personalized” estimates of benefit–harm trade-offs. Risk dimensions that are important for decision making include the risk of the primary outcome of interest (as patients at higher risk often have greater potential for benefit) and the risk of treatment-related harm. Disaggregating patients into strata defined by these risks can yield information about effects that may be obscured in the overall average and in conventional subgroup analysis. Another important point of agreement was that information on both harms and benefits of treatment across these different risk strata should be presented on an absolute scale—rather than a relative risk scale—to support clinical decision making.
While the principles for these “predictive” HTE analyses of randomized controlled trials were introduced more than a decade ago (Kent et al., 2010; Rothwell et al., 2005), speakers at the conference noted that recent developments and refinements in such analyses provide reasons for optimism, including the investment of more resources in patient-centered outcomes research (particularly through the Patient-Centered Outcomes Research Institute [PCORI]); the priority PCORI has given to research accounting for HTE; advances in “big data” in medicine (and in the broader culture) that facilitate development, validation, and continual updating of prediction models; new methods for prediction using machine learning (discussed by Fan Li; see Chapter 4); new adaptive research designs developed to cope with and leverage patient heterogeneity (discussed by Derek Angus; see Chapter 2); the broad dissemination of electronic health records (EHRs) and incentives for their “meaningful use”; specific support in the Patient Protection and Affordable Care Act for shared decision making; and the “open data” movement encouraging new models for clinical trial data sharing, enabling individual patient meta-analysis capable of supporting well-powered predictive HTE analysis.
Additional dimensions of evidence individualization discussed herein include the need for effective implementation strategies for the use of prediction models that promote physician and patient acceptance (discussed by John Spertus; see
Chapter 5); developing new quality measures to incentivize personalized care that transcends binary all-or-none rules, which tend to promote low-value care (discussed by Rod Hayward; see Chapter 5);enhancement of restrictive formularies to permit doctors and patients the latitude to select pharmaceuticals that work best at the individual level; and new value frameworks for pharmaceutical pricing that take this heterogeneity into account (discussed by Robert Dubois; see Chapter 3).
Despite substantial progress and many points of agreement, the workshop also highlighted numerous controversies, challenges, and research gaps. These included determining the appropriate role for observational data, understanding the comparative performance of machine learning methods compared with traditional statistical approaches for predicting HTE, and developing guidance on methods for assessing the effectiveness or validity of models that predict benefit (i.e., the difference among potential outcomes with alternative treatments, rather than just predicting outcome and prognosis).
In summary, there was broad agreement that while the challenges remain formidable, a better understanding of the heterogeneity in treatment effects has the potential to truly transform medical care, improve health outcomes, and reduce unnecessary or ineffective therapies by targeting treatments to those most likely to benefit. The discussions captured in this volume are critically important for moving this conversation—and medicine in general—forward in the decades to come.
We would like to thank all of the attendees at the workshop on which this Special Publication is based for their generous and robust conversations. We would also like to thank Mahnoor Ahmed and Danielle Whicher of the NAM, Jessica Paulus of Tufts University, and Robert Pool of Hired Pens LLC, all of whom, along with David Kent, contributed significantly to the drafting and editing of this Special Publication.
David Kent, M.D., M.S.
Director
Predictive Analytics and Comparative Effectiveness (PACE) Center
Tufts Medical Center
Joseph Selby, M.D., M.P.H.
Executive Director
Patient-Centered Outcomes Research Institute (PCORI)
J. Michael McGinnis, M.D., M.P.P.
Executive Officer
National Academy of Medicine
CONTENTS
2 The Promise of Personalized Evidence-Based Medicine
Using Risk-Based Forecasting to Personalize Medicine
Development of a Decision Score to Optimize Treatment Decisions
Designing Randomized Controlled Trials with Heterogeneous Treatment Effects in Mind
Regulatory Utility of Understanding Heterogeneous Treatment Effects
3 Patient Perspectives of the Significance of Understanding Heterogeneous Treatment Effects
Engaging Patients in Discussions About Heterogeneous Treatment Effects
The Problem with Treatments Aimed at the “Average Patient”
Taking Patient Preferences into Account
Providing Patients with Decision-Making Tools
4 New Methods for the Prediction of Treatment Benefit and Model Evaluation
Methodological Issues Related to Predictive Scores
Absolute Risk Versus Relative Risk
5 Next Steps for Implementation
Using Heterogeneous Treatment Effects in Routine Clinical Care
Applying Pharmacogenomics in Clinical Care
Improving Performance Measures
Identifying Clinically Meaningful Heterogeneous Treatment Effects
6 A Research Agenda for Personalizing Care and Improving Treatment Outcomes
Designing Research to Meet the Needs of End-Users
A Research Agenda for Understanding and Leveraging Treatment Heterogeneity to Improve Patient Care
BOX AND FIGURES
BOX
FIGURES
2-1 Distribution of mortality risk in medically treated patients with acute myocardial infarction
2-4 Typical risk distributions in clinical trials are left-shifted
2-5 In adaptive platform trials, a promising treatment can be more quickly validated
4-1 Potential outcomes for a patient undergoing a medical treatment
5-1 Reduction in bleeding after introduction of the ePRISM system
5-3 Platelet aggregation response to clopidogrel varies by CYP2C19 variants
5-4 Relationship between A1c and lifetime risk of blindness
