Page 191 Cite

Suggested Citation:"Appendix C: Workshop Two Agenda." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.

×

Appendix C

Workshop Two Agenda

Workshop Two: Practical Approaches

March 6–7, 2018

National Academy of Sciences Building, Room 120
2101 Constitution Avenue, NW, Washington, DC 20418

The National Academies of Sciences, Engineering, and Medicine is convening a three-part workshop series examining how real-world evidence development and uptake can enhance medical product development and evaluation. The workshops will advance discussions and common knowledge about complex issues relating to the generation and usage of real-world evidence, including fostering development and implementation of the science and technology of real-world evidence generation and usage.

Workshop One (September 19–20, 2017) focused on how to align incentives to support collection and use of real-world evidence in health product review, payment, and delivery. Incentives need to address barriers impeding the uptake of real-world evidence, including barriers to transparency.
Workshop Two (March 6–7, 2018) will illuminate what types of data are appropriate for what specific purposes and suggest practical approaches for data collection and evidence use by developing and working through example use cases.

Page 192 Cite

Suggested Citation:"Appendix C: Workshop Two Agenda." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.

×

Workshop Three (July 17–18, 2018) will examine and suggest approaches for operationalizing the collection and use of real-world evidence.

DAY 1: MARCH 6, 2018

8:30 a.m.

Breakfast Available Outside Room 120

8:40 a.m.

Welcome and Opening Remarks

GREGORY SIMON, Workshop Series Co-Chair

Investigator

Kaiser Permanente Washington Health Research Institute

Session I: When Can We Rely on Real-World Data?

Session Discussion Questions:

When can we have confidence in EHR data from real-world practice to accurately assess study eligibility, key prognostic factors, and study outcomes?
When can we have confidence in data generated outside of clinical settings (e.g., mobile phones, connected glucometers, connected blood pressure monitors)?
When does adjudication or other postprocessing of real-world data add value?

Moderator:	Gregory Daniel, Duke-Margolis Center for Health Policy
	Session Discussants JESSE BERLIN Vice President and Global Head, Epidemiology Johnson & Johnson
	ANDY BINDMAN Professor of Medicine University of California, San Francisco
9:00 a.m.	Introduction and Background to Inform the Discussion: Novel Oral Anticoagulants in Comparison with Warfarin
9:00 a.m.	ADRIAN HERNANDEZ Vice Dean for Clinical Research Duke University School of Medicine

Page 193 Cite

Suggested Citation:"Appendix C: Workshop Two Agenda." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.

×

9:20 a.m.	Open Discussion with Audience What questions can characterize the utility of any real-world data source and signal reliability before a study is performed (examples below)? When is accuracy good enough to reasonably and consistently identify the right population? When is accuracy good enough to reasonably and consistently assess the exposure or intervention? When is accuracy good enough to reasonably and consistently assess the right outcome? Are there any big safety issues that would be missed? Are there concerns about data collection or entry, particularly in relation to creating systemic bias? When is expert adjudication necessary to confirm that the recorded data are reliable and/or reasonably complete? What information is needed to answer such questions?
10:40 a.m.	BREAK (Workgroup Participants Gather to Synthesize Audience Feedback)
11:00 a.m.	Workgroup Presents Synthesis of Audience Feedback

Session II: When Can We Rely on Real-World Treatment?

Session Discussion Questions:

When conducting research in a real-world setting, are there situations that would require special guidance, knowledge, or experience in order for clinicians to adequately monitor participant safety and respond appropriately to adverse events?
When does variation between comparison groups (socioeconomic, demographic, etc.); in treatment fidelity; in provider behavior and preferences; or in adherence yield a valid signal about real-world effectiveness, and when is it just noise?

Page 194 Cite

Suggested Citation:"Appendix C: Workshop Two Agenda." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.

×

Moderator:	Khaled Sarsour, Genentech
	Session Discussants MICHAEL HORBERG Executive Director, Research, Community Benefit, and Medicaid Strategy Executive Director, Mid-Atlantic Permanente Research Institute Kaiser Permanente Mid-Atlantic Permanente Medical Group
	GREGORY SIMON Investigator Kaiser Permanente Washington Health Research Institute
	ROBERT TEMPLE Deputy Director for Clinical Science Center for Drug Evaluation and Research U.S. Food and Drug Administration
11:15 a.m.	Introduction and Presentation to Inform Discussion on Participant Monitoring: Study on Lithium for Suicidal Behavior in Mood Disorders
11:15 a.m.	IRA KATZ Senior Consultant for Program Evaluation U.S. Department of Veterans Affairs Office of Mental Health and Suicide Prevention
11:35 a.m.	Open Discussion with Audience What conditions make self-monitoring and reporting acceptable? Does this vary for treatments at different stages of product development or with different baseline knowledge about use in varied patient types and treatment conditions? Can we draw any generalizable lessons about cases in which self-monitoring is acceptable and safe?
12:15 p.m.	Introduction and Presentation to Inform Discussion on Signal Detection: Novel Oral Anticoagulants in Comparison with Warfarin

Page 195 Cite

Suggested Citation:"Appendix C: Workshop Two Agenda." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.

×

12:30 p.m.	Open Discussion with Audience What conditions and training prepare clinical care providers to monitor patient safety outside a tightly controlled environment? How does this vary for treatments at different stages of product development or with different baseline knowledge about use in varied patient types and treatment conditions? How do you decide which variables require strict adherence to “protocol” and which can be allowed to vary?
1:00 p.m.	BREAK (Lunch Available Outside Room 120) (Workgroup participants gather to synthesize audience feedback)
2:00 p.m.	Workgroup Presents Synthesis of Audience Feedback

Session III: When Can We Learn from Real-World Treatment Assignment

Session Discussion Questions:

When can we have confidence in inference from cluster-randomized or stepped-wedge study designs?
Under what conditions can we trust inference from observational or naturalistic comparisons?
How could we judge the validity of observational comparisons in advance, rather than waiting until we have observed the result?

Moderator:

Richard Platt, Harvard Medical School

Session Discussants

ROBERT CALIFF

Vice Chancellor, Health Data Science, Duke University Scientific Advisor, Verily Life Sciences

DAVID MADIGAN

Professor of Statistics

Dean, Faculty of Arts and Sciences

Columbia University

DAVID MARTIN

Associate Director for Real-World Evidence Analytics

U.S. Food and Drug Administration

Page 196 Cite

Suggested Citation:"Appendix C: Workshop Two Agenda." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.

×

2:20 p.m.	Introduction and Presentation to Inform the Discussion: Health Care Database Analyses of Medical Products for Regulatory Decision Making
2:20 p.m.	SEBASTIAN SCHNEEWEISS Professor of Medicine and Epidemiology Harvard Medical School Brigham & Women’s Hospital
2:50 p.m.	Open Discussion with Audience Random assignment is always preferable, but when is the cost (in time, money, infrastructure, patient exposure) truly necessary? How can we know that the effects from unmeasured confounders are not so large that they would change a decision based on information from an observational study? What are some of the conditions under which there is more confidence in inference from non-randomized comparisons (examples of some conditions below)? Expectation of large effects Outcome proximal to treatment High degree of similarity between comparison groups Pathway from treatment to outcome is relatively clear, and without lots of complexity or reciprocal effects Treatment allocation method is relatively transparent
3:40 p.m.	BREAK
4:00 p.m.	Open Discussion with Audience and Reflections from Panelists
5:00 p.m.	ADJOURN WORKSHOP DAY 1

Page 197 Cite

Suggested Citation:"Appendix C: Workshop Two Agenda." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.

×

DAY 2: MARCH 7, 2018

8:30 a.m.

Breakfast Available Outside Room 120

Session IV: Synthesizing the Use Cases

Session Objectives:

Discuss key considerations presented in each session on Day 1
Consider components of a potential “checklist” for using real-world evidence

9:00 a.m.	Welcome and Recap of Day 1
	MARK MCCLELLAN, Workshop Series Co-Chair Director Duke-Margolis Center for Health Policy
	GREGORY SIMON, Workshop Series Co-Chair Investigator Kaiser Permanente Washington Health Research Institute
9:20 a.m.	Open Discussion with Audience of Outputs from Day 1 and Potential Components to a “Checklist” for Using Real-World Evidence
10:40 a.m.	BREAK
11:00 a.m.	Open Discussion with Audience of Outputs from Day 1 and Potential Components to a “Checklist” for Using Real-World Evidence
12:30 p.m.	ADJOURN WORKSHOP DAY 2

Page 198 Cite

Suggested Citation:"Appendix C: Workshop Two Agenda." National Academies of Sciences, Engineering, and Medicine. 2019. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington, DC: The National Academies Press. doi: 10.17226/25352.

×

Future Workshop Objectives

WORKSHOP THREE: Examine and suggest approaches for operationalizing the collection and use of real-world evidence (July 17–18, 2018, Washington, DC)

Applications for using real-world evidence to supplement traditional clinical trials, pragmatic/effectiveness trials, or routine clinical application.
Mechanisms for determining which discrete types of real-world evidence could support regulatory decisions.
Operational challenges and barriers for generating and incorporating real-world evidence in the context of a learning health system and how clinicians can best be involved in the collection and usage/utilization of real-world evidence.