The Learning Healthcare System Workshop Summary (2007) / Chapter Skim
Currently Skimming:
2 The Evolving Evidence Base—Methodologic and Policy Challenges
2 The Evolving Evidence Base—Methodologic and Policy Challenges
Pages 81-150
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 81... ...
Califf presents an overview of the alternatives to large randomized controlled trials (RCTs) , and Telba Irony and David Eddy present three methods that have been developed to augment and improve current approaches to generating evidence.
|
From page 82... ...
Eddy presented his work with Archimedes to demonstrate how the use of mathematical models is a promising approach to help answer clinical questions, particularly to fill the gaps in empirical evidence. Many current gaps in evidence relate to unresolved questions posed at the conclusion of clinical trials; however most of these unanswered questions do not get specifically addressed in subsequent trials, due to a number of factors including cost, feasibility, and clinical interest.
|
From page 83... ...
Pharmacogenetics thus offers the potential ability to identify subpopulations of risk or benefit through the development of clinically useful diagnostics, but only if we begin to amass the data, methods, and resources needed to support pharmacogenetics research. The final cluster of papers in this chapter engage some of the policy issues in expanding sources of evidence, such as those related to the interoperability of electronic health records, expanding post-market surveillance and the use of registries, and mediating an appropriate balance between patient privacy and access to clinical data.
|
From page 84... ...
workshop, sponsored by the National Cancer Policy Forum, examined some of the issues surrounding HIPAA and its effect on research, and a formal IOM study on the topic is anticipated in the near future. EVOLVING METHODS: ALTERNATIVES TO LARGE RANDOMIzED CONTROLLED TRIALS Robert M
|
From page 85... ...
Nevertheless, the sheer volume of clinical decisions made in the absence of support from randomized controlled trials requires that we understand the best alternative methods when classical RCTs are unavailable, impractical, or inapplicable. This discussion elaborates upon some of the alternatives to large RCTs, including practical clinical trials, cluster randomized trials, observational treatment comparisons, interrupted time series, and instrumental variables analysis, and reviews some of the potential benefits and pitfalls of each approach.
|
From page 86... ...
Well-designed PCTs offer a solution to the "outsourcing" of clinical trials to populations of questionable relevance to therapeutic questions better addressed in settings where the treatments are intended to be used. Of course, the growth of clinical trials remains important for therapies that will actually be used in developing countries, and appropriate trials in these countries should be encouraged (Califf 2006a)
|
From page 87... ...
In other words, there is a disassociation between the experts in analysis of observational clinical data and the decision makers. There are also an increasing number of sources of data for decision making, with more and more healthcare systems and multispecialty practices developing data repositories.
|
From page 88... ...
For example, certain sites are assigned to intervention A, others use intervention B, and a third group serves as a control. In large regional quality improvement projects, factorial designs can be used to test more than one intervention.
|
From page 89... ...
Resolving some of these issues with definitions of outcomes and missing data will be greatly aided by development of interoperable clinical research networks that work together over time with support from government agencies. One example is the National Electronic Clinical Trials and Research (NECTAR)
|
From page 90... ...
Based on these findings, many clinicians and policy makers assumed that by giving a drug to manage the anemia and improve hematocrit levels, outcomes would also be improved. Thus, erythropoietin treatment was developed and, on the basis of observational studies and very short term RCTs, has become a national practice standard.
|
From page 91... ...
These examples of highly touted observational studies that were ultimately seen to have provided incorrect answers (both positive and negative for different interventions) highlight the need to improve methods aimed at mitigating these methodological pitfalls.
|
From page 92... ...
. Thus the application of clinical practice guidelines and performance measures seems to be working, but all of us continue to dream about improving the available evidence base and using this evidence on a continuous basis.
|
From page 93... ...
However, it needs to be replaced in many situations by the PCT, which has a distinctly different methodology but includes the critical element of randomization. Given the enormous number of decisions that could be improved by appropriate decision support however, alternative methods for assessing the relationships between input variables and clinical outcomes must be used.
|
From page 94... ...
Through this initiative, CDRH is expanding current efforts to promote scientific innovation in product development, focusing device research on cutting-edge science, modernizing the review of innovative devices, and facilitating a least burdensome approach to clinical trials. Ongoing efforts include the development of guidance documents to improve clinical trials and to maximize the information gathered by such trials, the expansion of laboratory research, a program to improve the quality of the review of submissions to the CDRH, and expansion of the clinical and scientific expertise at the FDA.
|
From page 95... ...
It seeks to conduct smaller and possibly shorter trials, and to create a better decision-making process. Well-designed and conducted clinical trials are at the center of clinical decision making today and the clinical trial gold standard is the prospectively planned, randomized, controlled clinical trial.
|
From page 96... ...
. They are often not controversial when the prior information is based on empirical evidence such as prior clinical trials.
|
From page 97... ...
. Another example is the clinical trial designed to assess the safety and effectiveness of the LT-CAGEâ„¢ Tapered Fusion Device, by Medtronic Sofamor Danek, approved in September 2000.
|
From page 98... ...
As a consequence, CDRH is committed to a Post-market Transformation Initiative and recently published two documents on the post-market safety of medical devices. One describes CDRH's post-market tools and the approaches used to monitor and address adverse events and risks associated with the use of medical devices that are currently on the market (see "Ensuring the Safety of Marketed Medical Devices: CDRH's Medical Device Post-market Safety Framework")
|
From page 99... ...
It also helps reduce the burden, uncertainty, and variations that plague decisions based on subjective judgments. Ideally, we would answer every important question with a clinical trial or other equally valid source of empirical observations.
|
From page 100... ...
First, we must understand that models will never be able to completely replace clinical trials. There are several reasons.
|
From page 101... ...
For example, one could have a fairly high degree of confidence in a model's results if the question is about a subpopulation of an existing trial whose overall results the model has already predicted. Other examples of analyses about which we could be fairly confident are the following: • Head-to head comparisons of different drugs, all of which have been studied in their own placebo-controlled trials, such as com paring atorvastatin, simvastatin, and pravastatin; • Extension of a trial's results to settings with different levels of phy sician performance and patient compliance; • Studies of different doses of drugs, or combinations of drugs, for which there are good data from phase II trials on biomarkers, and there are other trials connecting the biomarkers to clinical outcomes; • Extensions of a trial's results to longer follow-up times; and • Analyses of different mixtures of patients, such as different propor tions of people with CAD, particular race/ethnicities, comorbidities,
|
From page 102... ...
This is certainly an encouraging finding and is sufficient to stimulate interest in diabetes prevention. However 2.8 years or even 4 years is far too short to determine the effects of these interventions on the long term progression of diabetes or any of its complications; for example: • Do the prevention programs just postpone diabetes or do they prevent it altogether?
|
From page 103... ...
. Basically, the core of the model is a set of ordinary and differential equations that represent human physiology at roughly the level of detail found in general medical textbooks, patient charts, and clinical trials.
|
From page 104... ...
. The Archimedes model also accurately simulated several trials that 0.5 0.45 Trial: control group Trial: Metformin 0.4 Trial: Lifestyle 0.35 Model: control group Fraction of Patients Model: Metformin 0.3 Model: Lifestyle 0.25 0.2 0.15 0.1 0.05 0 0 0.5 1 1.5 2 2.5 3 3.5 4 Years FIGURE 2-1 Model's predictions of outcomes in Diabetes Prevention Program.
|
From page 105... ...
Thus the prevention of diabetes in high-risk people meets the criteria outlined above -- it is impractical or impossible to answer the important questions with real clinical trials, there is a model capable of addressing the questions at the appropriate level of physiological detail, and the model has been successfully validated against a wide range of adjacent clinical trials. Methods Use of the Archimedes model to analyze the prevention of diabetes in high risk people has been reported in detail elsewhere (Eddy et al.
|
From page 106... ...
Lifestyle modification, as offered in the DPP and continued until a person develops diabetes, would reduce the incidence of diabetes to about 61 percent, for a relative reduction of 15 percent. Thus, over a 30-year horizon the DPP lifestyle modification would actually prevent diabetes in about 11 percent of cases, while delaying it in the remaining 61 percent.
|
From page 107... ...
2-3 betes are shown in Table 2-1. The 30-year rate of serious complications (including myocardial infarctions, congestive heart failure, retinopathy, stroke, nephropathy, and neuropathy)
|
From page 108... ...
The DPP had three arms: control, metformin begun immediately (i.e., when the patient is at risk of developing diabetes, but has not yet developed diabetes) , and lifestyle modification begun immediately.
|
From page 109... ...
16.125 0.276 QALYs (discounted 3%) 11.319 0.159 ABBREVIATIONS: DPP – Diabetes Prevention Program; CAD – coronary artery disease; CHF – congestive heart failure; MI – myocardial infarction; ESRD – end-stage renal disease; CHD – coronary heart disease; QALY – quality-adjusted life-year NOTE: For each time horizon, the entries are the chance of having a complication or the decrease in chance of a complication, up to the end of that time horizon.
|
From page 110... ...
to Health Plan DPP Lifestyle vs No Program 4000 Lifestyle after FPG+125 3000 vs. No Program 2000 1000 0 0 200 400 600 -1000 -2000 Annual Cost of Lifestyle Program FIGURE 2-4 Costs of two programs for diabetes prevention.
|
From page 111... ...
One is to ignore the lack of information about the magnitudes of the effects and promote the prevention activities on the general principle that their benefits have been shown. Since this option is not deterred by a lack of information about actual health or economic effects, it might as well promote the most expensive and effective intervention -- in this case intensive lifestyle modification begun immediately.
|
From page 112... ...
The main factor that will determine the pace of improvement is the availability of person-specific data. Access to such data should increase with the spread of EHRs, as more clinical trials are conducted, as the person-specific data from existing trials are made more widely accessible, as models push deeper into the underlying physiology, and as modelers focus more on validating their models against the data that do exist.
|
From page 113... ...
The major concern for clinical/health services researchers and policy makers is the identification of appropriate "inference groups." To whom are the results of trials being applied and for what purpose? Patients with multiple comorbidities are commonly excluded from clinical trials.
|
From page 114... ...
Some have argued for expanded use of experimental designs (n of 1 trials, multiple time series crossover studies, matched pair analyses) that, unlike parallel group clinical trials, can examine individual treatment effects directly.
|
From page 115... ...
With respect to clinical trials, the literature and clinical experience suggest that the problem of identifying subgroups that may be differentially affected by the same treatment is critical, both when the trial results are small or negative and when the trials demonstrate a positive average treatment effect. It has been assumed in devising guidelines, paying for treatments, and setting up quality measures that subgroups behave similarly to the population average.
|
From page 116... ...
is one reason trial results don't apply equally to all patients. SOURCE: Adapted from Kent, D, and 2-5 R Hayward, When averages hid individual differences in clinical trials, American Scientist, 2007, vol.
|
From page 117... ...
. A second RCT focusing on patients with low baseline outcome risk (APACHE < 25)
|
From page 118... ...
. Therefore, in these patients the effectiveness of aggressive treatment is substantially lower than that observed in clinical trials because these trials were conducted with younger patients without such comorbidities (Greenfield et al.
|
From page 119... ...
2006) demonstrated that a composite risk score derived from a multivariate model predicting outcome risk alone significantly increased statistical power when assessing HTE compared to doing individual comparisons.
|
From page 120... ...
The recommended applications of these phases for studying each of the four sources of HTE are outlined in Table 2-3 and described below. Baseline outcome risk in clinical trials.
|
From page 121... ...
2003) can be modeled to identify predictors of baseline outcome risk for pre-trial subgroup specification at a much lower cost per subject than a second RCT.
|
From page 122... ...
. The most critical issue for understanding competing risk is deciding when a clinical quality threshold measure shown to be "effective" in clinical trials (e.g., HbA1c < 7.0 percent for diabetes)
|
From page 123... ...
. When treatment effects are underestimated for a subgroup, treatments that are arduous, have multiple side effects, or are burdensome over long periods of time may be rejected by doctors and patients, even if the patient belongs to a subgroup likely to benefit from treatment.
|
From page 124... ...
In this particular case, there is virtually no information on who will get TD or who will suffer severe weight gain such that they would not continue medication. These types of results unambiguously constitute a call to arms to the genetics community because this is an area in which we can truly add value by helping clinicians to distinguish patients and guide clinical decision making.
|
From page 125... ...
These are the types of things we are working toward in the pharmacogenetics community, and it looks as though some of these problems are quite crackable and there will be genetic diagnostics of significant relevance and impact to clinical decisions. The idea of moving toward doing more genetic studies in the context of a trial is quite exciting because the data will be quite rich.
|
From page 126... ...
If pharmacogenetic predictors of adverse events could prevent the exposure of genetically vulnerable patients and so preserve even a single drug, the costs of any large-scale research effort in pharmacogenetics could be fully recovered. An example of this is vigabatrin, which is a good antiepilepsy drug in terms of efficacy, and for some types of seizures (e.g., infantile spasms)
|
From page 127... ...
The examples presented illustrate several of the challenges and opportunities for pharmacogenetics. These types of information will be increasingly generated, but we need to think about how such information will be useful and utilized for clinical decision making.
|
From page 128... ...
In addition, policy development should strive to find the right alignment of incentives and controls to accelerate adoption of evidence on medicine, technology, and services that advance the standard of care. The current landscape of health care in the United States is one of organized chaos where providers, payers, employers, patients, and manufacturers often have different vantage points and objectives that can result in inadequate patient access, poor care delivery, inconsistent quality, and increasing costs.
|
From page 129... ...
To properly evaluate new products we need to acknowledge the advantages and limitations of the methods we have historically used for regulatory approval. Randomized clinical trials with blinding are currently used in the approval of drugs and higher-risk devices to ensure high internal validity of findings.
|
From page 130... ...
. Randomized clinical trials with blinding are the gold standard for drug evaluations of safety and efficacy but may not be possible in device studies.
|
From page 131... ...
that a common dataset would qualify for both safety assessments and coverage determinations; (2) input from both agencies for specific data collection requirements to inform the design of the data collection tools and the management of the data; (3)
|
From page 132... ...
For example, the data contained within the placebo arm of randomized controlled trials could provide a wealth of information when pooled together and would provide a larger cohort of populations for comparative analyses. Another example of potential collaboration is in the design of these new EHR systems, especially for
|
From page 133... ...
establishing appropriate peer review processes to ensure rigor and timeliness; (4) requiring intensive education and training of health professionals on evidence generation in clinical practice; and (5)
|
From page 134... ...
We believe a collaborative effort supported by patients, physicians, payers, industry, and regulators can accomplish the goal of a learning healthcare system with an open and transparent process toward developing standards and interoperable capabilities.
|
From page 135... ...
Moreover, estimation of benefits from RCTs may differ substantially from what is achieved for typical patients in real-world practice. How then should these different sources of information (e.g., randomized clinical trials, effectiveness studies, observational studies, models)
|
From page 136... ...
It is absolutely plausible that different payers will establish different rules leading to very different coverage decisions at substantially different costs, which will be attractive to different constituencies. However, potential elements in a taxonomy of decisions may include the quality of evidence, the magnitude of effect, the level of uncertainty regarding harms and benefits, the existence or absence of good treatment alternatives, the potential for decision regret, precedent, and acceptability.
|
From page 137... ...
We can also benefit from participation by nontraditional professionals, such as actuaries and operations research scientists, who use different methodologic approaches. To create a taxonomy of decisions and identify the level of certainty required for each, it will be necessary to convene payers, health professionals, and patients along with methodologic experts.
|
From page 138... ...
Concerns cited during the workshop about these HIPAA regulations revealed broad implications for the notion of a learning healthcare system. The prospect of learning as a by-product of everyday care rests on the notion of collecting the data that results from these everyday clinical interactions.
|
From page 139... ...
was enacted to improve the portability and continuity of health insurance; combat waste, fraud, and abuse in health insurance and healthcare delivery; promote medical savings accounts; improve access to long-term care services and coverage; and simplify the administration of health insurance. The Administrative Simplification "Standards for Privacy of Individually Identi fiable Health Information" (the Privacy Rule)
|
From page 140... ...
. This workshop, which brought together participants from a variety of public, private and scientific sectors, including researchers, research funders, and those who had participated in preparation of the Privacy Rule, identified a number of issues to be addressed when clinical data are used to generate evidence and cast light on the lack of data about the quantitative and qualitative effects of HIPAA on the conduction of clinical research.
|
From page 141... ...
During speeches to Congress and the public in 1993, Former President Bill Clinton touted a prototype "health security card" that would allow Americans to carry summaries of their medical records in their wallets. In response, Former Senate Minority Leader Bob Dole decried the health plan as "a compromise of privacy none of us can accept." And yet, in his State of the Union address last month, President Bush advocated "computerizing health records [in order to]
|
From page 142... ...
People have lost jobs and suffered stigma and embarrassment when details about their medical treatment were made public. Putting health information in electronic form, and creating the technical capacity to merge it with the push of a button, only magnifies the risk.
|
From page 143... ...
First Principles In resolving the conjoined dilemmas of linking personal health information and maintaining confidentiality, the Health Privacy Project urges an adherence to the following first principles: • Any system of linkage or identification must be secure, limiting dis closures from within and preventing unauthorized outside access. • An effective system of remedies and penalties must be implemented and enforced.
|
From page 144... ...
1983. Outcome in one-vessel coronary artery disease.
|
From page 145... ...
2004. Primary prevention of cardio vascular disease with atorvastatin in type 2 diabetes in the Collaborative Atorvastatin Diabetes Study (CARDS)
|
From page 146... ...
Analyzing the results of clinical trials to expose individual patients' risks might help doctors make better treatment decisions. American Scientist 95(1)
|
From page 147... ...
2005. High-dose atorvastatin vs usual-dose simvastatin for secondary prevention after myocardial infarction: the IDEAL study: A randomized controlled trial.
|
From page 148... ...
2005. Impact of the pulmonary artery catheter in critically ill patients: meta-analysis of random ized clinical trials.
|
From page 149... ...
2003. Practical clinical trials: Increasing the value of clini cal research for decision making in clinical and health policy.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.