“Our clinicians engage in research now. It’s the usual care of the patient, and we generate research information from it.” —Louis Fiore, VA Boston Healthcare System
The workshop session on innovations in trial design began with a series of challenging questions by the moderator, Michael Krams, Vice President, Head of Neurology Franchise, Johnson & Johnson. How can we conduct clinical trials such that actionable results come from them? How can we translate the creation of knowledge into impact for society? Will innovative trial designs let us emulate the stunning performance improvements that have been accomplished in computing? And, as was mentioned by Bram Zuckerman, Director, Division of Cardiovascular Devices, Center for Devices and Radiological Health (CDRH), FDA, from a practical standpoint, can we create a library of case studies so that people can see how these methods work?
While the annual number of new drug approvals in the United States has remained relatively flat for the past several decades—hovering more or less between 18 to 25—what has not been constant are the costs of drug trials, which have been increasing at about triple the inflation rate. Part of the problem stems from the success in genomic sequencing and the explosion in the number of new, less well validated targets, with their resultant high failure rates.
1 Material in this section is based on the presentation by Michael Parides, Professor of Biostatistics, Department of Health Evidence and Policy, Mount Sinai School of Medicine.
These rising costs can be considered not as the price of success, but the price of failure—an insight credited to Robert Hemmings.2 If clinical trial designs could detect failure sooner, in phase I or even phase II, then trials would not proceed to the later phases, where costs have been increasing most dramatically.
There are many ways for a trial to fail, said Michael Parides, Mount Sinai School of Medicine. Perhaps the compound simply does not work, or not at the dose being tested, or not in the expected patient population. Sometimes the study design is not optimal or the entire drug development plan is flawed. Many phase III trials that fail have problems that can be traced to phase I and II trials that did not produce the quality of information needed for the confirmatory trial to be designed appropriately.
Improved clinical trial designs hold great promise for making the clinical trial enterprise more efficient, primarily by earlier detection of inadequate benefit. At the same time, treatments that do offer benefit need to be accurately recognized, so that they are not prematurely abandoned, he said. Reliably discarding compounds that do not work and keeping those that do increases the overall trial success rate.
A promising approach to improving trial design is “adaptive design.” Adaptive design is not a new idea, but it is becoming increasingly interesting to researchers. In general, adaptive designs use interim data to modify an ongoing trial without undermining its validity and integrity or introducing bias. Modifications might include correcting inaccurate assumptions or reestimating the sample size. The adaptations are carefully planned in advance and are prespecified, such that, while the trial design is flexible, it is not completely open-ended. There are numerous variations on the adaptive design theme, some more accepted than others.
Recent developments have made adaptive trial designs more feasible. Perhaps most important is the increased use of Bayesian statistical methods, made feasible by desktop computing power. Bayesian approaches allow continual reassessment of trial findings with respect to, for example, maximum tolerable dose. Rather than assigning patients to trial doses according to an algorithm that does not make dose-limiting toxicity explicit, in the Bayesian approach, the researcher makes an assumption about the relationship between dose and toxicity; data are collected; the relationship is reassessed; and the process repeats through some number of cycles. The key element, Parides said, is the notion of continuous learning: Each new patient has the benefit of what was learned from each previous patient. Most such applications require simulations, an approach
2 Dr. Hemmings is Statistics Unit Manager of the U.K. Medicines and Healthcare Products Regulatory Agency.
that fits the drug development paradigm well, as a series of revisions of an original idea, updated with additional information. This is a useful model for exploratory trials.
A second new element is the bolder nature of some types of adaptive design, for example, unblinded sample size estimation, changing the primary end point, patient enrichment (that is, once a trial starts, the investigator perceives there may be subgroups for which the treatment works better and stops randomizing people outside of those subgroups), and seamless phase II/phase III designs. Strategies such as these require rigorous statistical management.
Information gained from the use of these statistical methods allows researchers to abandon a trial arm or curtail an entire trial early in development if it is not expected to work. For example, “adaptive dose-finding randomization” helps the researcher decide to drop treatment arms based on initial responses, whether due to toxicity, efficacy, or both; and even in a phase III confirmatory trial, where adjustments must be made cautiously, group sequential procedures are an accepted way to gain information that can lead to midcourse corrections or even trial termination.
A real-world trial conducted in LVAD recipients—people with end-stage heart failure surgically implanted with mechanical heart pumps— provides an example. The investigators wanted to know whether the trial was generating enough information to warrant continuation. Parides showed how the trial data would look if there were 20 patients each in the control group and the active therapy group. The number of treatment failures in the two groups was 13 (controls) and 10 (treatment). Conventional statistical methods would make it hard to judge whether the trial should be continued. The p value is 0.52, with a wide confidence interval. However, the customary reliance on p values in this case would be misguided, Parides said. When the same data are analyzed using Bayesian approaches, the first step is to assess the success probability for both groups. Due to the assumptions of clinical equipoise, these probabilities are the same, albeit unknown. After the data are collected, they are refitted to the model, revealing that the probability that the treatment is better than control is 75 percent. In this case the Bayesian analysis was a much more accurate way to present the data and one that made the decision to move forward with the trial clear.
At the more controversial end of the spectrum of potential trial adaptations is changing the study’s primary endpoint. Serious problems can arise, as occurred a decade ago in the CAPRICORN trial, a multicenter, multinational, randomized, double-blind, placebo-controlled trial of whether beta-blockers plus routine medical management performed better than routine management alone after a heart attack (Colucci, 2004). The initial primary endpoint was all-cause mortality. As the trial was
under way, the mortality rate was lower than expected, but, because the study was blinded, the investigators did not know whether that was because one of the interventions was working very well, or whether both were effective. At that point, the researchers elevated a prespecified secondary endpoint—cardiovascular hospitalization—to be the coprimary endpoint. Unfortunately, it turned out that the result for this combined endpoint was not statistically significant, while the original would have been (a 23 percent reduction in all-cause mortality).
While adaptive designs undeniably have appeal, Parides said, they are not always better and not necessarily logistically simple or less expensive. On a trial-by-trial basis, adaptive designs may cost more, but they save money overall, he said, by preventing investment in futile exercises.
Motivating researchers to change the way they work and methods they use is difficult, Krams said. “As we all know, in the clinical R&D environment, culture eats strategy for lunch.” When the number of scientists in academia, industry, and at FDA who are familiar with adaptive research methods gradually increases, these new methods may become more acceptable, said Parides. Methodological problems will be resolved, some approaches will fall by the wayside, and some will eventually become second nature.
Currently, researchers do what is feasible because it is feasible, rather than because it will produce the most useful endpoints, Krams said. The incentives are geared toward obtaining results as quickly as possible. Krams advised looking beyond an individual trial to an entire research program and assessing how many times a second trial is needed because of inconclusive answers to the research question. A more productive way of framing the incentives, therefore, is to work toward achieving the best information value per research unit invested.
Randomized clinical trials remain the gold standard for determining a treatment’s safety and efficacy, but their high costs and extended timelines and the delayed integration of their results into clinical care are problematic. Observational studies are less expensive and produce quicker results, but their findings are less reliable. Louis Fiore, Assistant Professor, VA Boston Healthcare System, described an initiative to meld the two methods, called “point-of-care clinical trials,” which uses randomization
3 This section of the report is based primarily on a presentation by Louis Fiore, Assistant Professor, U.S. Department of Veterans Affairs (VA) Boston Healthcare System.
to remove selection bias in an observational study. The method is being tested within the VA and led by the Boston VA Healthcare System. The long-term goal of this new research approach is to create a true learning health care system,4 he said.
In most medical facilities, research and clinical departments are completely separate. Poorly funded researchers raise their own money and not until some years after their project ends do their findings become adopted by the clinical side of the house. A point-of-care approach would speed up adoption of treatment improvements and allow researchers to leverage the resources of the clinical services.
The VA is well positioned to try this approach for several reasons: its clinicians are interested in it; VA patients continually return to the system over a period of years; and the VA has a sophisticated EHR system, which allows patient records to be accessed from any VA facility.
According to Fiore, point-of-care clinical research is a quality improvement strategy, and it has a number of advantages over traditional research methods:
• The studies compare existing approved drugs with known toxicities and therefore do not involve IND applications or require IRB approval—they ask “which is better?” questions.
• They are designed so that a substantial portion of their operations can be conducted by clinical staff as part of routine care delivery, with study data captured passively in the EHR.
• They do not disrupt regular clinical care, require lengthy and repeated discussions with patients, or demand unusual data collection.
• Participating patients are drawn from the day-to-day caseload of the VA hospital, with minimal exclusion criteria, and thus are a generalizable population.
• Minimal extra outpatient visits are required; when participants are discharged and return as outpatients, the next set of data is captured.
• The researcher has access to real experiences and outcomes, not surrogate ones, and follow-up can continue as long as desired.
• The research is low cost, even through the follow-up phase, costing only an estimated 10 to 30 percent of the cost of an industry-sponsored trial.
4 The learning health care system can be defined as the seamless and continuous development and application of evidence in the course of patient care. In such a system, each patient care experience naturally reflects the best available evidence, and, in turn, adds seamlessly to learning what works best in different circumstances (IOM, 2008).
• Finally, since the research is being done in the same health care system that is going to use the results, physicians are more likely to have confidence in them. In fact, they have generated the results themselves.
In short, this is a pragmatic approach to study design that produces very pragmatic results, germane to a hospital’s specific patient population (see Box 7-1). Generalizing to a different facility requires analyzing differences in the health care systems and patients served, not the treatment being tested, Fiore said.
Administratively, point-of-care research is facilitated by what Fiore terms a “rational approach” to regulatory oversight and obtaining informed consent. The question of whether clinicians are participating in a trial, versus merely going about their regular business, is an important
To test the feasibility of point-of-care research, VA investigators at the eight VA hospitals in the six-state New England region are comparing weight-based versus sliding-scale determinations of insulin dose in non-intensive care unit patients with diabetes. VA physicians and the clinical literature have been divided on which approach is best. The study’s primary endpoint is hospital length-of-stay, and the secondary endpoint is glycemic control and readmission within 30 days.
When patients enter the hospital needing insulin, physicians (via EHR) are presented with three options. This is the “point of care.” Options 2 and 3 provide an insulin regimen according to usual weight-based or sliding-scale protocols. The first option is “no preference” and invites clinicians to enroll their patients in a study comparing the two protocols. If they choose the study, the patient is automatically randomized to one or the other treatment, a nurse obtains consent, and a progress note about the study is automatically entered in the record.
From there the computer takes over, writing the orders for the clinician to sign. The study is fully integrated into the hospital’s informatics system, with the EHRs tracking which of the two treatments produces the best outcomes. Using adaptive randomization and Bayesian approaches, randomization may start out 1:1, but as one arm of the trial becomes statistically superior, randomization will change to 60:40, 70:30, and so on. Eventually, new patients will be randomized 99:1 to the effective arm, and the study will conclude. In effect, the more successful treatment will become the standard of practice in the facility “directly as the study is happening,” Fiore said.
a Based on the presentation by Louis Fiore, Assistant Professor, VA Boston Healthcare System.
one. Declaring them “researchers” might impose regulatory requirements related to, for example, serious adverse-event reporting. If drugs are well established (like warfarin), Fiore queried, is it really necessary to report adverse events (like bleeding)? Similarly, is written informed consent needed for low-risk comparison studies—for example, one insulin type versus another—or will oral consent suffice? Ideally, Fiore believes, when a patient is admitted to a VA facility and provides the usual consent to care, the form would include consent to participate in this type of research. This blanket “opt-in” consent would be documented in the EHR.
The electronic record is key to the efficiency and national scalability of point-of-care research. Researchers work with information technology staff to modify the record to incorporate tools and bits of code that allow it to randomize, extract data, and create notifications. At the end of the study, the randomization node can be easily changed to a decision-support node. And, economic analyses are simple because all the costs are already recorded in the health system.
Because VA patients’ electronic records are available at any VA facility nationwide, additional opportunities to participate in trials present themselves. For example, a veteran at a VA facility remote from any research center might have prostate cancer (or other tissue) analyzed, with the results recorded in the EHR. At another VA site engaged in prostate cancer research, a drug trial might be under way for which the patient would be an appropriate participant. Mining the EHR data allows that patient to be identified and facilitates the patient’s engagement in the trial. Fiore said this would move research to the patient, rather than the patient to the research.
Another potential benefit of point-of-care trials would be to create a culture change in the way clinicians and patients think about treatment trials. If doctors and patients want treatments based on the best medical knowledge, with strategies that have been tried and tested—in other words, if they want to provide and receive evidence-based medicine— then they need to be part of the evidence-gathering process.
A fundamental challenge to the diffusion of point-of-care research is the lack of appreciation and reward for collaborative work within academic medicine. The days of lone investigators owning data and carrying out projects in isolation are numbered in the clinical sciences, Fiore believes. Yet, the academic infrastructure has not even begun to dismantle these silos. Additionally, Zuckerman mentioned that the training environment needs to change, so that medical schools produce physicians who are clinical trialists and clinical research courses become a standard part of the curriculum.
The point-of-care model requires a reconsideration of the relationship between clinical care and research. Clinical effectiveness research is “engineering,” and as much as it is needed, there are too many research
questions, too few investigators, and too little funding. Clinical care dollars, being spent in any case, can generate the data. If the health care system used point-of-care research methods to learn from its experiences, it could, Fiore said, “make taking care of patients a whole lot quicker, more effective, and probably cheaper.”
FDA leaders perceive a role for the agency in encouraging innovation and promoting efficient development of new and improved medical products, said Douglas C. Throckmorton, Deputy Director for Regulatory Programs, CDER, FDA. FDA staff attempt to make researchers’ jobs easier by designing clear and thoughtful rules and interpretations, publishing guidances regarding them, and ensuring they are applied equally to everyone. The goal is to help innovation, he said, not hinder it (FDA, 2010, 2011d). For example, draft FDA guidance on the use of adaptive trial designs has been released. The document starts with adaptive designs that are used regularly and would be rather easy to accept and ends with some of the more complicated and problematic adaptations that might take a fair degree of discussion.
Promoting an efficient process for medical products development means more than waiting for researchers to submit their trial applications, Throckmorton said. It means supporting appropriate collaborations, building opportunities, developing standards that enable efficient drug development, and building support for academic science. FDA staff frequently engages in partnerships, collaborations, and consortia, such as CTTI, a public-private collaboration with Duke University. CTTI members, drawn from government, industry, academia, and patient groups, are examining and prioritizing the major challenges in the conduct of clinical trials, with the goal of increasing their quality and efficiency.
In addition, FDA is attempting to make its own operations more efficient, Throckmorton said. The agency is working to focus the clinical trials monitoring program on trial sites where the most problems are likely, rather than treating all sites equally. In an effort to build quality into a clinical trial from the beginning, Pfizer and FDA staff are conducting a pilot test in which they are simultaneously designing a phase III study and its monitoring program.
A wide range of regulatory approaches is necessary to carry out FDA’s regulatory authority over devices, which include everything from
5 This section is based on the workshop presentations of Douglas C. Throckmorton, Deputy Director for Regulatory Programs, Center for Drug Evaluation and Research (CDER), FDA; and of Bram Zuckerman, Director, Division of Cardiovascular Devices, CDRH, FDA.
sterile gloves to LVADs. The most stringent regulations cover the Class III, high-risk, life-supporting products that require premarket approval (PMA), which in many ways is similar to the approval pathway for a new drug.
The agency’s challenge is to ensure the safety and effectiveness of medical devices even as science is continually evolving, devices are becoming increasingly complex, and the existing regulatory pathways to market were established in 1976. In the meantime, Zuckerman said, some of the distinctions between “drug” and “device” have blurred.
A practical consideration for device developers is the need to engage with the FDA’s CDRH “early and often” regarding its clinical trial strategy, Zuckerman noted. Adaptive designs may be particularly helpful to device developers, as many of them are small companies with correspondingly small research budgets. But even large device manufacturers may want trial results quickly, because a device’s life cycle is often relatively short. An estimated 10 to 15 percent of device applications currently include some combination of Bayesian or adaptive designs.
This page intentionally left blank.