Ethical and Scientific Issues in Studying the Safety of Approved Drugs (2012) / Chapter Skim
Currently Skimming:
3 Evidence and Decision-Making
3 Evidence and Decision-Making
Pages 121-168
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 121... ...
This chapter focuses on the evaluation of the scientific evidence and on how FDA should use evidence in its decisions. Just as courts determine when evidence is admis sible and which standard of proof to apply in a given case, scientific evidence must be evaluated for its quality and applicability to the public health question that is the focus of regulatory decision-making.
|
From page 122... ...
In the context of a drug study, the "proposition" is a hypothesis about a drug effect, often stated in the form of a scientific question, such as "Do broadspectrum antibiotics increase the risk of colitis"? In the broader context of FDA's regulatory decisions, the proposition may be implicit in the public health question that prompts the need for a regulatory decision, such as, "Does the risk of coli tis caused by broad-spectrum antibiotics outweigh their benefits to the public's health"?
|
From page 123... ...
The frequentist approach to statistical inference is familiar to medical research ers and is the basis for most FDA rules and guidance. The Bayesian approach is less widely used and understood, however, it has many attractive properties that can both elucidate the reasons for disagreements, and provide an analytic model for decision-making.
|
From page 124... ...
has published guidance for the use of Bayesian statistics in medical device clinical trials (FDA, 2010a) and FDA has used Bayesian approaches in regulatory decisions.
|
From page 125... ...
But in most common statistical situations, there exists a strongest possible Bayes factor, and that can be defined as a function of the observed P value. That relationship can be used to calculate the maximum chance that the non-null hypothesis is true as a function of the P value and a prior probability (Goodman, 2001; Royall, 1997)
|
From page 126... ...
For that, a prior probability is needed, and the Bayes factor. If the mechanism or some preliminary observa tions justified a 25 percent prior chance of a harmful effect, the same evidence would raise that to at most a 78 percent chance of harm -- that is, at least a 22 percent chance that the drug does not cause that harm.
|
From page 127... ...
As demonstrated in the above paragraph, biologic plausibility and other forms of external evidence are currently accommodated qualitatively; Bayesian approaches allows that to be done quantitatively, providing a formal structure by which both prior evidence and other sources of information (for example, on common mechanisms underlying different harms, or their relationship to disease processes) should affect decisions.
|
From page 128... ...
If other drugs in the same class have been associated with a particular adverse effect, the drug has a higher prior probability of causing that effect than a drug in a class whose mem
|
From page 129... ...
If a drug has a mechanism of action that has been implicated in a particular adverse effect, it has a higher prior probability of causing that effect than a drug for which such a mechanism is implausible. For example, the prior probability that a topical steroid would produce significant internal injury would be very low because what is known about the absorption, metabolism, and physiologic actions of topical steroids makes it difficult to imagine how such an injury could occur, but the prior probability of an adverse dermatologic effect would be much higher.
|
From page 130... ...
Those who have a good understanding of this particular set of pathways might interpret the explanation differently and establish a different starting point for the probability of such an effect. It is unlikely, though, that on the basis of such evidence general consensus could be garnered for a high prior probability of effect.
|
From page 131... ...
Even if two scientists agree about what evidence new data provides, if they have different assessments of the strength of prior evidence they might disagree about the probability of a higher drug risk. Such a disagreement might appear outwardly to be about the new evidence when in fact the disagreement is about the prior probability.
|
From page 132... ...
The quality of databases is variable. In the case of the AERS database, for instance, reporting of adverse events is incomplete, and the quality of the information about the adverse events that are reported may be poor.
|
From page 133... ...
The denominators of membership are known, and entry into and exit from the cohort of patients can be reasonably well defined, allowing calculation of the risk of adverse events. Health insurance databases are likely to capture most drug exposures and serious adverse events requiring medical care, although the complete ascertainment of outcomes may require the use of multiple administrative files.
|
From page 134... ...
The International Classification of Dis eases codes are used worldwide and provide consistency in information on effects, but the codes are periodically updated, and the updates can affect health data both within a study over time and in comparisons among different studies. In addition, nonstandard definitions of endpoints, economic incentives for listing particular diagnoses, and insufficient detail about key variables of interest can affect data quality.
|
From page 135... ...
So data-quality issues can have a central, sometimes irresolvable role in creating disagreement among scientists about numerical results; at best, the plausible range of estimates that would be consistent with their qualita tive disagreement can be calculated with sensitivity analyses. Confidence in a Design's Ability to Eliminate Bias The science of drug safety concerns questions of causal, not just statistical,
|
From page 136... ...
If the evidence pointing to such a relationship has been generated by a well-designed, well-conducted clinical trial in which drug treatment has been randomly assigned and there is adequate size and time for adverse effects to appear, confidence is typically fairly high that the difference in drug exposure is the cause of any differences in benefits or risks. However, if deviations from initial randomization occur (such as that caused by dropouts, missing data, or poor adherence)
|
From page 137... ...
The three main types of bias that affect the internal validity of a study are confounding, selection bias, and information bias, which are described in Box 3-2. Confidence in the Transportability of Results The Concept of Transportability A study estimate of the benefit or risk associated with a drug can deviate from the results that patients would actually experience in wider clinical practice if the study participants were not representative of the wider target population.
|
From page 138... ...
are less likely to have experienced the harm simply because they remained in the study. Other types of selection bias that affect estimates in randomized controlled trials and observational studies include missing data and non response bias, healthy-worker bias, and self-selection bias (Hernán and Robins, 2012)
|
From page 139... ...
The assessed risks in a given population can differ according to how an adverse effect is elicited from the patient. Studies that depend on passive report ing of adverse events versus those that ask patients about specific adverse events can affect the reported frequency several-fold (Bent et al., 2006; Ioannidis et al., 2006)
|
From page 140... ...
Issues related to transportability were raised repeatedly (using the more familiar term generalizability) in the FDA briefing document for the rosiglitazone hearings.
|
From page 141... ...
Observational drug safety studies are often criticized because they lack the experimental design rigor of a controlled clinical trial. Specifically, there is often concern that patients who are prescribed a particular medicine are different from those who are prescribed an alternative treatment, in ways that may be correlated with the outcome of
|
From page 142... ...
Furthermore, the risk estimates from the observational studies are generally similar to those from the meta-analyses of clinical trials. Thus, dismissing the results of the observational studies simply because the observed measures of risk may be due to channeling bias may not be appropriate.
|
From page 143... ...
Confounding by contraindication, however, is not a major concern in studies of unexpected adverse events. If the risk itself or the factors that affect it are unknown, treatment cannot be based on avoidance of the risks (Golder et al., 2011)
|
From page 144... ...
Preapproval RCTs are also likely to miss adverse effects resulting from chronic use or those arising after a long latent period, whereas observational studies, particularly those based on existing data, can typically provide longer followup. Observational studies based on data sources collected from large populations with long follow up can often report a greater number of adverse events than typical RCTs.
|
From page 145... ...
drug risks, RCTs are inca pable of detecting adverse events that may arise only in the populations excluded from the trial, which are often characterized by a wider array of comorbidities, different disease severity, concomitant treatments, or other risk factors (such as age, sex, low socioeconomic status, poor monitoring of dose, adherence, or outcomes) that may modify the effects of treatment.
|
From page 146... ...
are treated. If researchers evaluate many adverse events statistically and only a few are observed to have increased risks, the strength of evidence of those adverse events depends on whether the "data" are treated as all the comparisons taken together or as each taken separately for the specific adverse events whose risks seem to be increased.
|
From page 147... ...
When people drop out of a study or are otherwise lost to followup, their outcomes cannot be ascertained. As a result, regardless of whether the study is an observational study or an RCT, the ITT effect cannot be calculated directly.
|
From page 148... ...
One way to assess the sensitivity of effect estimates to such assumptions is to conduct both ITT analysis to estimate the effect of treatment assignment (with and without adjust ment for loss to followup) and analyses adjusted for adherence to estimate the effect of continuous treatment (via the statistical approaches mentioned)
|
From page 149... ...
The timing of the adverse event relative to the drug exposure might also be an issue; the relevant time window might vary among studies, reflecting disagreement among scientists. Such disagreements are often manifested as arguments among scientists about whether particular aspects of study design are "right" or "wrong".
|
From page 150... ...
. Noninferiority studies are particularly problematic for evaluating safety endpoints (Fleming, 2008; Kaul and Diamond, 2006, 2007)
|
From page 151... ...
This is also a domain in which Bayesian approaches can be helpful which can be used to calculate the probability that either the risk or the benefit–risk margin is within an acceptable range (Kaul and Diamond, 2006)
|
From page 152... ...
That those problems continue and are encountered by FDA was documented in a 2011 report by FDA scientists that outlined the challenges of using meta-analysis to study drug risk (Hammad et al., 2011)
|
From page 153... ...
Jenkins, director of OND, stated in a memorandum (2010) that in weighing the available data for rosiglitazone the primary signals of concern arise from meta-analyses of controlled clinical trials that were not designed to rigorously collect CV outcome data and observational studies.
|
From page 154... ...
, potentially greatly enhancing the value of this kind of evidence synthesis. IPD meta-analysis is a form of data pooling in which the analyst has access to the original data of a study instead of merely the summary effect estimates and can thus adjust for covariates and investigate group effects with stronger confounding control than possible with study-level summaries, and can adjust better for design differences between studies (Cooper and Patall, 2009; Fisher et al., 2011; Jones et al., 2009; Kufner et al., 2011)
|
From page 155... ...
Graham and Gelperin, in the FDA Office of Surveillance and Epidemiology, in their presentation to a July 13–14, 2010, joint meeting of FDA's Endocrinologic and Metabolic Drugs Advisory Committee and Drug Safety and Risk Management Advisory Committee on rosiglitazone (Graham and Gelperin, 2010b) , noted that • The cost of a wrong decision is not symmetric.
|
From page 156... ...
While I cannot dispute that fact, I believe withdrawal of rosiglitazone in the setting of scientific uncertainty is an inappro priate display of FDA's authority to make a decision for all healthcare providers because of concern that these trained professionals can not reasonably decide on or take responsibility for the use of this drug. I am also concerned that such an action would set an unsettling precedent for future regulatory decisions or may be referenced in legal challenges to the FDA to withdraw other drugs based on meta-analyses and observational studies of similar uncertainty for drug risk.
|
From page 157... ...
REPRODUCIBLE RESEARCH, DATA SHARING, AND TRANSPARENCY In addition to direct elicitation of the reasons for disagreements, which were well outlined in the rosiglitazone case, adherence to principles of reproducible research -- an emerging set of standards or principles for presentation of com plex and scientific findings -- would be of substantial help to FDA in enforcing a transparency standard for all results on which regulatory decisions will be made. Principles of reproducible research have been outlined for epidemiologic research (Peng et al., 2006)
|
From page 158... ...
New approaches are needed to facilitate the publication of safety data submitted to FDA for approved drugs, and to find ways to release similar data for drugs that are disapproved, but whose information might be extremely valuable for the interpretation of safety information from approved drugs in the same class. FINDINGS AND RECOMMENDATIONS Finding 3.1 Some of FDA's most difficult decisions are those in which experts disagree about how compelling the evidence that informs the public health question is.
|
From page 159... ...
on the basis of different assessments of prior evidence, the quality of new data, the adequacy of confounding control in the relevant studies, the trans portability of results, the appropriateness of the statistical analysis, the relevance of the new evidence to the public health question, how the evidence should be weighed and synthesized, or the threshold for regulatory actions. Recommendation 3.1 FDA should use the framework for decision-making proposed in Recom mendation 2.1 to ensure a thorough discussion and clear understanding of the sources of disagreement about the available evidence among all participants in the regulatory decision-making process.
|
From page 160... ...
In assessing the relevance of study findings to a public health question, the transportability of the study results is as important as the determinants of its internal validity. Recommendation 3.3 In assessing the benefits and risks associated with a drug in the postmarketing context, FDA should develop guidance and review processes that ensure that observational studies with high internal validity are given appropriate weight in the evaluation of drug harms and that transportability is given emphasis similar to that given bias and other errors in assessing the weight of evidence that a study provides to inform a public health question.
|
From page 161... ...
Recommendation 3.6 For drugs that are likely to have required postmarketing observational stud ies or trials, FDA should use the BRAMP to specify potential public health questions of interest as early as possible; should prospectively recommend standards for uniform definition of key variables and complete ascertainment of events among studies or convene researchers in the field to suggest such standards and promote data-sharing; should prospectively plan meta-analyses of the data with reference to specified exposures, outcomes, comparators, and covariates; should conduct the meta-analyses of the data; and should make appropriate regulatory decisions in a timely fashion. FDA can also improve the validity of meta-analyses by monitoring and encouraging compliance with FDAAA requirements for reporting to ClinicalTrials.gov.
|
From page 162... ...
The guid ance should include discussion of criteria for choosing the standard therapy to be used in the active-treatment control arm; of methods for selecting a noninferiority margin in safety trials and ensuring high-quality trial conduct; of the optimal analytic methods, including Bayesian approaches; and of the interpretation of the findings in terms of the drug's benefit–risk profile. Recommendation 3.7.2 FDA should closely scrutinize the design and conduct of any noninferiority safety studies for aspects that may inappropriately make the arms appear similar.
|
From page 163... ...
2010a. Guidance for industry and FDA staff: Guidance for the use of Bayesian statistics in medical device clinical trials.
|
From page 164... ...
approval process: The FDA paradigm and reflections on hypothesis testing. Controlled Clinical Trials 20(1)
|
From page 165... ...
1995. Robust Bayesian methods for monitoring clinical trials.
|
From page 166... ...
2008. Publication of clinical trials supporting successful new drug applications: A literature analysis.
|
From page 167... ...
Panel on handling missing data in clinical trials. Washington, DC: The National Acad emies Press.
|
From page 168... ...
2009. Progress and deficiencies in the registration of clinical trials.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.