Categories of Scientific Evidence—Human Information and Data
GUIDING PRINCIPLES: A credible report or study finding of a serious adverse event (or experience)1 in humans that is associated with use of a dietary supplement ingredient raises concern about the ingredient’s safety and requires further information gathering and evaluation. A final judgment, however, will require consideration of the totality of the evidence. In considering the evidence, historical use should not be used as prima facie evidence that the ingredient does not cause harm. It may be appropriate, however, to give considerable weight to a lack of adverse events in large, high-quality, randomized clinical trials or epidemiological studies that are adequately powered and designed to detect adverse effects, including those adverse effects with established serious risks for human morbidity or mortality and that are known to rarely occur de novo in the population.
Information about human use of dietary supplement ingredients may be in the form of formal studies, such as clinical studies or trials and epidemiological studies; in the form of spontaneously reported adverse event reports and literature case reports; or in the form of information about historical use of the ingredient. Because dietary supplements are not required to undergo formal studies before marketing, formal study data available on dietary supplements are less commonly available than adverse event reports or information about historical use.
The different types of data about human use can be useful either as (1) an indicator of possible risk or (2) a mitigator of concerns raised by other data. For example, spontaneous reports are generally used to detect concerns, and historical use information is often presented as a mitigator of concern. Formal studies are less likely to be available, but if they are, they can be the source of information about adverse events in individuals, or they can be used to demonstrate an overall increase in risk of a particular adverse event associated with ingestion of an ingredient. Formal studies can also be used as mitigators of concern if they are adequately designed and powered to detect adverse events.
Within each type of human data, questions can be asked about the nature and quality of the scientific information to determine whether the information raises the level of concern regarding the probability to cause harm. In the sections that follow, the nature of evidence that increases concern is described and illustrated in a spectrum of concern figure. Information that independently raises a higher level of concern requires immediate attention to evaluate the potential of the ingredient to cause harm. For observations categorized as lower to moderate concern based on their placement on the spectrum, it is important to consider whether other types of human, animal, in vitro, or related data, as well as information about potential interactions (see Chapter 8), add to the level of concern. For many dietary supplement ingredients, human data regarding their safety or risk will not be available. A lack of data should not be interpreted as an absence of risk. Other types of data must be examined and weighed appropriately to understand the risk.
SPONTANEOUSLY REPORTED ADVERSE EVENTS
Adverse events associated with product use—whether reported directly to the Food and Drug Administration (FDA), manufacturers, distributors,
spasm requiring intensive treatment in an emergency room or at home, blood dyscrasias or convulsions that do not result in inpatient hospitalization, or the development of drug dependency or drug abuse” (21 C.F.R. § 600.80  and 21 C.F.R. § 314.80 ).
or poison control centers—constitute important sources of safety information, along with published case reports and case series about adverse events. All unsolicited reports from health professionals or consumers received by FDA via either the voluntary or mandatory route are called spontaneous reports, so classified because they are clinical observations that originate outside of a formal study (Faich, 1986). The large-scale regulatory agency safety databases are composed of adverse event information generated by reporting from all sources, including scientific literature case reports and case series; each type of product (e.g., drug, biologic, device, and dietary supplement) is represented.
While different products have aspects unique to their specific type, the principles of postmarketing safety monitoring apply to all; thus, what has been, and what will be, learned from one product realm is generally globally applicable. Dietary supplements, many of which contain biologically active ingredients, are no exception; that the guiding scientific principles for postmarketing safety surveillance have predominantly resulted from experience with pharmaceuticals (drugs and biologics) in no way invalidates their applicability to other substances, such as dietary supplement ingredients.
However, while the underlying principles for safety monitoring are globally valid, the regulatory situation of dietary supplements impacts the use of their associated adverse event reports in several ways. Unlike drugs, biologics, and medical devices, dietary supplements do not undergo premarketing evaluation for safety and efficacy by FDA, and a formal benefit/risk assessment is not performed as part of an approval process. There is no evaluation of product quality (including purity, content uniformity, and stability) prior to marketing, and there is no requirement for manufacturers to collect or report adverse events to FDA. Yet, at the same time, the threshold for concern or action for a dietary supplement is lower than for regulated medical products, as under the applicable law they are considered to be similar to foods.
In this section, the strengths and limitations of using adverse event reports for medical products in general are described, as well as how significant differences in laws and regulations impact the evaluation of adverse event reports associated with dietary supplement use.
Strengths and Limitations
The limitations of spontaneous adverse event reporting systems are well recognized, and include subjectivity and imprecision of adverse event recognition (Karch et al., 1976; Koch-Weser et al., 1977), underreporting (Chen et al., 1994; Chyka and McCommon, 2000; Rawlins, 1995), reporting biases (Sachs and Bortnichak, 1986), lack of precise exposure data (Begaud et al., 1994), and variability in report quality (Goldman, 1998). In
particular, the lack of precise numerator (number of cases) and denominator (number of patients exposed) data render the computation of incidence rates from spontaneous reports problematic (Begaud et al., 1994), if not totally unfeasible. In addition, as spontaneous reports originate under conditions of everyday use rather than under study conditions, there are possible confounding factors to be considered when evaluating reports, such as multiple concomitant medications (prescribed or over-the-counter [OTC]), multiple concomitant dietary supplements, concomitant medical devices, underlying disease states, or alcohol use.
At the same time, these systems entail considerable concomitant strengths. Large-scale and relatively inexpensive (Fletcher, 1991), spontaneous adverse event reporting systems serve as the basis for safety-related hypothesis generation (Strom and Tugwell, 1990) and foster suspicions (Finney, 1971) that generate signals of potential problems warranting further study, while enabling individuals (health professionals and consumers alike) to contribute to public health (Goldman, 1998). This sentinel signaling function is critical, and the appropriateness of using a spontaneous reporting system in this regard is well documented and scientifically accepted (Blum et al., 1994; Goldman, 1996; Rossi and Knapp, 1984).
The cases spontaneously reported to any surveillance program will generally represent only a small percentage of the number that have actually occurred.2 However, if the submitted reports are of high quality, irrespective of number, the effect of underreporting can be somewhat mitigated (Goldman, 1998). In the particular case of dietary supplements, a recent FDA-commissioned study estimated that FDA receives reports on less than 1 percent of all adverse events associated with their use (Walker, 2000). (With the majority of dietary supplement adverse event reporting to FDA done by consumers rather than by health professionals, it is possible that consumers might be less likely to associate dietary supplements with untoward effects, as opposed to making such attribution with a drug product, either prescribed or OTC). In addition, it has been found that consumers often do not inform their physicians about their use of dietary supplements (Eisenberg et al., 1998). Hence, reporting by physicians and other health professionals of adverse events on these products may well be minimal.
Challenges Particular to Dietary Supplements
Scientifically, the use of adverse event reports to assess the safety of dietary supplements should be very similar to how the safety of drug and other medical products are assessed, but the unique regulatory situation of dietary supplements provides some additional challenges. Beyond the limitations inherent in postmarketing surveillance systems, this unique regulatory environment renders the assessment of dietary supplement adverse event reports of greater complexity versus that performed on reports regarding drugs, biologics, or medical devices.
Because premarketing safety studies are not required for dietary supplements, standard drug premarketing data, such as clinical pharmacology studies, are not likely to be available. As a result, significant clinical information such as how the product is absorbed, metabolized, and excreted is generally not available, nor is the product evaluated for possible interactions with foods, drugs, biologics, or devices. The lack of such clinical information generally does not allow specific populations at possible increased risk for adverse effects (such as children, the elderly, or those with renal and/or hepatic dysfunction) to be identified in a systematic manner.
In further important distinction from medical products, there are no current FDA regulations establishing a baseline mandatory standard for dietary supplement manufacturing (CFSAN, 2001). As a result, the situation of multiple manufacturers of a specific dietary supplement can result in significant variation from product to product. An illustrative example is a study of the St. John’s wort products available in Germany, which detailed wide differences among the content of hypericin and hyperforin products, and notable interbatch variability in some of the products (Wurglics et al., 2001). Thus, unlike the situation with prescription drugs, in which there is standardized quality control of the innovator product and generic versions in manufacturing, adverse event reports on a particular dietary supplement may entail several different products that can vary significantly in dietary ingredient concentration, both among and within individual products.
FDA’s proposed rule to establish current good manufacturing practices for manufacturers, with respect to both the production and labeling of dietary supplements (FDA, 2003a), addresses some manufacturing issues, but not all aspects of product-to-product variability. According to FDA, this proposed rule “would, for the first time, establish standards to ensure that dietary supplements and dietary ingredients are not adulterated with contaminants or impurities, and are labeled to accurately to reflect the active ingredients and other ingredients in the product” (FDA, 2003b).
The challenge of underreporting adverse events was described above as a general challenge in using spontaneous adverse event reports. A factor particular to dietary supplements, however, is that, unlike drugs, biologics,
and medical devices, dietary supplement manufacturers and distributors are not required to disclose to FDA the adverse event reports that they receive (CFSAN, 2001; OIG, 2001). As a result, an important established source of reports for medical products is not duplicated for dietary supplements, and the evolution of a concomitant culture of adverse event reporting among manufacturers and distributors is not encouraged.
The lack of manufacturer disclosure requirements is especially important given that consumers may be less likely to report adverse reactions to practitioners, a usual source of adverse event reports. A study sought to determine whether botanical remedy users would report adverse reactions to such products differently from similar adverse reactions experienced with the use of OTC medications (Barnes et al., 1998). While approximately 30 percent would consult their general practitioner irrespective of which type of ingested product was being used, and another 43 percent would not consult in either case, 26 percent would consult their general practitioner for a serious OTC-associated adverse reaction, but not for a similar adverse reaction associated with use of an herbal remedy.3
In summary, adverse event report assessment of dietary supplement ingredients is of heightened complexity and ambiguity compared with that of medical products.
Using Spontaneous Reports
Assessing the Strength of Association Between Event and Product
Spontaneous reports entail an assumed association between the denoted adverse event and product in question, but careful evaluation of accumulated cases is needed to assess the actual strength of the association. Achieving certain proof of causality is not necessary to determine that an unreasonable or significant risk exists, especially if other types of data support the same conclusion.
Regarding numbers of cases needed for such assessment, when the medical product-adverse event relationship is stronger and the incidence of the adverse event occurring de novo/naturalistically is lower (i.e., the event
is rarer), fewer case reports are needed to perceive causality (Auriche and Loupi, 1993). For rare4 serious adverse events, such as toxic epidermal necrolysis, coincidental medical product-event associations have been found to be so unlikely that they merit little concern in spontaneous reporting, with more than three reports seen to represent a signal necessitating further study (Begaud et al., 1994). Further, it has been suggested that the combination of a temporal relationship between medical product and adverse event, positive dechallenge, and rechallenge can make individual reports conclusive as to product-event association (Temple et al., 1979).
Notwithstanding, there is no definitive number of cases for generating a signal of safety concern. It is dependent on the characteristics of the individual adverse event itself, including such factors as clinical manifestations and severity, potential for significant morbidity (reversible and irreversible) or mortality, potential populations at risk, and the overall risk to public health.
In judging any association to be causal, biological plausibility and reasonable strength of association are useful (Rawlins, 1995). However, given the outlined limitations of spontaneous reports, achieving certain proof of causality through postmarketing surveillance is unusual (Auriche and Loupi, 1993). Attaining a prominent degree of suspicion is much more likely, and, given that it may be deemed an ample basis for regulatory decisions on drugs (Auriche and Loupi, 1993), it is an ample basis for regulatory decisions on dietary supplement ingredients.
Regarding factors that can be useful in assessing the strength of association between any medical product and a reported adverse event, international consensus produced the following list (CIOMS, 1990):
The chronology of administration of agent, including beginning and ending of treatment and adverse event onset,
The course of the adverse event when the suspected agent stopped (dechallenge) or continued,
The etiologic roles of agents and diseases in regard to an adverse event,
Response to readministration (rechallenge) of the agent, and
Laboratory test results.
Considering the Nature of the Adverse Event
Beyond the factors listed above, the nature of the adverse event itself is an important consideration in assessing spontaneous reports. The nature of the adverse event encompasses knowledge about its naturalistic frequency of occurrence and its potential for significant morbidity or mortality.
As previously discussed, premarket clinical trials in humans have inherent limitations that significantly affect their ability to detect adverse events (Goldman et al., 1995); to have a 95 percent chance of detecting an adverse event that occurs in 1 in 1,000 people, 3,000 people must be exposed (Lewis, 1981). For adverse events that occur de novo even more rarely, such as 1 in 100,000, 300,000 people must be exposed for there to be a good chance of detection. Many of the most serious adverse events occur relatively infrequently; that is why spontaneous report systems, which are designed to cover entire populations, are able to detect rare, serious events not discovered during premarket testing.
This important outcome of spontaneous report systems for dietary supplement ingredients is the same as for other products. There are serious adverse events that by their very nature necessitate increased attention and scrutiny due to their potential for significant morbidity and/or mortality. FDA, in its recently published Proposed Rule for Safety Reporting Requirements for Human Drug and Biological Products, specifically addressed this issue by proposing a new designation, “Always Expedited Reports” for specific suspected adverse drug reactions (SADRs) of medical significance (FDA, 2003c). The following SADRs,5 regardless of expectedness, would be subject to expedited reporting due to their very nature: “congenital anomalies, acute respiratory failure, ventricular fibrillation, torsades de pointe, malignant hypertension, seizure, agranulocytosis, aplastic anemia, toxic epidermal necrolysis, liver necrosis, acute liver failure, anaphylaxis, acute renal failure, sclerosing syndromes, pulmonary hypertension, pulmonary fibrosis, confirmed or suspected transmission of an infectious agent by a marketed drug or biological product, confirmed or suspected endotoxin shock” (FDA, 2003c). Reporting of any of these adverse events in association with dietary supplement use should trigger the same degree of concern as for any other product, and the same heightened need for a timely assessment.
Considering the Amount Ingested
Some reports of adverse events are based on intakes that exceed the amount specified on the particular dietary supplement label for “intended use.” A number of factors make it difficult, if not impossible, to determine the contribution of dose to the effects reported. The lack of adequate premarketing data to establish the validity of labeled dosing for many dietary supplements is coupled with significant variability in the amount of the particular dietary supplement constituents in different preparations. Known pharmacokinetic variability among individuals also makes it difficult, if not impossible, to determine the contribution of dose to the effects reported. However, in conclusion, if an adverse event report provides credible evidence that a serious adverse event is associated with a dietary supplement ingredient (see criteria in the earlier section), its utility in establishing a level of concern should not be discounted simply because the intake (resulting in adverse events) exceeds that specified in current dietary supplement labeling.
Summary of Spontaneous Report Use
In summary, higher concern is warranted in situations where one or more well-documented serious adverse events manifests positive temporality, and other factors (e.g., positive dechallenge, biological plausibility, or laboratory results) combine to strengthen the perceived association between the dietary supplement ingested and the adverse event in question. Given the inherent limitations of spontaneous reports (including report quality) in general, and those concerning dietary supplement ingredients in particular, not all of this information will be available in many cases.
Tables 4-1 and 4-2 illustrate the relative spectra of concern. The columns on the right describe situations that warrant higher concern because the greatest risk to public health exists, while situations described on the left are of lower concern. The level of concern increases in proportion to the completeness of information provided and the likelihood of confounding decreasing. However, with serious, unexpected adverse events, possible confounding in the associated reports should not automatically lessen the level of concern, but rather heighten the attempt to obtain more reports of the highest possible quality to maximize the signaling function of spontaneous report systems. In addition, the nature of the serious adverse event itself and its potential for significant harm should stimulate appropriate heightened concern.
The great utility of spontaneous reports lies in the generation of hypotheses about relationships between supplement ingredients and untoward effects, thus highlighting potential problems and signals that the agency may want to explore in greater depth. Evaluation of signals from
TABLE 4-1 Relative Spectrum of Concern for Individual Spontaneous Adverse Event Reports
Describes a serious adverse event with less information than would justify moderate or strong concern, and/or with prominent confounding factors (e.g., multiple concomitant substances and/or conditions)
Describes a serious adverse event with some, but not all, characteristics associated with strong concern
Describes a well-documented serious adverse event with plasma levels (if available) at a relevant range and demonstrates dechallenge and rechallenge (if possible), temporality, and strong attribution
TABLE 4-2 Relative Spectrum of Concern for Case Series of Spontaneous Adverse Event Reports
Describes a series of serious adverse events, with less information than would justify moderate or strong concern, and/or prominent confounding factors (e.g., multiple concomitant substances and/or conditions)
Describes a series of serious adverse events, with some, but not all, characteristics associated with strong concern
Describes a series of well-documented cases demonstrating consistent serious adverse events and clinical findings, and dechallenge (if possible), temporality, and strong attribution
spontaneous reports should entail use of supplementary information available about the substance, such as animal data, in vitro data, epidemiological studies, or clinical trials in which formal hypothesis testing can occur. Such studies can be used to further evaluate the strength of the association between the adverse event and dietary supplement in question because, as described in the previous paragraphs, establishing a definitive causal relationship solely through use of spontaneous reports is rarely possible. However, as noted, regulatory decisions that directly involve the safety of the public’s health do not necessitate definitive proof.
In the specific case of dietary supplements, the threshold for action is clearly stated in the Dietary Supplement and Health Education Act: demonstration of a “significant or unreasonable risk of illness or injury.” Given the state of the art in adverse event report evaluation, pharmaco-
vigilance, and risk management, and given the potential risk to the public entailed by serious adverse events, regulatory action can be justified on the basis of adverse event report analysis alone or as the predominant source of information. There is ample precedent for this approach from the realms of regulated medical products, such as drugs, biologics, and medical devices. Dietary supplements, as agents with biological activity, should be no exception.
Experience from generations of use by humans is often referred to as evidence of safety for modern dietary supplements that bear resemblance to substances used historically. Some botanicals, for example, have a long history of medicinal use in many cultures. It may be useful to consider that there is both ancient (thousands of years) and recent history (perhaps the last 100 years). Ancient history may include traditional Chinese, Ayurvedic, and Native American medicines. Information about the preparation and use of ancient remedies is more difficult to locate and verify.
Information about historical use is of less importance when relevant clinical, epidemiological, or animal toxicity data exist because if these types of data document harm, then this outweighs historical use that may show no harm. However, for many dietary supplement ingredients, the amount of scientific and experimental data are insufficient for a critical analysis of safety. Recognizing that a full range of data are unlikely to be available for many dietary supplement ingredients, historical use may be taken into account as a surrogate measure for safety in the absence of relevant scientific and experimental data. In doing so, it is important to consider the relevance of the traditional use to the current use and, as such, FDA must have information regarding both the traditional use and the current use to determine if the traditional use sheds any light on the potential risk associated with current use.
Identifying Historical Uses of a Related Substance
A starting place for obtaining information regarding traditional use of dietary supplement ingredients is secondary references. If secondary references suggest that the traditional use is similar to the current use and that the historical use has been without observed complications, then it is important to verify this information with primary sources before relying on it as even a weak surrogate indicator of safety. The quality of the original source of information should be considered before placing much value in secondary sources, as questionable information appears to be cited repeatedly, with problems in obtaining original information.
If the supplement ingredient is a botanical or medicinal plant, it may be of value to understand the medical system of use from which the information is derived. This information is found in a number of books and symposium reports (e.g., Bannerman et al., 1983; Chadwick and Marsh 1990; Dobelis, 1997; Griggs, 1997; Prance et al., 1994). Traditional or folkloric uses are extensively described for more than 15,000 species of flowering plants in NAPRALERT, as are in vitro data, in vivo data, and reports of human use of extracts of flowering plants (Farnsworth, 2003). Information regarding current conditions of use, such as duration and amount ingested, can be obtained from third-party literature, labeling, marketing data, and survey data.
Considering the Relevance of Historical Use
The discussion in this section focuses on questions to consider when assessing whether the information about traditional use is relevant to current use conditions. These questions are listed in Box 4-1 and are explained in more detail in the following paragraphs.
Is the supplement ingredient one that was commonly used within the context of a traditional medical system? Moreover, if there are traditional cautions in the use of the supplement ingredient, are these cautions typically heeded? Some dietary supplement ingredients, including some botanicals, were traditionally prescribed by practitioners knowledgeable about contraindications to their use. It is scientifically appropriate to take contraindications in traditional use into account when considering the safety of the ingredient. If, for example, an ingredient traditionally contraindicated for pregnant women is currently being marketed to pregnant women or is frequently consumed by pregnant women due to its expected effects, then FDA should be more concerned about the safety of this ingredient.
How “safe” is the historical use? Were the adverse events in question capable of being detected by the practitioners and, if observed, would they have been recorded? The fact that a substance was consumed over a number of years does not indicate that it was consumed without adverse effects. This is especially true for effects that are not acutely apparent, but even acute adverse effects may have been tolerated with the medicinal use because better treatments options did not exist. In understanding the relevance of historical use to safe current use, it is thus helpful to have information documenting safe historical use.
Was the substance traditionally ingested? Safe administration by non-oral routes of administration should not be taken as an indication of safety via oral administration.
If the supplement ingredient is a botanical, is the part of the plant marketed the same as the plant part that was traditionally used? Historical information is only useful if the product in question is not so far removed from the original substance as to constitute a distinct entity. For example, a whole root extract that was traditionally used for three days to treat a cold is not comparable with a fraction of a leaf extract promoted for long-term use to treat cancer. Safety comparisons for botanicals can only be made when the same plant part used in traditional preparations is used in the modern preparation because seeds, roots, leaves, and other parts may have distinct safety profiles due to difference in composition, as is evidenced by the differential distribution of toxins in some uneaten parts of common food plants. In summary, indication of safe use of one plant part does not indicate that other plant parts might also be used safely.
Are current intake levels or recommended intake levels clearly different from traditional use? A frequently quoted axiom of toxicology from Paracelsus is that “only the dose makes the poison.” Unfortunately, differences in traditional and modern formulations render dose comparisons between traditional and modern formations difficult or even impossible. It is a rare case when the levels of potentially dangerous bioactive compounds in traditionally used formulations have been quantified and can be com-
pared with modern formulations. In most cases, however, dosing comparisons are so imprecise that they should probably only be attempted when the modern formulation clearly provides doses that may be orders of magnitude higher than traditional doses. For example, consumption of a culinary botanical in small amounts is very different, and thus may have different effects, than consumption of large amounts of the same encapsulated botanical, rendering a safety extrapolation from culinary to supplemental use inappropriate. In summary, if the current level of intake is significantly above what has been traditionally recommended, then the level of concern should be increased.
How similar is the current preparation to that used traditionally? Is the preparation a crude preparation, extract, or concentrate; a selected fraction; an isolated compound; or a mixture of these? As discussed above, the toxicity of a substance depends on the amount ingested. The method of preparation will impact the amount and types of chemical compounds ingested, thus potentially impacting an ingredient’s safety. The different methods of preparation are most clearly illustrated with botanicals. Traditionally, many orally ingested medicinal botanicals were administered as crude aqueous extractions of plant parts that were soaked, steeped, or boiled in water. Today’s supplement ingredients are often sold in a different form—as encapsulated dried botanicals, fluid extracts, solid extracts (e.g., capsules or tablets), or foodstuffs containing botanical extracts. The same plant can be used as an extract prepared from dried plant materials (an infusion) or as lyophilized plant made from whole fresh materials.
Whether a botanical with a history of benign use in infusions (teas) manifests new toxic effects when concentrated, lyophilized, or encapsulated will depend on where any toxic components are localized in the plant, their water solubility, their potency, and the likelihood that a person could consume enough of the active ingredients to cause an ill effect. Differences in safety profiles could also be expected for alcoholic versus aqueous extracts of plants with known toxic components. Alcohol and water extract different compounds, so alcohol extracts may contain a higher concentration of toxic compounds than aqueous extracts. Wormwood (Artemisia absinthium), for example, in an aqueous extract contains little thujone (a neurotoxin) (Tegtmeier and Harnischfeger, 1994), but may contain substantial amounts of thujone in alcohol extracts. In summary, if the method of preparation concentrates the bioactive compounds to a degree not known to be consistent with safe historical use, the level of concern should be raised.
In addition to preparations that might result in increased concentration of bioactives in the products ingested, it is also important to note that modern formulations may simply make the same substances more likely to be ingested in excessive amounts, which should raise concern. If a substance
is compacted in a capsule where taste and sheer volume of the material does not limit consumption, then there is a greater likelihood of an adverse reaction compared with the same botanical that was traditionally ingested in smaller amounts.
Is the modern duration of use consistent with historical use patterns? The duration of use needs to be considered because acute, short-term, and long-term intakes all have different safety implications. A lack of adverse events reported for a botanical traditionally used only for a few days has little or no relevance to safety of the same botanical when it is chronically ingested. When considering how the current duration of use compares with traditional duration of use, it may be helpful to also consider whether the modern day indication is consistent with traditional indications. The modern uses of some botanicals, especially for nonmedical indications such as memory enhancement and ergogenics, may lead consumers to chronically use dietary supplements that were never used chronically in traditional medicine. In summary, concern is increased if the substance is now used for longer duration than it was traditionally.
Is the modern reason for using the substance consistent with historical indication for its use? The reason for using the substance does not in itself provide information about its safety as a dietary supplement, but comparing the modern and traditional indications may provide clues for comparing historical and modern use. For example, some indications are more consistent with external use versus ingestion. Similarly, some indications are more consistent with long-term use (e.g., to lose weight) compared with short-term use (e.g., to treat an asthma attack).
Is the target population similar to that which used the substance historically? Is the current user population similar to that which used the substance historically? People vary in their response to bioactive compounds. Due to physiological condition or other reasons, particular subpopulations may be more likely to suffer a serious adverse reaction than other groups. Thus, the modern use of a dietary supplement ingredient by populations that have not traditionally consumed the ingredients reduces the relevance of safe historical use information. A change in usage does not necessarily in and of itself raise the level of concern; however, if the supplement ingredient is now used by a subpopulation that may be more susceptible to adverse effects, concern may be warranted.
Summary of Historical Data Use
The fact that a botanical or other dietary supplement ingredient has been used for centuries is not prima facie evidence that it is safe. If the current preparation concentrates constituents, if the current use is more frequent or of longer duration, if the substance was not historically in-
TABLE 4-3 Relative Spectrum of Concern Raised by Historical Evidence of Toxicity
Traditional cautions (contraindications) exist regarding use in certain populations or circumstances
Traditional cautions (contraindications) exist regarding use in certain populations or circumstances that, if ignored, might be associated with a serious adverse effect (e.g., do not use during pregnancy)
There is clear evidence that traditional use causes conditions considered to be serious adverse events (e.g., hallucination, lethal poisoning)
gested, if a different plant part is now used, if the ingredient is formulated or processed differently, or if a different population is using the substance, then the level of concern should be raised. It is clear from these questions that historical use of a substance, even widespread historical use without documented ill effects, is no guarantor of long-term safety.
Historical use information is very useful when it describes a relationship between untoward effects and an ingested substance, as illustrated in Table 4-3. It is less useful in predicting safe use, especially if there are other reasons to be concerned about the possibility of effects that do not occur immediately following exposure. However, in the absence of other data that raise concerns about the safety of the substance, information about safe historical use may provide indirect evidence for lack of serious acute harmful effects if its relevance to current use conditions is carefully considered.
Information about the historical use of an ingredient may be most useful if it suggests potential adverse effects that could be anticipated. It is also helpful to compare relevant historical use information with other types of information that suggest possible harm (e.g., in vitro data, animal data, other human data, or data about related substances). While the historical use information should not be considered as more important than the scientific evidence, it may be appropriate to take information about the history of use into account if years of previous use would be expected to uncover the adverse effect under consideration. In such cases, historical use information may mitigate concerns to some degree.
Clinical studies evaluate the efficacy and/or safety of health care interventions in humans. There are several types of clinical studies, which differ
as to whether the study includes a control group, the methods used to assemble the comparison groups, the extent of blinding (if any) of investigators and subjects, and other measures taken to minimize biases.
Although the double-blinded, randomized controlled trial (RCT) is often considered the “gold standard” for evaluating efficacy of many health care interventions, such studies have a much smaller role regarding safety concerns because they are not ordinarily primarily designed for that purpose. While adverse events are required to be monitored, collected, and evaluated during the course of an RCT, their known limitations (i.e., relatively small sample size, relatively short duration, narrowness of population studied, and narrowness of indication studied) (Goldman et al., 1995) make it almost impossible for a serious adverse effect that occurs relatively infrequently to be detected during the course of such a study. There is no scientific reason to think that efficacy studies of dietary supplements would be any exception. Given their limitations and the highly controlled settings in which most randomized trials are conducted, they are inadequate to fully assess the potential for harm of an intervention when it is routinely used in the target population. It is impossible to study all interactions of an intervention with combinations of comorbidities and concurrent medications or dietary supplements that may be present in the real world using a limited number of randomized trials. In addition, because dietary supplements are considered similarly to food, even if randomized controlled trials were performed to assess their benefits, there may be fewer perceived concerns for their safety and therefore a reporting bias on the part of the subjects and/or the investigators.
If it existed, an RCT designed to evaluate safety of a dietary supplement would include a sufficiently large number of diverse subjects who were systematically monitored for a sufficient amount of time to detect a wide array of adverse effects or physiological changes that might warrant concern. The physiological parameters focused upon in monitoring human subjects would be determined, in part, by effects found in preclinical (animal) studies. Extensive preclinical studies, however, are not often completed for dietary supplements.
It is the usual practice in an RCT to query subjects for possible adverse events at defined intervals and to record and evaluate these events as “definitely,” “probably,” “possibly,” or “not” related to the ingested substance (ICH, 1995). Randomization and use of control groups enable investigators to determine the likelihood that adverse effects are actually due to the substance rather than to confounding factors. However, as previously noted, many RCTs available for dietary supplements are designed to assess beneficial effects and thus would not be expected to provide complete information relative to the safety of the dietary supplement under evaluation. In health care intervention studies, perhaps due to the greater tendency for
authors to report positive findings and perhaps due to limited amount of space in journal articles, efficacy results are more consistently reported and are reported in greater detail than safety data (Ioannidis and Lau, 2001). Although investigators conducting efficacy trials are expected to observe and report adverse reactions, the extent and detail of this reporting is highly variable (Ioannidis and Lau, 2001). In some cases, however, investigators may be able to supply unpublished data useful in the safety evaluation, as the published results may not contain all the available information about adverse events (Ioannidis et al., 2002).
While investigators may be able to provide additional unpublished data, characteristics of the study design itself may limit usefulness in predicting safety because even large studies may lack sufficient statistical power to detect adverse events of low incidence. Clinical trials generally are designed to detect one primary endpoint; thus secondary events, such as adverse effects, will frequently be inadequately reported (Ioannidis and Lau, 2001). A major cause of an inadequate safety evaluation is that an unexpected adverse event may not be noticed by the subject or detected by the investigator. For these reasons, a study to test the effects of a dietary supplement ingredient on mood, for example, may not detect potentially dangerous cardiovascular effects of the supplement if heart function is not monitored.
These known limitations of RCTs regarding safety limit their sensitivity to be able to detect adverse events that occur infrequently, only after extended exposure, or predominantly in subpopulations (Goldman et al., 1995). For example, events that occur at the rate of 1 in 1,000 would require a study with at least 3,000 subjects at risk to have a 95 percent chance of being detected (Lewis, 1981). Differences between the study and the target population and administration of the substance during the RCT compared with its actual use by the general population, coupled with the inability to exhaustively evaluate for all possible interactions with drugs and other dietary supplements in RCTs, limits the generalizability of results from these types of studies.
However, while clinical trials can be limited in their sensitivity, they do provide valuable information when adverse events are detected. Utility of information from clinical studies is strengthened by providing the following (Counsell, 1997; ICH, 1995; Moher et al., 2001):
Demographic information on the study population,
Inclusion and exclusion criteria to determine whether the results are generalizable,
A description of the condition or disease and comorbidities of the study population,
A description of the intervention (supplement ingredient [composition], dose, and duration of exposure),
A list of prior and concomitant ingested substances, including dietary supplements and drugs, and
A description of the adverse event, including temporal relationship to ingestion of supplement ingredient (i.e., response to dechallenge and rechallenge, where appropriate).
Clearly, RCTs cannot be relied upon as the sole source of information to assess the safety of a dietary supplement. Because there are no regulatory requirements to demonstrate the efficacy of dietary supplements and because studies are lengthy and expensive, it is unlikely that many well-conducted RCTs for dietary supplements will be available. Even if they do exist, the limitations discussed earlier in this section make RCTs inappropriate for detecting rare adverse events. Epidemiological studies and spontaneously reported adverse events are better suited in general to provide this important information, as are other types of nonhuman data, as described in the next chapter.
While RCTs may not reveal the entire range of possible adverse events and the occurrence of adverse events may not be adequately reported, on occurrence adverse events are reported in this type of study. Although uncommon, some RCTs have been specifically designed to assess adverse events as the primary outcome. Because an RCT uses a randomized comparison group to minimize confounders and biases in the assessment of outcomes, statistical differences in adverse events in the treatment group warrant a good deal of attention.
A statistically significant increased rate in adverse events indicates that a sufficiently large number of events have occurred to allow one to conclude that the observation is unlikely due to chance alone. The relative concern appropriate for the different types of information typical of a clinical study is described in Table 4-4. In general, the highest concern (the right column) is raised when there is a statistically significant higher rate of a serious adverse event or serious abnormalities in clinical laboratory or other diagnostic test values in the dietary supplement group when compared with the control population. Examples of serious abnormalities in clinical laboratory or other diagnostic test values might include, for example, aberrant electrocardiography findings or electrolyte changes that indicate a very high risk of serious cardiovascular or neurological consequences.
Even if an adverse event is truly associated with the use of a dietary supplement ingredient, sometimes only a nonstatistically significant trend toward increased serious adverse events is observed in RCTs. This situation
TABLE 4-4 Spectrum of Relative Concerns with Clinical Studies Data
Describes a serious adverse event, but with less information than would justify moderate or strong concern, and/or the interpretation of the clinical study is hampered by the presence of prominent confounding factors (e.g., multiple concomitant substances and/or conditions) that could not be controlled by balancing
Prominent methodological concerns (e.g., unexplained high level of dropouts, lack of control groups)
Nonsignificant, but clinically important, trend of a higher rate of a serious adverse event
Abnormalities in clinical laboratory values
Other abnormalities, such as electrocardiographic findings in the dietary supplement ingredient group
A significantly higher incidence of a serious adverse event
Other potentially dangerous abnormalities, such as in clinical laboratory values that are associated with risk of serious adverse events
Other abnormalities, such as electrocardiographic findings in the dietary supplement ingredient group
may occur if the study is small, too few adverse events were observed, or the study is otherwise underpowered. For example, the appearance of liver enzyme abnormalities in two or three subjects taking a dietary supplement may not produce a statistically significant difference compared with the control group. Therefore, it cannot be concluded with confidence that the liver abnormality is due to the dietary supplement, but the information can be used to augment other data related to safety of the dietary supplement ingredient. However, in populations and settings where no adverse event is expected, a nonsignificant trend also warrants attention.
Finally, a concern still exists even if there is not a trend that would justify a moderate or statistically significant concern. This often occurs when a firm conclusion cannot be reached due to confounding factors. A frequent confounding factor with dietary supplements, for example, is the concomitant consumption of other xenobiotics. Single cases of serious adverse events, such as death or liver failure in the dietary supplement group, also warrant special attention.
Value of Epidemiological Studies
Epidemiological studies that contain information on the use of dietary supplements, when available, are valuable sources of information for evaluating their safety. Data from these studies complement information from RCTs and adverse events reports. This type of data is rare; however, this is likely to change as additional research is conducted on dietary supplement use, efficacy, and safety. The considerations described below will be helpful in considering epidemiological studies to assess unreasonable or significant risk.
As discussed above, among the limitations inherent to many RCTs is that relatively small size and short duration limit sensitivity to detect adverse events (Goldman et al., 1995). Latent or delayed effects that occur long after exposure may not be detected. Information about these latent and infrequent effects often comes from epidemiological studies that retrospectively or prospectively examine the effects of ingested substances on large populations. Another benefit of epidemiological studies is that the number of individuals exposed to supplements is expanded to the general population. Data from these studies usually contain a large number of individuals compared with the number exposed in clinical trials.
Like RCTs, the value of epidemiological studies also depends on the endpoints examined. For example, if a study evaluates the incidence of cancer, death, or liver damage but does not evaluate anemia, the study is unlikely to detect interference with iron absorption.
The assessment of the level of concern regarding the safety of a dietary supplement ingredient is dependent on the quality of the study and analyses, the estimated risk or odds ratio, the clinical significance of the risk, and the statistical significance of the estimate. Situations in the right column of Table 4-5 indicate higher levels of concern.
TABLE 4-5 Spectrum of Relative Concerns with Epidemiological Data
Case-control or cohort study (including registries), with small,a but statistically significant, relative risk or odds ratio of a serious adverse event
Large relative risk or odds ratio of a serious adverse event that is not statistically significant
Poorly conducted studies with large or significant effects
Case-control or cohort study (including registries), with moderate, statistically significant relative risk or odds ratio of a serious adverse event
Moderate relative risk or odds ratio of a serious adverse event that is not statistically significant but that implies a trend
Well-conducted case-control or cohort study (including registries), with large, statistically significant relative risk or odds ratio of a serious adverse event
a Two or less is generally considered a weak association, and 3 or more is considered strong, but this is only a very general “rule of thumb” guidance and somewhat debatable.
Using Epidemiological Data on Dietary Supplements
Assessing causality from epidemiological data requires the specific study designs that are described here, for example, case-control and cohort studies. However, other types of epidemiological studies often have been used in combination with other study designs (e.g., RCTs, case reports) to draw or strengthen conclusions (GAO, 1992).
For any type of epidemiological study, the quality of the study and the analyses depends on the quality of the data. Missing data or, in the case of surveys, poor participation may cause biased results. Quality also may be affected by a conflict of interest by an author or study sponsor. Safeguards should be in place to prevent biased reporting of study results or where a conflict of interest is present. Errors in design, data collection, and analyses can also lead to poor-quality studies. Possible flaws of epidemiological studies have been well described in the medical and health care literature (Altman, 1998; Gardner et al., 1986).
Cohort studies generally evaluate a group of individuals (either prospectively or retrospectively) and estimate incidence rates of an event in
exposed and unexposed individuals. From this information, the risk of the event occurring in the exposed group relative to the risk for the unexposed group (the relative risk) can be estimated. Unlike well-conducted RCTs, results from cohort studies can be influenced by selection bias and confounding—if they exist. Selection bias occurs when there are unmeasured factors that are related to the outcome of safety and also affect selection for use or non-use of a supplement. Confounding can occur when use of a dietary supplement is strongly correlated to other characteristics of individuals that also affect the safety outcome. These two potential study design problems should be considered when adjusting the level of concern warranted by cohort studies.
Primarily due to their relative expense, greater need for dedicated personnel, and inability to detect rare, serious adverse events, cohort studies are likely to be used less frequently than case-control studies in a postmarketing environment. Expense of cohort studies decreases when computerized medical records can be used, and thus as more records include information about intake of dietary supplement ingredients, the availability of cohort studies for dietary supplements is likely to increase. Nonetheless, for identified rare, serious adverse events, case-control studies may be more common.
Case-Control Epidemiological Studies
Case-control epidemiological studies are uniquely useful at estimating the likelihood that an ingested substance causes an adverse event when the occurrence of the event is rare or occurs following a long latency period. In a case-control study, cases (persons with the event of interest) and controls (persons who do not have the event of interest) are identified. The exposure rates among cases and among controls are then estimated.7 Using the estimated exposure (dose is rarely known) in the controls and in the cases, the odds of the event in the exposed group relative to that in the unexposed group (the odds ratio) is estimated. (The odds of an event are equal to the probability of the event divided by one minus this probability).
Case-control studies require information on fewer individuals than cohort studies (see next section). However, in case-control studies there is the potential for bias caused by inappropriate selection of the control group or
inaccurate assignment of exposure status. Most textbooks on epidemiology cover this topic in detail (e.g., Rothman and Greenland, 1998).
If exposure to a dietary supplement is widespread, case-control studies could be useful for assessing the association of the supplement with an adverse event—even if the adverse event is rare. However, a case-control study also will be useful when assessing risk from a dietary supplement even when use is not widespread if the adverse event rate among users is high. Both situations—wide exposure and a rare outcome and limited exposure with a common outcome—make case-control studies a useful tool for assessing safety of dietary supplements.
Clinical Significance of Estimated Relative Risk or Odds Ratios from Cohort and Case-Control Studies: Using the Statistics Values
Relative risk is a measure of the association between the exposure to some factor (in this case, a dietary supplement ingredient) and the risk of some outcome (e.g., a serious adverse event). It is calculated as the incidence rate of a serious adverse event among persons taking the dietary supplement ingredient divided by the incidence rate of the serious adverse event among persons not taking the ingredient. An incidence rate is the ratio of the number of events (e.g., of a serious adverse event) over a period of time and the number in the population being studied during the time period. For example, relative risk of 2.5 means that the group exposed to the ingredient is 2.5 times (or 150 percent) more likely to have a particular serious adverse event than those not exposed to the ingredient. A relative risk of 1.0 shows no additional risk in the exposed group while a relative risk < 1.0 indicates less risk in the exposed group. For example, a relative risk of 0.67 means the exposed group has 0.67 times the risk (two-thirds the risk or 33 percent less risk) of the event than does the unexposed group.
An odds ratio is approximately equal to the relative risk when the probability of the adverse event is small in both the exposed and unexposed groups. Therefore, odds ratios often are described and interpreted as if they were relative risk values.
If case-control or cohort studies on a dietary supplement ingredient have been completed, the reviewer should consider the magnitude of the odds ratio or relative risk values, p values, or confidence intervals (see below), and the seriousness and severity of the adverse event in question when determining the relative concern about the safety of a dietary supplement ingredient. In general, the reviewer should be alert to relative risks or odds ratios of greater than 2, as described in Table 4-5. That said, a numerical cutoff is not appropriate and greater sensitivity to low values may be appropriate as the seriousness of the adverse event and the number of individuals exposed increases. As a “rule of thumb,” a relative risk or odds
ratio of 3 or more generally represents a strong association, although choice of this cut-point is debatable (Stolley, 1990; Temple, 1999). Similarly, as a rule of thumb, a relative risk or odds ratio of 2 or less generally represents a weak association (Temple, 1999).
In addition to the relative risk or odds ratio values, a measurement of uncertainty (e.g., standard error) is needed to properly assess the level of concern that should be attached to a finding. Confidence intervals and p values use both the estimated relative risk or odds ratio and the standard error of these estimates in their calculations. In most medical, biological, and health services literature, a p value (the probability of the observed data if the null hypothesis is true) of 0.05 or less is considered statistically significant. This cut-point translates into incorrectly rejecting the null hypothesis 1 time or fewer out of 20, on average. Although a p value of ≤ 0.05 is commonly used to determine when a result warrants attention, in interpreting studies regarding safety, a cutoff point of p < 0.05 is often not appropriate because of its implications. Dietary supplements are regulated similarly to foods and are presumed to be safe (the null hypothesis). A p value under this null hypothesis reflects the probability of the observed data when assuming that the dietary supplement is safe. A p value of 0.05 means that on average, one would incorrectly reject the assumption that a supplement is safe 1 time out of 20. For serious adverse events, when there is a high prevalence of use or when the supplement is used in special populations, a p value greater that 0.05 might raise the level of concern substantially. A p value of 0.10 would mean that the probability of the observed number of adverse events is 0.10 if one assumes that the supplement is safe (i.e., on average one expects to see this number of adverse events 1 time out of 10 if the supplement is safe). This type of finding could be enough to raise the concern level. Knowledge that at this level one incorrectly rejects the hypothesis of safety 1 out of 10 times on average should enable this information to be appropriately integrated with other types of information (e.g., animal data).
Finally, confidence intervals of relative risks or odds ratios may be more useful than p values in interpreting results. A 95 percent confidence interval typically is used and means that 95 percent of the calculated confidence intervals are expected to contain the true relative risk if the estimation were repeated a large number of times in similar study settings. However, as with p values, the choice of the value 95 percent should be used with full understanding of its consequences and meaning. Studies where a 95 percent confidence interval for a relative risk or an odds ratio covers 1 (thus indicating lower significance statistically) may still offer important information about safety. A recent example where it was decided that there existed a high safety concern even without having statistical significance is
the hormone replacement therapy randomized control trial designed to look at benefits of the therapy over time (Rossouw et al., 2002).
Assessing the Strength of Association with Epidemiological Studies
Epidemiologists tend to agree on characteristics of epidemiological studies that suggest an increasingly strong association between an adverse effect and an ingested substance, describing this in terms of establishing “causality” between a substance and an effect (Hennekens et al., 1987; Hill, 1971; Rothman and Greenland, 1998; Sackett et al., 1991). While these characteristics can certainly be used to demonstrate whether the threshold of causality has been met, meeting such a threshold is less important for dietary supplements for which it is only necessary to determine whether an unreasonable or significant risk exists.
For a dietary supplement ingredient studied in epidemiological studies, concern increases as more of the following criteria are met: large relative risk; consistency of findings in different studies or in different populations; association that “makes sense” because other plausible causes are ruled out and results are consistent with current knowledge of cause and effect in humans, animals, and cells in vitro; association that is limited to a single potential cause and a single type of adverse event;8 a dose-response relationship9 and temporality (i.e., the adverse event occurs after a dietary supplement is ingested). In the evaluation of safety of dietary supplements, one high-quality epidemiological study alone can cause a high level of concern, as shown on the right side of Table 4-5. Epidemiological studies that do not meet these criteria may be used to form hypotheses about safety or to strengthen or generalize the results from RCTs or other data.
As described in the guiding principles, a credible report or study finding of a serious adverse event in humans that is associated with use of a dietary supplement ingredient raises concern about the ingredient’s safety. While historical use should not be used as prima facie evidence that the ingredient does not cause harm, it may be appropriate to give considerable weight to a lack of adverse events in large, high-quality, randomized clinical trials or
epidemiological studies that are adequately powered and designed to detect adverse effects.
The basis for this guiding principle is described in the sections of this chapter, as are the following important corollaries and specific guidance:
There are significant limitations in using clinical efficacy trials to predict that an adverse event will not occur because of their limited sensitivity. Clinical trials do provide valuable information when adverse events are detected.
A statistically significant increased rate in adverse events indicates that a sufficiently large number of events have occurred to allow one to conclude that the observation is unlikely due to chance alone. However, in populations and settings where no adverse event is expected, a nonsignificant trend also warrants some concern and consideration.
Epidemiological studies that contain information on the use of dietary supplements, when available, are valuable sources of information for evaluating their safety.
Given the state of the art in adverse event report evaluation and pharmacovigilance and risk management, and given the potential risk to the public entailed by serious adverse events, regulatory action can be justified on the basis of spontaneously reported adverse event report analysis alone or as the predominant source of information. Reports of certain adverse events warrant heightened concern because they have a known potential for significant morbidity (and in some cases, mortality).
If spontaneously reported adverse event reports are of high quality, irrespective of number, the effect of underreporting can be somewhat mitigated. The stronger the product-adverse event relationship and the lower the incidence of (and thus rarer) the adverse event occurring de novo/ naturalistically, the fewer the number of case reports that will be needed to perceive causality.
Recognizing that a full range of data is unlikely to be available for many dietary supplement ingredients, historical use may be taken into account as a surrogate measure for safety in the absence of relevant scientific and experimental data. In doing so, it is important to consider the relevance of the traditional use to the current use and, as such, FDA must have information regarding both the traditional use and the current use to determine if the traditional use sheds any light on the potential risk associated with current use.
The fact that a substance was consumed over a number of years does not indicate that it was consumed without adverse effects. Safe administration by non-oral routes of administration should not be taken as an indication of safety via oral administration. Historical information is useful only if the product in question is not so far removed from the original
substance as to constitute a distinct entity. If the current level of intake is significantly above what has been traditionally recommended, then the level of concern should be increased.
Altman DG. 1998. Statistical reviewing for medical journals. Stat Med 17:2661–2674.
Auriche M, Loupi E. 1993. Does proof of casualty ever exist in pharmacovigilance? Drug Safe 9:230–235.
Bannerman RH, Burton J, Wen-Chieh C, eds. 1983. Traditional Medicine and Health Care Coverage. A Reader for Health Administrators and Practitioners. Geneva: World Health Organization.
Barnes J, Mills SY, Abbot NC, Willoughby M, Ernst E. 1998. Different standards for reporting ADRs to herbal remedies and conventional OTC medicines: Face-to-face interview with 515 users of herbal remedies. Br J Clin Pharmacol 45:496–500.
Begaud B, Moride Y, Tubert-Bitter P, Chaslerie A, Haramburu F. 1994. False-positives in spontaneous reporting: Should we worry about them? Br J Clin Pharmacol 38:401–404.
Blum MD, Graham DJ, McCloskey CA. 1994. Temafloxacin syndrome: Review of 95 cases. Clin Infect Dis 18:946–950.
CFSAN (Center for Food Safety and Applied Nutrition). 2001. Overview of Dietary Supplements. Online. Food and Drug Administration. Available at http://www.cfsan.fda.gov/~dms/ds-oview.html. Accessed February 22, 2002.
Chadwick DJ, Marsh J, eds. 1990. Bioactive Compounds from Plants. Ciba Foundation Symposium 154. New York: John Wiley and Sons.
Chen RT, Rastogi SC, Mullen JR, Hayes SW, Cochi SL, Donlon JA, Wassilak SG. 1994. The Vaccine Adverse Event Reporting System (VAERS). Vaccine 12:542–550.
Chyka PA, McCommon SW. 2000. Reporting of adverse drug reactions by poison control centers in the US. Drug Safe 23:87–93.
CIOMS (Council for International Organizations of Medical Sciences). 1990. Standardization of definitions and criteria of causality assessment of adverse drug reaction. Drug-induced liver disorders: Report of an International Consensus Meeting. Int J Clin Pharmacol 28:317–322.
CIOMS. 1999. Guidelines for Preparing Core Clinical-Safety Information on Drugs, 2nd ed. Report of CIOMS Working Groups III and V. Geneva: CIOMS.
CIOMS. 2001. Current Challenges in Pharmacolvigilance: Pragmatic Approaches. Report of CIOMS Working Group V. Geneva: CIOMS.
Counsell C. 1997. Formulating questions and locating primary studies for inclusion in systematic reviews. Ann Intern Med 127:380–387.
Dobelis IN. 1997. Magic and Medicine of Plants. A Practical Guide to the Science, History, Folklore, and Everyday Uses of Medical Plants. Pleasantville, NY: Reader’s Digest Association.
Eisenberg DM, Davis RB, Ettner SL, Appel S, Wilkey S, Van Rompay M, Kessler RC. 1998. Trends in alternative medicine use in the United States, 1990–1997: Results of a follow-up national survey. J Am Med Assoc 280:1569–1575.
Faich GA. 1986. Adverse-drug-reaction monitoring. N Engl J Med 314:1589–1592.
Farnsworth NR. 2003. NAPRALERT (Natural Products ALERT) Database. Chicago: University of Chicago.
FDA (Food and Drug Administration). 2003a. Current good manufacturing practice in manufacturing, packing, or holding dietary ingredients and dietary supplements. Proposed rule. Fed Regis 68:12157–12263.
FDA. 2003b. FDA News. FDA Proposes Labeling and Manufacturing Standards for All Dietary Supplements. Online. Available at http://www.fda.gov/bbs/topics/NEWS/2003/NEW00876.html. Accessed July 2, 2003.
FDA. 2003c. Safety reporting requirements for human drug and biological products. Proposed rule. Fed Regis 68:12406–12497.
Finney DJ. 1971. Statistical aspects of monitoring for dangers in drug therapy. Methods Inf Med 10:1–8.
Fletcher AP. 1991. Spontaneous adverse drug reaction reporting vs event monitoring: A comparison. J R Soc Med 84:341–344.
GAO (General Accounting Office). 1992. Cross Design Synthesis: A New Strategy for Medical Effectiveness Research. GAO/PEMD-92-18. Washington, DC: GAO.
Gardner MJ, Machin D, Campbell MJ. 1986. Use of check lists in assessing the statistical content of medical studies. Br Med J 292:810–812.
Goldman SA. 1996. The Clinical Impact of Adverse Event Reporting. A MedWatch Continuing Education Article. Food and Drug Administration. Available at http://www.fda.gov/medwatch/articles.htm. Accessed February 25, 2003.
Goldman SA. 1998. Limitations and strengths of spontaneous reports data. Clin Ther 20:C40–C44.
Goldman SA, Kennedy DL, Lieberman R, eds. 1995. Clinical Therapeutics and the Recognition of Drug-Induced Disease. A MedWatch Continuing Education Article. Food and Drug Administration. Available at http://www.fda.gov/medwatch/articles.htm. Accessed June 4, 2003.
Griggs B. 1997. Green Pharmacy. The History and Evolution of Western Herbal Medicine. 2nd ed. Rochester, VT: Healing Arts Press.
Hennekens CH, Buring JE, Mayrent SL. 1987. Epidemiology in Medicine. Boston: Little, Brown.
Hill AB. 1971. Principles of Medical Statistics. 9th ed. New York: Oxford University Press.
ICH (International Conference on Harmonisation). 1995. Guidance for Industry. E3. Structure and Content of Clinical Study Reports. Rockville, MD: Drug Information Branch, Center for Drug Evaluation and Research, FDA.
Ioannidis JP, Lau J. 2001. Completeness of safety reporting in randomized trials: An evaluation of 7 medical areas. J Am Med Assoc 285:437–443.
Ioannidis JPA, Chew P, Lau J. 2002. Standardized retrieval of side effects data for metaanalysis of safety outcomes: A feasibility study in acute sinusitis. J Clin Epidemiol 55:619–626.
Karch FE, Smith CL, Kerzner B, Mazzullo JM, Weintraub M, Lasagna L. 1976. Adverse drug reactions—A matter of opinion. Clin Pharmacol Ther 19:489–492.
Kessler DA. 1993. Introducing MEDWatch: A new approach to reporting medication and device adverse effects and product problems. J Clin Res Drug Dev 7:183–190.
Koch-Weser J, Sellers EM, Zacest R. 1977. The ambiguity of adverse drug reactions. Eur J Clin Pharmacol 11:75–78.
Lewis JA. 1981. Post-marketing surveillance: How many patients? Trends Pharmacol Sci 2:93–94.
Moher D, Schulz KF, Altman DG. 2001. The CONSORT statement: Revised recommendations for improving the quality of reports of parallel group randomized trials. BMC Med Res Methodol 1:2–8.
OIG (Office of Inspector General). 2001. Adverse Event Reporting for Dietary Supplements. An Inadequate Safety Valve. Online. Food and Drug Administration. Available at http://oig.hhs.gov/oei/reports/oei-01-00-00180.pdf. Accessed February 26, 2003.
Prance GT, Chadwick DJ, Marsh J, eds. 1994. Ethnobotany and the Search for New Drugs. Ciba Foundation Symposium 185. New York: John Wiley and Sons.
Rawlins MD. 1995. Pharmacovigilance: Paradise lost, regained or postponed? J R Coll Physicians Lond 29:41–49.
Rossi AC, Knapp DE. 1984. Discovery of new adverse drug reactions. A review of the Food and Drug Administration’s spontaneous reporting system. J Am Med Assoc 252:1030–1033.
Rossouw JE, Anderson GL, Prentice RL, LaCroix AZ, Kooperberg C, Stefanick ML, Jackson RD, Beresford SA, Howard BV, Johnson KC, Kotchen JM, Ockene J. 2002. Risks and benefits of estrogen plus progestin in healthy postmenopausal women: Principal results from the Women’s Health Initiative randomized control trial. J Am Med Assoc 288:321–333.
Rothman KJ, Greenland S. 1998. Modern Epidemiology. 2nd ed. Philadelphia, PA: Lippincott-Raven.
Sachs RM, Bortnichak EA. 1986. An evaluation of spontaneous adverse drug reaction monitoring systems. Am J Med 81:49–55.
Sackett DL, Haynes RB, Tugwell P. 1991. Deciding whether your treatment has done harm. In: Sackett DL, ed. Clinical Epidemiology: A Basic Science for Clinical Medicine. 2nd ed. Boston: Little Brown.
Scott HD, Rosenbaum SE, Waters WJ, Colt AM, Andrews LG, Juergens JP, Faich GA. 1987. Rhode Island physicians’ recognition and reporting of adverse drug reactions. R I Med J 70:311–316.
Skjeldestad FE, Amundsen T, Hoibraaten E. 2000. Reporting of adverse drug reactions to the Norwegian Medicines Control Agency—A discipline with few participants. Tidsskr Nor Laegeforen 120:336–338.
Stolley PD. 1990. How to interpret studies of adverse drug reactions. Clin Pharmacol Ther 48:337–339.
Strom BL, Tugwell P. 1990. Pharmacoepidemiology: Current status, prospects, and problems. Ann Intern Med 113:179–181.
Tegtmeier M, Harnischfeger G. 1994. Methods for the reduction of thujone content in pharmaceutical preparations of Artemisia, Salvia and Thuja. Eur J Pharm Biopharm 40:337–340.
Temple R. 1999. Meta-analysis and epidemiologic studies in drug development and postmarketing surveillance. J Am Med Assoc 281:841–844.
Temple RJ, Jones JK, Crout JR. 1979. Adverse effects of newly marketed drugs. N Engl J Med 300:1046–1047.
Walker A. 2000. The Relation between Voluntary Notification and Material Risk in Dietary Supplement Safety. FDA Commissioned Paper. FDA Docket 00N-1200(41).
Wurglics M, Westerhoff K, Kaunzinger A, Wilke A, Baumeister A, Dressman J, Schubert-Zsilavecz M. 2001. Comparison of German St. John’s wort products according to hyperforin and total hypericin content. J Am Pharm Assoc 41:560–566.