+1 Recommend
0 collections
      • Record: found
      • Abstract: found
      • Article: found
      Is Open Access

      Meta-analyses of Adverse Effects Data Derived from Randomised Controlled Trials as Compared to Observational Studies: Methodological Overview

      1 , * , 2 , 3

      PLoS Medicine

      Public Library of Science

      Read this article at

          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.


          Su Golder and colleagues carry out an overview of meta-analyses to assess whether estimates of the risk of harm outcomes differ between randomized trials and observational studies. They find that, on average, there is no difference in the estimates of risk between overviews of observational studies and overviews of randomized trials.



          There is considerable debate as to the relative merits of using randomised controlled trial (RCT) data as opposed to observational data in systematic reviews of adverse effects. This meta-analysis of meta-analyses aimed to assess the level of agreement or disagreement in the estimates of harm derived from meta-analysis of RCTs as compared to meta-analysis of observational studies.

          Methods and Findings

          Searches were carried out in ten databases in addition to reference checking, contacting experts, citation searches, and hand-searching key journals, conference proceedings, and Web sites. Studies were included where a pooled relative measure of an adverse effect (odds ratio or risk ratio) from RCTs could be directly compared, using the ratio of odds ratios, with the pooled estimate for the same adverse effect arising from observational studies. Nineteen studies, yielding 58 meta-analyses, were identified for inclusion. The pooled ratio of odds ratios of RCTs compared to observational studies was estimated to be 1.03 (95% confidence interval 0.93–1.15). There was less discrepancy with larger studies. The symmetric funnel plot suggests that there is no consistent difference between risk estimates from meta-analysis of RCT data and those from meta-analysis of observational studies. In almost all instances, the estimates of harm from meta-analyses of the different study designs had 95% confidence intervals that overlapped (54/58, 93%). In terms of statistical significance, in nearly two-thirds (37/58, 64%), the results agreed (both studies showing a significant increase or significant decrease or both showing no significant difference). In only one meta-analysis about one adverse effect was there opposing statistical significance.


          Empirical evidence from this overview indicates that there is no difference on average in the risk estimate of adverse effects of an intervention derived from meta-analyses of RCTs and meta-analyses of observational studies. This suggests that systematic reviews of adverse effects should not be restricted to specific study types.

          Please see later in the article for the Editors' Summary

          Editors' Summary


          Whenever patients consult a doctor, they expect the treatments they receive to be effective and to have minimal adverse effects (side effects). To ensure that this is the case, all treatments now undergo exhaustive clinical research—carefully designed investigations that test new treatments and therapies in people. Clinical investigations fall into two main groups—randomized controlled trials (RCTs) and observational, or non-randomized, studies. In RCTs, groups of patients with a specific disease or condition are randomly assigned to receive the new treatment or a control treatment, and the outcomes (for example, improvements in health and the occurrence of specific adverse effects) of the two groups of patients are compared. Because the patients are randomly chosen, differences in outcomes between the two groups are likely to be treatment-related. In observational studies, patients who are receiving a specific treatment are enrolled and outcomes in this group are compared to those in a similar group of untreated patients. Because the patient groups are not randomly chosen, differences in outcomes between cases and controls may be the result of a hidden shared characteristic among the cases rather than treatment-related (so-called confounding variables).

          Why Was This Study Done?

          Although data from individual trials and studies are valuable, much more information about a potential new treatment can be obtained by systematically reviewing all the evidence and then doing a meta-analysis (so-called evidence-based medicine). A systematic review uses predefined criteria to identify all the research on a treatment; meta-analysis is a statistical method for combining the results of several studies to yield “pooled estimates” of the treatment effect (the efficacy of a treatment) and the risk of harm. Treatment effect estimates can differ between RCTs and observational studies, but what about adverse effect estimates? Can different study designs provide a consistent picture of the risk of harm, or are the results from different study designs so disparate that it would be meaningless to combine them in a single review? In this methodological overview, which comprises a systematic review and meta-analyses, the researchers assess the level of agreement in the estimates of harm derived from meta-analysis of RCTs with estimates derived from meta-analysis of observational studies.

          What Did the Researchers Do and Find?

          The researchers searched literature databases and reference lists, consulted experts, and hand-searched various other sources for studies in which the pooled estimate of an adverse effect from RCTs could be directly compared to the pooled estimate for the same adverse effect from observational studies. They identified 19 studies that together covered 58 separate adverse effects. In almost all instances, the estimates of harm obtained from meta-analyses of RCTs and observational studies had overlapping 95% confidence intervals. That is, in statistical terms, the estimates of harm were similar. Moreover, in nearly two-thirds of cases, there was agreement between RCTs and observational studies about whether a treatment caused a significant increase in adverse effects, a significant decrease, or no significant change (a significant change is one unlikely to have occurred by chance). Finally, the researchers used meta-analysis to calculate that the pooled ratio of the odds ratios (a statistical measurement of risk) of RCTs compared to observational studies was 1.03. This figure suggests that there was no consistent difference between risk estimates obtained from meta-analysis of RCT data and those obtained from meta-analysis of observational study data.

          What Do These Findings Mean?

          The findings of this methodological overview suggest that there is no difference on average in the risk estimate of an intervention's adverse effects obtained from meta-analyses of RCTs and from meta-analyses of observational studies. Although limited by some aspects of its design, this overview has several important implications for the conduct of systematic reviews of adverse effects. In particular, it suggests that, rather than limiting systematic reviews to certain study designs, it might be better to evaluate a broad range of studies. In this way, it might be possible to build a more complete, more generalizable picture of potential harms associated with an intervention, without any loss of validity, than by evaluating a single type of study. Such a picture, in combination with estimates of treatment effects also obtained from systematic reviews and meta-analyses, would help clinicians decide the best treatment for their patients.

          Additional Information

          Please access these Web sites via the online version of this summary at

          Related collections

          Most cited references 126

          • Record: found
          • Abstract: found
          • Article: not found

          Randomized, controlled trials, observational studies, and the hierarchy of research designs.

          In the hierarchy of research designs, the results of randomized, controlled trials are considered to be evidence of the highest grade, whereas observational studies are viewed as having less validity because they reportedly overestimate treatment effects. We used published meta-analyses to identify randomized clinical trials and observational studies that examined the same clinical topics. We then compared the results of the original reports according to the type of research design. A search of the Medline data base for articles published in five major medical journals from 1991 to 1995 identified meta-analyses of randomized, controlled trials and meta-analyses of either cohort or case-control studies that assessed the same intervention. For each of five topics, summary estimates and 95 percent confidence intervals were calculated on the basis of data from the individual randomized, controlled trials and the individual observational studies. For the five clinical topics and 99 reports evaluated, the average results of the observational studies were remarkably similar to those of the randomized, controlled trials. For example, analysis of 13 randomized, controlled trials of the effectiveness of bacille Calmette-Guérin vaccine in preventing active tuberculosis yielded a relative risk of 0.49 (95 percent confidence interval, 0.34 to 0.70) among vaccinated patients, as compared with an odds ratio of 0.50 (95 percent confidence interval, 0.39 to 0.65) from 10 case-control studies. In addition, the range of the point estimates for the effect of vaccination was wider for the randomized, controlled trials (0.20 to 1.56) than for the observational studies (0.17 to 0.84). The results of well-designed observational studies (with either a cohort or a case-control design) do not systematically overestimate the magnitude of the effects of treatment as compared with those in randomized, controlled trials on the same topic.
            • Record: found
            • Abstract: found
            • Article: not found

            Meta-analysis of the impact of 9 medication classes on falls in elderly persons.

            There is increasing recognition that the use of certain medications contributes to falls in seniors. Our objective was to update a previously completed meta-analysis looking at the association of medication use and falling to include relevant drug classes and new studies that have been completed since a previous meta-analysis. Studies were identified through a systematic search of English-language articles published from 1996 to 2007. We identified studies that were completed on patients older than 60 years, looking at the association between medication use and falling. Bayesian methods allowed us to combine the results of a previous meta-analysis with new information to estimate updated Bayesian odds ratios (ORs) and 95% credible intervals (95% CrIs) Of 11 118 identified articles, 22 met our inclusion criteria. Meta-analyses were completed on 9 unique drug classes, including 79 081 participants, with the following Bayesian unadjusted OR estimates: antihypertensive agents, OR, 1.24 (95% CrI, 1.01-1.50); diuretics, OR, 1.07 (95% CrI, 1.01-1.14); beta-blockers, OR, 1.01 (95% CrI, 0.86-1.17); sedatives and hypnotics, OR, 1.47 (95% CrI, 1.35-1.62); neuroleptics and antipsychotics, OR, 1.59 (95% CrI, 1.37-1.83); antidepressants, OR, 1.68 (95% CrI, 1.47-1.91); benzodiazepines, OR, 1.57 (95% CrI, 1.43-1.72); narcotics, OR, 0.96 (95% CrI, 0.78-1.18); and nonsteroidal anti-inflammatory drugs, OR, 1.21 (95% CrI, 1.01-1.44). The updated Bayesian adjusted OR estimates for diuretics, neuroleptics and antipsychotics, antidepressants, and benzodiazepines were 0.99 (95% CrI, 0.78-1.25), 1.39 (95% CrI, 0.94-2.00), 1.36 (95% CrI, 1.13-1.76), and 1.41 (95% CrI, 1.20-1.71), respectively. Stratification of studies had little effect on Bayesian OR estimates, with only small differences in the stratified ORs observed across population (for beta-blockers and neuroleptics and antipsychotics) and study type (for sedatives and hypnotics, benzodiazepines, and narcotics). An increased likelihood of falling was estimated for the use of sedatives and hypnotics, neuroleptics and antipsychotics, antidepressants, benzodiazepines, and nonsteroidal anti-inflammatory drugs in studies considered to have "good" medication and falls ascertainment. The use of sedatives and hypnotics, antidepressants, and benzodiazepines demonstrated a significant association with falls in elderly individuals.
              • Record: found
              • Abstract: found
              • Article: not found

              Cardiovascular risk and inhibition of cyclooxygenase: a systematic review of the observational studies of selective and nonselective inhibitors of cyclooxygenase 2.

              Evidence that rofecoxib increases the risk of myocardial infarction has led to scrutiny of other nonsteroidal anti-inflammatory drugs (NSAIDs). Regulatory agencies have provided variable advice regarding the cardiovascular risks with older nonselective NSAIDs. To undertake a systematic review and meta-analysis of controlled observational studies to compare the risks of serious cardiovascular events with individual NSAIDs and cyclooxygenase 2 inhibitors. Searches were conducted of electronic databases (1985-2006), scientific meeting proceedings, epidemiological research Web sites, and bibliographies of eligible studies. Eligible studies were of case-control or cohort design, reported on cardiovascular events (predominantly myocardial infarction) with cyclooxygenase 2 inhibitor, NSAID use, or both with nonuse/remote use of the drugs as the reference exposure. Of 7086 potentially eligible titles, 17 case-control and 6 cohort studies were included. Thirteen studies reported on cyclooxygenase 2 inhibitors, 23 on NSAIDs, and 13 on both groups of drugs. Two people independently extracted data and assessed study quality with disagreements resolved by consensus. Data were combined using a random-effects model. A dose-related risk was evident with rofecoxib, summary relative risk with 25 mg/d or less, 1.33 (95% confidence interval [CI], 1.00-1.79) and 2.19 (95% CI, 1.64-2.91) with more than 25 mg/d. The risk was elevated during the first month of treatment. Celecoxib was not associated with an elevated risk of vascular occlusion, summary relative risk 1.06 (95% CI, 0.91-1.23). Among older nonselective drugs, diclofenac had the highest risk with a summary relative risk of 1.40 (95% CI, 1.16-1.70). The other drugs had summary relative risks close to 1: naproxen, 0.97 (95% CI, 0.87-1.07); piroxicam, 1.06 (95% CI, 0.70-1.59); and ibuprofen, 1.07 (95% CI, 0.97-1.18). This review confirms the findings from randomized trials regarding the risk of cardiovascular events with rofecoxib and suggests that celecoxib in commonly used doses may not increase the risk, contradicts claims of a protective effect of naproxen, and raises serious questions about the safety of diclofenac, an older drug.

                Author and article information

                Role: Academic Editor
                PLoS Med
                PLoS Medicine
                Public Library of Science (San Francisco, USA )
                May 2011
                May 2011
                3 May 2011
                : 8
                : 5
                [1 ]Centre for Reviews and Dissemination, University of York, York, United Kingdom
                [2 ]School of Medicine, University of East Anglia, Norwich, United Kingdom
                [3 ]Department of Health Sciences, University of York, York, United Kingdom
                Leiden University Hospital, The Netherlands
                Author notes

                Conceived and designed the experiments: SG. Performed the study: SG YL. Analyzed the data: SG YL MB. Wrote the paper: SG YL MB. ICMJE criteria for authorship read and met: SG YL MB. Agree with the manuscript's results and conclusions: SG YL MB. Wrote the first draft of the paper: SG.

                Golder et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
                Page count
                Pages: 13
                Research Article
                Clinical Trials
                Case-Control Studies
                Cohort Studies
                Observational Studies
                Systematic Reviews
                Clinical Research Design



                Comment on this article