+1 Recommend
0 collections
      • Record: found
      • Abstract: found
      • Article: found
      Is Open Access

      Systematic reviews of adverse effects: framework for a structured approach

      1 , 2 , , 3 , the Cochrane Adverse Effects Methods Group

      BMC Medical Research Methodology

      BioMed Central

      Read this article at

          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.



          As every healthcare intervention carries some risk of harm, clinical decision making needs to be supported by a systematic assessment of the balance of benefit to harm. A systematic review that considers only the favourable outcomes of an intervention, without also assessing the adverse effects, can mislead by introducing a bias favouring the intervention.

          Much of the current guidance on systematic reviews is directed towards the evaluation of effectiveness; but this differs in important ways from the methods used in assessing the safety and tolerability of an intervention. A detailed discussion of why, how and when to include adverse effects in a systematic review, is required.


          This discussion paper, which presupposes a basic knowledge of systematic review methodology, was developed by consensus among experienced reviewers, members of the Adverse Effects Subgroup of The Cochrane Collaboration, and supplemented by a consultation of content experts in reviews methodology, as well as those working in drug safety.


          A logical framework for making decisions in reviews that incorporate adverse effects is provided. We explore situations where a comprehensive investigation of adverse effects is warranted and suggest strategies to identify practicable and clinically useful outcomes. The advantages and disadvantages of including observational and experimental study designs are reviewed. The consequences of including separate studies for intended and unintended effects are explained. Detailed advice is given on designing electronic searches for studies with adverse effects data. Reviewers of adverse effects are given general guidance on the assessment of study bias, data collection, analysis, presentation and the interpretation of harms in a systematic review.


          Readers need to be able to recognize how strategic choices made in the review process determine what harms are found, and how the findings may affect clinical decisions. Researchers undertaking a systematic review that incorporates adverse effect data should understand the rationale for the suggested methods and be able to implement them in their review.

          Related collections

          Most cited references 24

          • Record: found
          • Abstract: found
          • Article: not found

          Evaluating non-randomised intervention studies.

          To consider methods and related evidence for evaluating bias in non-randomised intervention studies. Systematic reviews and methodological papers were identified from a search of electronic databases; handsearches of key medical journals and contact with experts working in the field. New empirical studies were conducted using data from two large randomised clinical trials. Three systematic reviews and new empirical investigations were conducted. The reviews considered, in regard to non-randomised studies, (1) the existing evidence of bias, (2) the content of quality assessment tools, (3) the ways that study quality has been assessed and addressed. (4) The empirical investigations were conducted generating non-randomised studies from two large, multicentre randomised controlled trials (RCTs) and selectively resampling trial participants according to allocated treatment, centre and period. In the systematic reviews, eight studies compared results of randomised and non-randomised studies across multiple interventions using meta-epidemiological techniques. A total of 194 tools were identified that could be or had been used to assess non-randomised studies. Sixty tools covered at least five of six pre-specified internal validity domains. Fourteen tools covered three of four core items of particular importance for non-randomised studies. Six tools were thought suitable for use in systematic reviews. Of 511 systematic reviews that included non-randomised studies, only 169 (33%) assessed study quality. Sixty-nine reviews investigated the impact of quality on study results in a quantitative manner. The new empirical studies estimated the bias associated with non-random allocation and found that the bias could lead to consistent over- or underestimations of treatment effects, also the bias increased variation in results for both historical and concurrent controls, owing to haphazard differences in case-mix between groups. The biases were large enough to lead studies falsely to conclude significant findings of benefit or harm. Four strategies for case-mix adjustment were evaluated: none adequately adjusted for bias in historically and concurrently controlled studies. Logistic regression on average increased bias. Propensity score methods performed better, but were not satisfactory in most situations. Detailed investigation revealed that adequate adjustment can only be achieved in the unrealistic situation when selection depends on a single factor. Results of non-randomised studies sometimes, but not always, differ from results of randomised studies of the same intervention. Non-randomised studies may still give seriously misleading results when treated and control groups appear similar in key prognostic factors. Standard methods of case-mix adjustment do not guarantee removal of bias. Residual confounding may be high even when good prognostic data are available, and in some situations adjusted results may appear more biased than unadjusted results. Although many quality assessment tools exist and have been used for appraising non-randomised studies, most omit key quality domains. Healthcare policies based upon non-randomised studies or systematic reviews of non-randomised studies may need re-evaluation if the uncertainty in the true evidence base was not fully appreciated when policies were made. The inability of case-mix adjustment methods to compensate for selection bias and our inability to identify non-randomised studies that are free of selection bias indicate that non-randomised studies should only be undertaken when RCTs are infeasible or unethical. Recommendations for further research include: applying the resampling methodology in other clinical areas to ascertain whether the biases described are typical; developing or refining existing quality assessment tools for non-randomised studies; investigating how quality assessments of non-randomised studies can be incorporated into reviews and the implications of individual quality features for interpretation of a review's results; examination of the reasons for the apparent failure of case-mix adjustment methods; and further evaluation of the role of the propensity score.
            • Record: found
            • Abstract: found
            • Article: not found

            Completeness of safety reporting in randomized trials: an evaluation of 7 medical areas.

            Randomized trials with adequate sample size offer an opportunity to assess the safety of new medications in a controlled setting; however, generalizable data on drug safety reporting are sparse. To scrutinize the completeness of safety reporting in randomized trials. Survey of safety reporting in 192 randomized drug trials 7 diverse topics with sample sizes of at least 100 patients and at least 50 patients in a study arm (N = 130074 patients). Trial reports were identified from comprehensive meta-analyses in 7 medical areas. Adequate reporting of specific adverse effects and frequency and reasons for withdrawals due to toxic effects; article space allocated to safety reporting and predictors of such reporting. Severity of clinical adverse effects and laboratory-determined toxicity was adequately defined in only 39% and 29% of trial reports, respectively. Only 46% of trials stated the frequency of specific reasons for discontinuation of study treatment due to toxicity. For these 3 parameters, there was significant heterogeneity in rates of adequate reporting across topics (P =.003, P<.001, and P =.02, respectively). Overall, the median space allocated to safety results was 0.3 page. A similar amount of space was devoted to contributor names and affiliations (P =.16). On average, the percentage of space devoted to safety in the results section was 9.3% larger in trials involving dose comparisons than in those that did not (P<.001) and 3.8% smaller in trials reporting statistically significant results for efficacy outcomes (P =.047). The quality and quantity of safety reporting vary across medical areas, study designs, and settings but they are largely inadequate. Current standards for safety reporting in randomized trials should be revised to address this inadequacy.
              • Record: found
              • Abstract: found
              • Article: not found

              Comparison of evidence on harms of medical interventions in randomized and nonrandomized studies.

              Information on major harms of medical interventions comes primarily from epidemiologic studies performed after licensing and marketing. Comparison with data from large-scale randomized trials is occasionally feasible. We compared evidence from randomized trials with that from epidemiologic studies to determine whether they give different estimates of risk for important harms of medical interventions. We targeted well-defined, specific harms of various medical interventions for which data were already available from large-scale randomized trials (> 4000 subjects). Nonrandomized studies involving at least 4000 subjects addressing these same harms were retrieved through a search of MEDLINE. We compared the relative risks and absolute risk differences for specific harms in the randomized and nonrandomized studies. Eligible nonrandomized studies were found for 15 harms for which data were available from randomized trials addressing the same harms. Comparisons of relative risks between the study types were feasible for 13 of the 15 topics, and of absolute risk differences for 8 topics. The estimated increase in relative risk differed more than 2-fold between the randomized and nonrandomized studies for 7 (54%) of the 13 topics; the estimated increase in absolute risk differed more than 2-fold for 5 (62%) of the 8 topics. There was no clear predilection for randomized or nonrandomized studies to estimate greater relative risks, but usually (75% [6/8]) the randomized trials estimated larger absolute excess risks of harm than the nonrandomized studies did. Nonrandomized studies are often conservative in estimating absolute risks of harms. It would be useful to compare and scrutinize the evidence on harms obtained from both randomized and nonrandomized studies.

                Author and article information

                BMC Med Res Methodol
                BMC Medical Research Methodology
                BioMed Central (London )
                5 July 2007
                : 7
                : 32
                [1 ]School of Medicine, Health Policy and Practice, University of East Anglia, Norwich, NR4 7TJ, UK
                [2 ]Department of Clinical Pharmacology, University of Oxford, Radcliffe Infirmary, Woodstock Road, Oxford, OX2 6HE, UK
                [3 ]The UK Cochrane Centre, NHS R&D Programme, Summertown Pavilion, Middle Way, Oxford, OX2 7LG, UK
                Copyright © 2007 Loke et al; licensee BioMed Central Ltd.

                This is an Open Access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

                Technical Advance



                Comment on this article