8
views
0
recommends
+1 Recommend
0 collections
    0
    shares
      • Record: found
      • Abstract: found
      • Article: found
      Is Open Access

      Dealing with substantial heterogeneity in Cochrane reviews. Cross-sectional study

      research-article
      1 , , 1 , 1
      BMC Medical Research Methodology
      BioMed Central

      Read this article at

      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          Background

          Dealing with heterogeneity in meta-analyses is often tricky, and there is only limited advice for authors on what to do. We investigated how authors addressed different degrees of heterogeneity, in particular whether they used a fixed effect model, which assumes that all the included studies are estimating the same true effect, or a random effects model where this is not assumed.

          Methods

          We sampled randomly 60 Cochrane reviews from 2008, which presented a result in its first meta-analysis with substantial heterogeneity (I 2 greater than 50%, i.e. more than 50% of the variation is due to heterogeneity rather than chance). We extracted information on choice of statistical model, how the authors had handled the heterogeneity, and assessed the methodological quality of the reviews in relation to this.

          Results

          The distribution of heterogeneity was rather uniform in the whole I 2 interval, 50-100%. A fixed effect model was used in 33 reviews (55%), but there was no correlation between I 2 and choice of model (P = 0.79). We considered that 20 reviews (33%), 16 of which had used a fixed effect model, had major problems. The most common problems were: use of a fixed effect model and lack of rationale for choice of that model, lack of comment on even severe heterogeneity and of reservations and explanations of its likely causes. The problematic reviews had significantly fewer included trials than other reviews (4.3 vs. 8.0, P = 0.024). The problems became less pronounced with time, as those reviews that were most recently updated more often used a random effects model.

          Conclusion

          One-third of Cochrane reviews with substantial heterogeneity had major problems in relation to their handling of heterogeneity. More attention is needed to this issue, as the problems we identified can be essential for the conclusions of the reviews.

          Related collections

          Most cited references9

          • Record: found
          • Abstract: found
          • Article: not found

          Issues in the selection of a summary statistic for meta-analysis of clinical trials with binary outcomes.

          Meta-analysis of binary data involves the computation of a weighted average of summary statistics calculated for each trial. The selection of the appropriate summary statistic is a subject of debate due to conflicts in the relative importance of mathematical properties and the ability to intuitively interpret results. This paper explores the process of identifying a summary statistic most likely to be consistent across trials when there is variation in control group event rates. Four summary statistics are considered: odds ratios (OR); risk differences (RD) and risk ratios of beneficial (RR(B)); and harmful outcomes (RR(H)). Each summary statistic corresponds to a different pattern of predicted absolute benefit of treatment with variation in baseline risk, the greatest difference in patterns of prediction being between RR(B) and RR(H). Selection of a summary statistic solely based on identification of the best-fitting model by comparing tests of heterogeneity is problematic, principally due to low numbers of trials. It is proposed that choice of a summary statistic should be guided by both empirical evidence and clinically informed debate as to which model is likely to be closest to the expected pattern of treatment benefit across baseline risks. Empirical investigations comparing the four summary statistics on a sample of 551 systematic reviews provide evidence that the RR and OR models are on average more consistent than RD, there being no difference on average between RR and OR. From a second sample of 114 meta-analyses evidence indicates that for interventions aimed at preventing an undesirable event, greatest absolute benefits are observed in trials with the highest baseline event rates, corresponding to the model of constant RR(H). The appropriate selection for a particular meta-analysis may depend on understanding reasons for variation in control group event rates; in some situations uncertainty about the choice of summary statistic will remain. Copyright 2002 John Wiley & Sons, Ltd.
            Bookmark
            • Record: found
            • Abstract: found
            • Article: not found

            Medicine residents' understanding of the biostatistics and results in the medical literature.

            Physicians depend on the medical literature to keep current with clinical information. Little is known about residents' ability to understand statistical methods or how to appropriately interpret research outcomes. To evaluate residents' understanding of biostatistics and interpretation of research results. Multiprogram cross-sectional survey of internal medicine residents. Percentage of questions correct on a biostatistics/study design multiple-choice knowledge test. The survey was completed by 277 of 367 residents (75.5%) in 11 residency programs. The overall mean percentage correct on statistical knowledge and interpretation of results was 41.4% (95% confidence interval [CI], 39.7%-43.3%) vs 71.5% (95% CI, 57.5%-85.5%) for fellows and general medicine faculty with research training (P < .001). Higher scores in residents were associated with additional advanced degrees (50.0% [95% CI, 44.5%-55.5%] vs 40.1% [95% CI, 38.3%-42.0%]; P < .001); prior biostatistics training (45.2% [95% CI, 42.7%-47.8%] vs 37.9% [95% CI, 35.4%-40.3%]; P = .001); enrollment in a university-based training program (43.0% [95% CI, 41.0%-45.1%] vs 36.3% [95% CI, 32.6%-40.0%]; P = .002); and male sex (44.0% [95% CI, 41.4%-46.7%] vs 38.8% [95% CI, 36.4%-41.1%]; P = .004). On individual knowledge questions, 81.6% correctly interpreted a relative risk. Residents were less likely to know how to interpret an adjusted odds ratio from a multivariate regression analysis (37.4%) or the results of a Kaplan-Meier analysis (10.5%). Seventy-five percent indicated they did not understand all of the statistics they encountered in journal articles, but 95% felt it was important to understand these concepts to be an intelligent reader of the literature. Most residents in this study lacked the knowledge in biostatistics needed to interpret many of the results in published clinical research. Residency programs should include more effective biostatistics training in their curricula to successfully prepare residents for this important lifelong learning skill.
              Bookmark
              • Record: found
              • Abstract: found
              • Article: not found

              Statistical heterogeneity in systematic reviews of clinical trials: a critical appraisal of guidelines and practice.

              Heterogeneity between study results can be a problem in any systematic review or meta-analysis of clinical trials. Identifying its presence, investigating its cause and correctly accounting for it in analyses all involve difficult decisions for the researcher. Our objectives were: to collate recommendations on the subject of dealing with heterogeneity in systematic reviews of clinical trials; to investigate current practice in addressing heterogeneity in Cochrane reviews; and to compare current practice with recommendations. We review guidelines for those undertaking systematic reviews and examine how heterogeneity is addressed in practice in a sample of systematic reviews, and their protocols, from the Cochrane Database of Systematic Reviews. Advice to reviewers is on the whole consistent and sensible. However, examination of a sample of Cochrane protocols and reviews demonstrates that the advice is difficult to follow given the small numbers of studies identified in many systematic reviews, the difficulty of pre-specifying important effect modifiers for subgroup analysis or meta-regression and the unresolved debate concerning fixed versus random effects meta-analyses. There was disagreement between protocols and reviews, often either regarding choice of important potential effect modifiers or due to the review identifying too few studies to perform planned analyses. Guidelines that address practical issues are required to reduce the risk of spurious findings from investigations of heterogeneity. This may involve discouraging statistical investigations such as subgroup analyses and meta-regression, rather than simply adopting a cautious approach to their interpretation, unless a large number of studies is available. The notion of a priori specification of potential effect modifiers for a retrospective review of studies is ill-defined, and the appropriateness of using a statistical test for heterogeneity to decide between analysis strategies is suspect.
                Bookmark

                Author and article information

                Journal
                BMC Med Res Methodol
                BMC Medical Research Methodology
                BioMed Central
                1471-2288
                2011
                24 February 2011
                : 11
                : 22
                Affiliations
                [1 ]Nordic Cochrane Centre, Rigshospitalet and University of Copenhagen, Denmark
                Article
                1471-2288-11-22
                10.1186/1471-2288-11-22
                3056846
                21349195
                9c974032-5fca-4564-88f1-18069c7f9c00
                Copyright ©2011 Schroll et al; licensee BioMed Central Ltd.

                This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

                Categories
                Research Article

                Medicine
                Medicine

                Comments

                Comment on this article