82
views
0
recommends
+1 Recommend
0 collections
    0
    shares
      • Record: found
      • Abstract: found
      • Article: found
      Is Open Access

      Evaluation of the Cochrane Collaboration’s tool for assessing the risk of bias in randomized trials: focus groups, online survey, proposed recommendations and their implementation

      research-article

      Read this article at

      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          Background

          In 2008, the Cochrane Collaboration introduced a tool for assessing the risk of bias in clinical trials included in Cochrane reviews. The risk of bias (RoB) tool is based on narrative descriptions of evidence-based methodological features known to increase the risk of bias in trials.

          Methods

          To assess the usability of this tool, we conducted an evaluation by means of focus groups, online surveys and a face-to-face meeting. We obtained feedback from a range of stakeholders within The Cochrane Collaboration regarding their experiences with, and perceptions of, the RoB tool and associated guidance materials. We then assessed this feedback in a face-to-face meeting of experts and stakeholders and made recommendations for improvements and further developments of the RoB tool.

          Results

          The survey attracted 380 responses. Respondents reported taking an average of between 10 and 60 minutes per study to complete their RoB assessments, which 83% deemed acceptable. Most respondents (87% of authors and 95% of editorial staff) thought RoB assessments were an improvement over past approaches to trial quality assessment. Most authors liked the standardized approach (81%) and the ability to provide quotes to support judgements (74%). A third of participants disliked the increased workload and found the wording describing RoB judgements confusing. The RoB domains reported to be the most difficult to assess were incomplete outcome data and selective reporting of outcomes. Authors expressed the need for more guidance on how to incorporate RoB assessments into meta-analyses and review conclusions. Based on this evaluation, recommendations were made for improvements to the RoB tool and the associated guidance. The implementation of these recommendations is currently underway.

          Conclusions

          Overall, respondents identified positive experiences and perceptions of the RoB tool. Revisions of the tool and associated guidance made in response to this evaluation, and improved provision of training, may improve implementation.

          Related collections

          Most cited references7

          • Record: found
          • Abstract: found
          • Article: not found

          Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials.

          To determine if inadequate approaches to randomized controlled trial design and execution are associated with evidence of bias in estimating treatment effects. An observational study in which we assessed the methodological quality of 250 controlled trials from 33 meta-analyses and then analyzed, using multiple logistic regression models, the associations between those assessments and estimated treatment effects. Meta-analyses from the Cochrane Pregnancy and Childbirth Database. The associations between estimates of treatment effects and inadequate allocation concealment, exclusions after randomization, and lack of double-blinding. Compared with trials in which authors reported adequately concealed treatment allocation, trials in which concealment was either inadequate or unclear (did not report or incompletely reported a concealment approach) yielded larger estimates of treatment effects (P < .001). Odds ratios were exaggerated by 41% for inadequately concealed trials and by 30% for unclearly concealed trials (adjusted for other aspects of quality). Trials in which participants had been excluded after randomization did not yield larger estimates of effects, but that lack of association may be due to incomplete reporting. Trials that were not double-blind also yielded larger estimates of effects (P = .01), with odds ratios being exaggerated by 17%. This study provides empirical evidence that inadequate methodological approaches in controlled trials, particularly those representing poor allocation concealment, are associated with bias. Readers of trial reports should be wary of these pitfalls, and investigators must improve their design, execution, and reporting of trials.
            Bookmark
            • Record: found
            • Abstract: not found
            • Article: not found

            Systematic reviews in health care: Assessing the quality of controlled clinical trials.

              Bookmark
              • Record: found
              • Abstract: found
              • Article: not found

              Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials.

              Randomized trials without reported adequate allocation concealment have been shown to overestimate the benefit of experimental interventions. We investigated the robustness of conclusions drawn from meta-analyses to exclusion of such trials. Random sample of 38 reviews from The Cochrane Library 2003, issue 2 and 32 other reviews from PubMed accessed in 2002. Eligible reviews presented a binary effect estimate from a meta-analysis of randomized controlled trials as the first statistically significant result that supported a conclusion in favour of one of the interventions. We assessed the methods sections of the trials in each included meta-analysis for adequacy of allocation concealment. We replicated each meta-analysis using the authors' methods but included only trials that had adequate allocation concealment. Conclusions were defined as not supported if our result was not statistically significant. Thirty-four of the 70 meta-analyses contained a mixture of trials with unclear or inadequate concealment as well as trials with adequate allocation concealment. Four meta-analyses only contained trials with adequate concealment, and 32, only trials with unclear or inadequate concealment. When only trials with adequate concealment were included, 48 of 70 conclusions (69%; 95% confidence interval: 56-79%) lost support. The loss of support mainly reflected loss of power (the total number of patients was reduced by 49%) but also a shift in the point estimate towards a less beneficial effect. Two-thirds of conclusions in favour of one of the interventions were no longer supported if only trials with adequate allocation concealment were included.
                Bookmark

                Author and article information

                Contributors
                Journal
                Syst Rev
                Syst Rev
                Systematic Reviews
                BioMed Central
                2046-4053
                2014
                15 April 2014
                : 3
                : 37
                Affiliations
                [1 ]School of Social and Community Medicine, University of Bristol, Bristol, UK
                [2 ]Ottawa Integrative Cancer Centre, Ottawa, ON, Canada
                [3 ]Ottawa Hospital Research Institute, Ottawa, ON, Canada
                [4 ]Centre for Statistics in Medicine, University of Oxford, Oxford, UK
                [5 ]Department of Epidemiology and Community Medicine, Faculty of Medicine, University of Ottawa, Ottawa, ON, Canada
                [6 ]Centre for Reviews and Dissemination, University of York, York, UK
                Article
                2046-4053-3-37
                10.1186/2046-4053-3-37
                4022341
                24731537
                e5bf95c0-f1e6-4345-a105-8ed0d53ce3db
                Copyright © 2014 Savović et al.; licensee BioMed Central Ltd.

                This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.

                History
                : 23 July 2013
                : 10 March 2014
                Categories
                Research

                Public health
                survey,focus groups,bias assessment,quality assessment,systematic reviews
                Public health
                survey, focus groups, bias assessment, quality assessment, systematic reviews

                Comments

                Comment on this article