Risk of bias tools in systematic reviews of health interventions: an analysis of PROSPERO-registered protocols

There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

Abstract

Background

Systematic reviews of health interventions are increasingly incorporating evidence outside of randomized controlled trials (RCT). While non-randomized study (NRS) types may be more prone to bias compared to RCT, the tools used to evaluate risk of bias (RoB) in NRS are less straightforward and no gold standard tool exists. The objective of this study was to evaluate the planned use of RoB tools in systematic reviews of health interventions, specifically for reviews that planned to incorporate evidence from RCT and/or NRS.

Methods

We evaluated a random sample of non-Cochrane protocols for systematic reviews of interventions registered in PROSPERO between January 1 and October 12, 2018. For each protocol, we extracted data on the types of studies to be included (RCT and/or NRS) as well as the name and number of RoB tools planned to be used according to study design. We then conducted a longitudinal analysis of the most commonly reported tools in the random sample. Using keywords and name variants for each tool, we searched PROSPERO records by year since the inception of the database (2011 to December 7, 2018), restricting the keyword search to the “Risk of bias (quality) assessment” field.

Results

In total, 471 randomly sampled PROSPERO protocols from 2018 were included in the analysis. About two-thirds (63%) of these planned to include NRS, while 37% restricted study design to RCT or quasi-RCT. Over half of the protocols that planned to include NRS listed only a single RoB tool, most frequently the Cochrane RoB Tool. The Newcastle-Ottawa Scale and ROBINS-I were the most commonly reported tools for NRS (39% and 33% respectively) for systematic reviews that planned to use multiple RoB tools. Looking at trends over time, the planned use of the Cochrane RoB Tool and ROBINS-I seems to be increasing.

Conclusions

While RoB tool selection for RCT was consistent, with the Cochrane RoB Tool being the most frequently reported in PROSPERO protocols, RoB tools for NRS varied widely. Results suggest a need for more education and awareness on the appropriate use of RoB tools for NRS. Given the heterogeneity of study designs comprising NRS, multiple RoB tools tailored to specific designs may be required.

Related collections

Most cited references 20

Record: found
Abstract: found
Article: found

Is Open Access

Evaluation of the Cochrane tool for assessing risk of bias in randomized clinical trials: overview of published comments and analysis of user practice in Cochrane and non-Cochrane reviews

Lars V. Jørgensen, Asger S. Paludan-Müller, David R. T. Laursen … (2016)

Background The Cochrane risk of bias tool for randomized clinical trials was introduced in 2008 and has frequently been commented on and used in systematic reviews. We wanted to evaluate the tool by reviewing published comments on its strengths and challenges and by describing and analysing how the tool is applied to both Cochrane and non-Cochrane systematic reviews. Methods A review of published comments (searches in PubMed, The Cochrane Methodology Register and Google Scholar) and an observational study (100 Cochrane and 100 non-Cochrane reviews from 2014). Results Our review included 68 comments, 15 of which were categorised as major. The main strengths of the tool were considered to be its aim (to assess trial conduct and not reporting), its developmental basis (wide consultation, empirical and theoretical evidence) and its transparent procedures. The challenges of the tool were mainly considered to be its choice of core bias domains (e.g. not involving funding/conflicts of interest) and issues to do with implementation (i.e. modest inter-rater agreement) and terminology. Our observational study found that the tool was used in all Cochrane reviews (100/100) and was the preferred tool in non-Cochrane reviews (31/100). Both types of reviews frequently implemented the tool in non-recommended ways. Most Cochrane reviews planned to use risk of bias assessments as basis for sensitivity analyses (70 %), but only a minority conducted such analyses (19 %) because, in many cases, few trials were assessed as having “low” risk of bias for all standard domains (6 %). The judgement of at least one risk of bias domain as “unclear” was found in 89 % of included randomized clinical trials (1103/1242). Conclusions The Cochrane tool has become the standard approach to assess risk of bias in randomized clinical trials but is frequently implemented in a non-recommended way. Based on published comments and how it is applied in practice in systematic reviews, the tool may be further improved by a revised structure and more focused guidance. Electronic supplementary material The online version of this article (doi:10.1186/s13643-016-0259-8) contains supplementary material, which is available to authorized users.

0 comments Cited 91 times – based on 0 reviews      Review now

Bookmark

Record: found
Abstract: found
Article: not found

Issues relating to study design and risk of bias when including non-randomized studies in systematic reviews on the effects of interventions.

Julian PT Higgins, Craig Ramsay, Barnaby Reeves … (2013)

Non-randomized studies may provide valuable evidence on the effects of interventions. They are the main source of evidence on the intended effects of some types of interventions and often provide the only evidence about the effects of interventions on long-term outcomes, rare events or adverse effects. Therefore, systematic reviews on the effects of interventions may include various types of non-randomized studies. In this second paper in a series, we address how review authors might articulate the particular non-randomized study designs they will include and how they might evaluate, in general terms, the extent to which a particular non-randomized study is at risk of important biases. We offer guidance for describing and classifying different non-randomized designs based on specific features of the studies in place of using non-informative study design labels. We also suggest criteria to consider when deciding whether to include non-randomized studies. We conclude that a taxonomy of study designs based on study design features is needed. Review authors need new tools specifically to assess the risk of bias for some non-randomized designs that involve a different inferential logic compared with parallel group trials. Copyright © 2012 John Wiley & Sons, Ltd.

0 comments Cited 63 times – based on 0 reviews      Review now

Bookmark

Record: found
Abstract: found
Article: found

Is Open Access

Quasi-experimental study designs series—paper 5: a checklist for classifying studies evaluating the effects on health interventions—a taxonomy without labels

Barnaby Reeves, George A Wells, Hugh Waddington (2017)

Objectives The aim of the study was to extend a previously published checklist of study design features to include study designs often used by health systems researchers and economists. Our intention is to help review authors in any field to set eligibility criteria for studies to include in a systematic review that relate directly to the intrinsic strength of the studies in inferring causality. We also seek to clarify key equivalences and differences in terminology used by different research communities. Study Design and Setting Expert consensus meeting. Results The checklist comprises seven questions, each with a list of response items, addressing: clustering of an intervention as an aspect of allocation or due to the intrinsic nature of the delivery of the intervention; for whom, and when, outcome data are available; how the intervention effect was estimated; the principle underlying control for confounding; how groups were formed; the features of a study carried out after it was designed; and the variables measured before intervention. Conclusion The checklist clarifies the basis of credible quasi-experimental studies, reconciling different terminology used in different fields of investigation and facilitating communications across research communities. By applying the checklist, review authors' attention is also directed to the assumptions underpinning the methods for inferring causality.

0 comments Cited 52 times – based on 0 reviews      Review now

Bookmark

All references

Author and article information

Contributors

Kelly Farrah: kfarr081@uottawa.ca

Kelsey Young: kelsey.young@canada.ca

Matthew C. Tunis: matthew.tunis@canada.ca

Linlu Zhao: linlu.zhao@canada.ca

Journal

Journal ID (nlm-ta): Syst Rev

Journal ID (iso-abbrev): Syst Rev

Title: Systematic Reviews

Publisher: BioMed Central (London )

ISSN (Electronic): 2046-4053

Publication date (Electronic): 15 November 2019

Publication date PMC-release: 15 November 2019

Publication date Collection: 2019

Volume: 8

Electronic Location Identifier: 280

Affiliations

ISNI 0000 0001 0805 4386, GRID grid.415368.d, Centre for Immunization and Respiratory Infectious Diseases, , Public Health Agency of Canada, ; Ottawa, Canada

Article

Publisher ID: 1172

DOI: 10.1186/s13643-019-1172-8

PMC ID: 6857304

PubMed ID: 31730014

SO-VID: 4344a5ae-60c6-414d-addb-db40e2b9df59

License:

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

History

Date received : 27 March 2019

Date accepted : 27 September 2019

Custom metadata

ScienceOpen disciplines: Public health

Keywords: critical appraisal,non-randomized studies,prospero,risk of bias,systematic reviews

Data availability:

ScienceOpen disciplines: Public health

Keywords: critical appraisal, non-randomized studies, prospero, risk of bias, systematic reviews

Risk of bias tools in systematic reviews of health interventions: an analysis of PROSPERO-registered protocols

Read this article at

Abstract

Background

Methods

Results

Conclusions

Related collections

Karger: Digital Health

Most cited references 20

Evaluation of the Cochrane tool for assessing risk of bias in randomized clinical trials: overview of published comments and analysis of user practice in Cochrane and non-Cochrane reviews

Issues relating to study design and risk of bias when including non-randomized studies in systematic reviews on the effects of interventions.

Quasi-experimental study designs series—paper 5: a checklist for classifying studies evaluating the effects on health interventions—a taxonomy without labels

Author and article information

Contributors

Journal

Affiliations

Article

History

Categories

Custom metadata

Comments

Comment on this article

Similar content 205

Cited by 63

Most referenced authors 615