+1 Recommend
0 collections
      • Record: found
      • Abstract: found
      • Article: found
      Is Open Access

      The risk of bias in observational studies of exposures (ROBINS-E) tool: concerns arising from application to observational studies of exposures


      Read this article at

          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.



          Systematic reviews, which assess the risk of bias in included studies, are increasingly used to develop environmental hazard assessments and public health guidelines. These research areas typically rely on evidence from human observational studies of exposures, yet there are currently no universally accepted standards for assessing risk of bias in such studies. The risk of bias in non-randomised studies of exposures (ROBINS-E) tool has been developed by building upon tools for risk of bias assessment of randomised trials, diagnostic test accuracy studies and observational studies of interventions. This paper reports our experience with the application of the ROBINS-E tool.


          We applied ROBINS-E to 74 exposure studies (60 cohort studies, 14 case-control studies) in 3 areas: environmental risk, dietary exposure and drug harm. All investigators provided written feedback, and we documented verbal discussion of the tool. We inductively and iteratively classified the feedback into 7 themes based on commonalities and differences until all the feedback was accounted for in the themes. We present a description of each theme.


          We identified practical concerns with the premise that ROBINS-E is a structured comparison of the observational study being rated to the ‘ideal’ randomised controlled trial. ROBINS-E assesses 7 domains of bias, but relevant questions related to some critical sources of bias, such as exposure and funding source, are not assessed. ROBINS-E fails to discriminate between studies with a single risk of bias or multiple risks of bias. ROBINS-E is severely limited at determining whether confounders will bias study outcomes. The construct of co-exposures was difficult to distinguish from confounders. Applying ROBINS-E was time-consuming and confusing.


          Our experience suggests that the ROBINS-E tool does not meet the need for an international standard for evaluating human observational studies for questions of harm relevant to public and environmental health. We propose that a simpler tool, based on empirical evidence of bias, would provide accurate measures of risk of bias and is more likely to meet the needs of the environmental and public health community.

          Related collections

          Most cited references26

          • Record: found
          • Abstract: not found
          • Article: not found

          The hazards of scoring the quality of clinical trials for meta-analysis.

          Although it is widely recommended that clinical trials undergo some type of quality review, the number and variety of quality assessment scales that exist make it unclear how to achieve the best assessment. To determine whether the type of quality assessment scale used affects the conclusions of meta-analytic studies. Meta-analysis of 17 trials comparing low-molecular-weight heparin (LMWH) with standard heparin for prevention of postoperative thrombosis using 25 different scales to identify high-quality trials. The association between treatment effect and summary scores and the association with 3 key domains (concealment of treatment allocation, blinding of outcome assessment, and handling of withdrawals) were examined in regression models. Pooled relative risks of deep vein thrombosis with LMWH vs standard heparin in high-quality vs low-quality trials as determined by 25 quality scales. Pooled relative risks from high-quality trials ranged from 0.63 (95% confidence interval [CI], 0.44-0.90) to 0.90 (95% CI, 0.67-1.21) vs 0.52 (95% CI, 0.24-1.09) to 1.13 (95% CI, 0.70-1.82) for low-quality trials. For 6 scales, relative risks of high-quality trials were close to unity, indicating that LMWH was not significantly superior to standard heparin, whereas low-quality trials showed better protection with LMWH (P<.05). Seven scales showed the opposite: high quality trials showed an effect whereas low quality trials did not. For the remaining 12 scales, effect estimates were similar in the 2 quality strata. In regression analysis, summary quality scores were not significantly associated with treatment effects. There was no significant association of treatment effects with allocation concealment and handling of withdrawals. Open outcome assessment, however, influenced effect size with the effect of LMWH, on average, being exaggerated by 35% (95% CI, 1%-57%; P= .046). Our data indicate that the use of summary scores to identify trials of high quality is problematic. Relevant methodological aspects should be assessed individually and their influence on effect sizes explored.
            • Record: found
            • Abstract: found
            • Article: found
            Is Open Access

            The Navigation Guide Systematic Review Methodology: A Rigorous and Transparent Method for Translating Environmental Health Science into Better Health Outcomes

            Background: Synthesizing what is known about the environmental drivers of health is instrumental to taking prevention-oriented action. Methods of research synthesis commonly used in environmental health lag behind systematic review methods developed in the clinical sciences over the past 20 years. Objectives: We sought to develop a proof of concept of the “Navigation Guide,” a systematic and transparent method of research synthesis in environmental health. Discussion: The Navigation Guide methodology builds on best practices in research synthesis in evidence-based medicine and environmental health. Key points of departure from current methods of expert-based narrative review prevalent in environmental health include a prespecified protocol, standardized and transparent documentation including expert judgment, a comprehensive search strategy, assessment of “risk of bias,” and separation of the science from values and preferences. Key points of departure from evidence-based medicine include assigning a “moderate” quality rating to human observational studies and combining diverse evidence streams. Conclusions: The Navigation Guide methodology is a systematic and rigorous approach to research synthesis that has been developed to reduce bias and maximize transparency in the evaluation of environmental health information. Although novel aspects of the method will require further development and validation, our findings demonstrated that improved methods of research synthesis under development at the National Toxicology Program and under consideration by the U.S. Environmental Protection Agency are fully achievable. The institutionalization of robust methods of systematic and transparent review would provide a concrete mechanism for linking science to timely action to prevent harm. Citation: Woodruff TJ, Sutton P. 2014. The Navigation Guide systematic review methodology: a rigorous and transparent method for translating environmental health science into better health outcomes. Environ Health Perspect 122:1007–1014; http://dx.doi.org/10.1289/ehp.1307175
              • Record: found
              • Abstract: found
              • Article: found
              Is Open Access

              Systematic Review and Evidence Integration for Literature-Based Environmental Health Science Assessments

              Background: Systematic-review methodologies provide objectivity and transparency to the process of collecting and synthesizing scientific evidence in reaching conclusions on specific research questions. There is increasing interest in applying these procedures to address environmental health questions. Objectives: The goal was to develop a systematic-review framework to address environmental health questions by extending approaches developed for clinical medicine to handle the breadth of data relevant to environmental health sciences (e.g., human, animal, and mechanistic studies). Methods: The Office of Health Assessment and Translation (OHAT) adapted guidance from authorities on systematic-review and sought advice during development of the OHAT Approach through consultation with technical experts in systematic review and human health assessments, as well as scientific advisory groups and the public. The method was refined by considering expert and public comments and through application to case studies. Results and Discussion: Here we present a seven-step framework for systematic review and evidence integration for reaching hazard identification conclusions: 1) problem formulation and protocol development, 2) search for and select studies for inclusion, 3) extract data from studies, 4) assess the quality or risk of bias of individual studies, 5) rate the confidence in the body of evidence, 6) translate the confidence ratings into levels of evidence, and 7) integrate the information from different evidence streams (human, animal, and “other relevant data” including mechanistic or in vitro studies) to develop hazard identification conclusions. Conclusion: The principles of systematic review can be successfully applied to environmental health questions to provide greater objectivity and transparency to the process of developing conclusions. Citation: Rooney AA, Boyles AL, Wolfe MS, Bucher JR, Thayer KA. 2014. Systematic review and evidence integration for literature-based environmental health science assessments. Environ Health Perspect 122:711–718; http://dx.doi.org/10.1289/ehp.1307972

                Author and article information

                61 (0)286271881 , Lisa.bero@sydney.edu.au
                Syst Rev
                Syst Rev
                Systematic Reviews
                BioMed Central (London )
                21 December 2018
                21 December 2018
                : 7
                : 242
                [1 ]ISNI 0000 0004 1936 834X, GRID grid.1013.3, Charles Perkins Centre and School of Pharmacy, Faculty of Medicine and Health, , The University of Sydney, ; D17, The Hub, 6th floor, Sydney, New South Wales 2006 Australia
                [2 ]ISNI 0000 0004 1936 834X, GRID grid.1013.3, School of Medical Sciences, Faculty of Medicine and Health, , The University of Sydney, ; Sydney, Australia
                [3 ]ISNI 0000 0004 0643 4678, GRID grid.431143.0, National Health and Medical Research Council, ; Canberra, Australia
                [4 ]ISNI 0000 0001 2297 6811, GRID grid.266102.1, Department of Ob/Gyn & the Institute for Health Policy Studies, , University of California, ; San Francisco, USA
                [5 ]ISNI 0000 0001 0728 3670, GRID grid.253557.3, Department of Health Sciences, , California State University, East Bay, ; San Francisco, USA
                [6 ]ISNI 0000 0001 2297 6811, GRID grid.266102.1, School of Pharmacy, , University of California, ; San Francisco, USA
                [7 ]ISNI 0000 0004 1936 834X, GRID grid.1013.3, Charles Perkins Centre, , The University of Sydney, ; Sydney, Australia
                [8 ]ISNI 0000 0004 1936 834X, GRID grid.1013.3, School of Pharmacy, Faculty of Medicine and Health and Charles Perkins Centre, , The University of Sydney, ; Sydney, Australia
                Author information
                © The Author(s). 2018

                Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

                : 9 May 2018
                : 9 December 2018
                Funded by: national health and medical research council
                Award ID: APP1139997
                Award Recipient :
                Custom metadata
                © The Author(s) 2018

                Public health
                systematic review,risk of bias,quality assessment,public health guidelines,guidelines,grade,cochrane,nutrition,environment,observational study


                Comment on this article