2
views
0
recommends
+1 Recommend
0 collections
    0
    shares
      • Record: found
      • Abstract: found
      • Article: found
      Is Open Access

      Claims of ‘no difference’ or ‘no effect’ in Cochrane and other systematic reviews

      systematic-review

      Read this article at

      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          Estimates of treatment effects/differences derived from controlled comparisons are subject to uncertainty, both because of the quality of the data and the play of chance. Despite this, authors sometimes use statistical significance testing to make definitive statements that ‘no difference exists between’ treatments. A survey to assess abstracts of Cochrane reviews published in 2001/2002 identified unqualified claims of ‘no difference’ or ‘no effect’ in 259 (21.3%) out of 1212 review abstracts surveyed. We have repeated the survey to assess the frequency of such claims among the abstracts of Cochrane and other systematic reviews published in 2017. We surveyed the 643 Cochrane review abstracts published in 2017 and a random sample of 643 abstracts of other systematic reviews published in the same year. We excluded review abstracts that referred only to a protocol, lacked a conclusion or did not contain any relevant information. We took steps to reduce biases during our survey. 'No difference/no effect' was claimed in the abstracts of 36 (7.8%) of 460 Cochrane reviews and in the abstracts of 13 (6.0%) of 218 other systematic reviews. Incorrect claims of no difference/no effect of treatments were substantially less common in Cochrane reviews published in in 2017 than they were in abstracts of reviews published in 2001/2002. We hope that this reflects greater efforts to reduce biases and inconsistent judgements in the later survey as well as more careful wording of review abstracts. There are numerous other ways of wording treatment claims incorrectly. These must be addressed because they can have adverse effects on healthcare and health research.

          Related collections

          Most cited references4

          • Record: found
          • Abstract: not found
          • Article: not found

          Survey of claims of no effect in abstracts of Cochrane reviews.

            Bookmark
            • Record: found
            • Abstract: found
            • Article: found
            Is Open Access

            Reporting and interpretation of results from clinical trials that did not claim a treatment difference: survey of four general medical journals

            Objectives To describe and summarise how the results of randomised controlled trials (RCTs) that did not find a significant treatment effect are reported, and to estimate how commonly trial reports make unwarranted claims. Design We performed a retrospective survey of published RCTs, published in four high impact factor general medical journals between June 2016 and June 2017. Setting Trials conducted in all settings were included. Participants 94 reports of RCTs that did not find a difference in their main comparison or comparisons were included. Interventions All interventions. Primary and secondary outcomes We recorded the way the results of each trial for its primary outcome or outcomes were described in Results and Conclusions sections of the Abstract, using a 10-category classification. Other outcomes were whether confidence intervals (CIs) and p values were presented for the main treatment comparisons, and whether the results and conclusions referred to measures of uncertainty. We estimated the proportion of papers that made claims that were not justified by the results, or were open to multiple interpretations. Results 94 trial reports (120 treatment comparisons) were included. In Results sections, for 58/120 comparisons (48.3%) the results of the study were re-stated, without interpretation, and 38/120 (31.7%) stated that there was no statistically significant difference. In Conclusions, 65/120 treatment comparisons (54.2%) stated that there was no treatment benefit, 14/120 (11.7%) that there was no significant benefit and 16/120 (13.3%) that there was no significant difference. CIs and p values were both presented by 84% of studies (79/94), but only 3/94 studies referred to uncertainty when drawing conclusions. Conclusions The majority of trials (54.2%) inappropriately interpreted a result that was not statistically significant as indicating no treatment benefit. Very few studies interpreted the result as indicating a lack of evidence against the null hypothesis of zero difference between the trial arms.
              Bookmark
              • Record: found
              • Abstract: found
              • Article: found
              Is Open Access

              RevManHAL: towards automatic text generation in systematic reviews

              Background Systematic reviews are a key part of healthcare evaluation. They involve important painstaking but repetitive work. A major producer of systematic reviews, the Cochrane Collaboration, employs Review Manager (RevMan) programme—a software which assists reviewers and produces XML-structured files. This paper describes an add-on programme (RevManHAL) which helps auto-generate the abstract, results and discussion sections of RevMan-generated reviews in multiple languages. The paper also describes future developments for RevManHAL. Methods RevManHAL was created in Java using NetBeans by a programmer working full time for 2 months. Results The resulting open-source programme uses editable phrase banks to envelop text/numbers from within the prepared RevMan file in formatted readable text of a chosen language. In this way, considerable parts of the review’s ‘abstract’, ‘results’ and ‘discussion’ sections are created and a phrase added to ‘acknowledgements’. Conclusion RevManHAL’s output needs to be checked by reviewers, but already, from our experience within the Cochrane Schizophrenia Group (200 maintained reviews, 900 reviewers), RevManHAL has saved much time which is better employed thinking about the meaning of the data rather than restating them. Many more functions will become possible as review writing becomes increasingly automated.
                Bookmark

                Author and article information

                Journal
                BMJ Evid Based Med
                BMJ Evid Based Med
                ebmed
                ebm
                BMJ Evidence-Based Medicine
                BMJ Publishing Group (BMA House, Tavistock Square, London, WC1H 9JR )
                2515-446X
                2515-4478
                June 2021
                7 January 2020
                : 26
                : 3
                : 118-120
                Affiliations
                [1 ] departmentScience Policy Research Unit , University of Sussex , Brighton, Brighton and Hove, UK
                [2 ] UK Cochrane Centre , Oxford, Oxfordshire, UK
                [3 ] departmentSchool of Medicine , University of Nottingham , Nottingham, Nottinghamshire, UK
                [4 ] departmentCentre for Evidenced-Based Medicine , University of Oxford , Oxford, Oxfordshire, UK
                Author notes
                [Correspondence to ] Phoebe Rose Marson Smith, Science Policy Research Unit, University of Sussex, Brighton BN1 9RH, UK; phoebemarsonsmith@ 123456gmail.com
                Author information
                http://orcid.org/0000-0002-3188-3884
                Article
                bmjebm-2019-111257
                10.1136/bmjebm-2019-111257
                8165142
                31911421
                3aee4e83-997a-471d-a90b-2db7fceb6ef2
                © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

                This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

                History
                : 17 December 2019
                Categories
                Research Methods and Reporting
                1506
                Custom metadata
                unlocked

                public health,health policy
                public health, health policy

                Comments

                Comment on this article