+1 Recommend
0 collections
      • Record: found
      • Abstract: found
      • Article: not found

      Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias

      Read this article at

          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.



          The increased use of meta-analysis in systematic reviews of healthcare interventions has highlighted several types of bias that can arise during the completion of a randomised controlled trial. Study publication bias has been recognised as a potential threat to the validity of meta-analysis and can make the readily available evidence unreliable for decision making. Until recently, outcome reporting bias has received less attention.

          Methodology/Principal Findings

          We review and summarise the evidence from a series of cohort studies that have assessed study publication bias and outcome reporting bias in randomised controlled trials. Sixteen studies were eligible of which only two followed the cohort all the way through from protocol approval to information regarding publication of outcomes. Eleven of the studies investigated study publication bias and five investigated outcome reporting bias. Three studies have found that statistically significant outcomes had a higher odds of being fully reported compared to non-significant outcomes (range of odds ratios: 2.2 to 4.7). In comparing trial publications to protocols, we found that 40–62% of studies had at least one primary outcome that was changed, introduced, or omitted. We decided not to undertake meta-analysis due to the differences between studies.


          Recent work provides direct empirical evidence for the existence of study publication bias and outcome reporting bias. There is strong evidence of an association between significant results and publication; studies that report positive or significant results are more likely to be published and outcomes that are statistically significant have higher odds of being fully reported. Publications have been found to be inconsistent with their protocols. Researchers need to be aware of the problems of both types of bias and efforts should be concentrated on improving the reporting of trials.

          Related collections

          Most cited references 37

          • Record: found
          • Abstract: found
          • Article: not found

          Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors.

          To examine the extent and nature of outcome reporting bias in a broad cohort of published randomised trials. Retrospective review of publications and follow up survey of authors. Cohort All journal articles of randomised trials indexed in PubMed whose primary publication appeared in December 2000. Prevalence of incompletely reported outcomes per trial; reasons for not reporting outcomes; association between completeness of reporting and statistical significance. 519 trials with 553 publications and 10,557 outcomes were identified. Survey responders (response rate 69%) provided information on unreported outcomes but were often unreliable--for 32% of those who denied the existence of such outcomes there was evidence to the contrary in their publications. On average, over 20% of the outcomes measured in a parallel group trial were incompletely reported. Within a trial, such outcomes had a higher odds of being statistically non-significant compared with fully reported outcomes (odds ratio 2.0 (95% confidence interval 1.6 to 2.7) for efficacy outcomes; 1.9 (1.1 to 3.5) for harm outcomes). The most commonly reported reasons for omitting efficacy outcomes included space constraints, lack of clinical importance, and lack of statistical significance. Incomplete reporting of outcomes within published articles of randomised trials is common and is associated with statistical non-significance. The medical literature therefore represents a selective and biased subset of study outcomes, and trial protocols should be made publicly available.
            • Record: found
            • Abstract: found
            • Article: not found

            Evidence b(i)ased medicine--selective reporting from studies sponsored by pharmaceutical industry: review of studies in new drug applications.

            To investigate the relative impact on publication bias caused by multiple publication, selective publication, and selective reporting in studies sponsored by pharmaceutical companies. 42 placebo controlled studies of five selective serotonin reuptake inhibitors submitted to the Swedish drug regulatory authority as a basis for marketing approval for treating major depression were compared with the studies actually published (between 1983 and 1999). Multiple publication: 21 studies contributed to at least two publications each, and three studies contributed to five publications. Selective publication: studies showing significant effects of drug were published as stand alone publications more often than studies with non-significant results. Selective reporting: many publications ignored the results of intention to treat analyses and reported the more favourable per protocol analyses only. The degree of multiple publication, selective publication, and selective reporting differed between products. Thus, any attempt to recommend a specific selective serotonin reuptake inhibitor from the publicly available data only is likely to be based on biased evidence.
              • Record: found
              • Abstract: found
              • Article: not found

              Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research.

              The reporting of outcomes within published randomized trials has previously been shown to be incomplete, biased and inconsistent with study protocols. We sought to determine whether outcome reporting bias would be present in a cohort of government-funded trials subjected to rigorous peer review. We compared protocols for randomized trials approved for funding by the Canadian Institutes of Health Research (formerly the Medical Research Council of Canada) from 1990 to 1998 with subsequent reports of the trials identified in journal publications. Characteristics of reported and unreported outcomes were recorded from the protocols and publications. Incompletely reported outcomes were defined as those with insufficient data provided in publications for inclusion in meta-analyses. An overall odds ratio measuring the association between completeness of reporting and statistical significance was calculated stratified by trial. Finally, primary outcomes specified in trial protocols were compared with those reported in publications. We identified 48 trials with 68 publications and 1402 outcomes. The median number of participants per trial was 299, and 44% of the trials were published in general medical journals. A median of 31% (10th-90th percentile range 5%-67%) of outcomes measured to assess the efficacy of an intervention (efficacy outcomes) and 59% (0%-100%) of those measured to assess the harm of an intervention (harm outcomes) per trial were incompletely reported. Statistically significant efficacy outcomes had a higher odds than nonsignificant efficacy outcomes of being fully reported (odds ratio 2.7; 95% confidence interval 1.5-5.0). Primary outcomes differed between protocols and publications for 40% of the trials. Selective reporting of outcomes frequently occurs in publications of high-quality government-funded trials.

                Author and article information

                Role: Editor
                PLoS ONE
                PLoS ONE
                Public Library of Science (San Francisco, USA )
                28 August 2008
                : 3
                : 8
                [1 ]Centre for Medical Statistics and Health Evaluation, University of Liverpool, Liverpool, United Kingdom
                [2 ]Centre for Statistics in Medicine, University of Oxford, Oxford, United Kingdom
                [3 ]Clinical Pharmacology Unit, UASP Hospital Clínic, Barcelona, Spain
                [4 ]Moorfields Eye Hospital, London, United Kingdom
                [5 ]Randomized Controlled Trials Unit, Canadian Institutes of Health Research, Ottawa, Canada
                [6 ]Healthier Communities/Public Health, Greenwich Council, London, England
                [7 ]Clinical Epidemiology Unit, DIM-Hospices Civils de Lyon, Lyon, France
                [8 ]Department of HIV/GUM, King's College London, London, United Kingdom
                [9 ]Institute of Social and Preventive Medicine, University of Bern, Bern, Switzerland
                [10 ]German Cochrane Centre, Department of Medical Biometry and Medical Informatics, University Medical Centre Freiburg, Freiburg, Germany
                [11 ]NHMRC Clinical Trials Centre, Camperdown, Australia
                [12 ]Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece
                [13 ]Institute for Clinical Research and Health Policy Studies, Department of Medicine, Tufts Medical Center, Tufts University School of Medicine, Boston, Massachusetts, United States of America
                [14 ]National Health and Medical Research Council (NHMRC) Clinical Trials Centre, University of Sydney, Sydney, Australia
                Medical Research Council South Africa, South Africa
                Author notes

                Conceived and designed the experiments: KD DGA CG PW. Performed the experiments: KD. Analyzed the data: KD. Contributed reagents/materials/analysis tools: JAA JB AWC EC ED PE EvE CG DG JPAI JS PW. Wrote the paper: KD DGA. Performed the literature search and data extraction, contacted all authors of included studies and wrote the first draft: KD. Contributed to the development of the protocol: KD DGA CG PW. Provided comments on the manuscript: DGA ED EvE CG JPAI JS PW. Gave permission to include data from their study in the review, responded to queries and provided extra information when needed and available: JAA JB AWC EC ED PE EvE DG JPAI.

                Dwan et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
                Page count
                Pages: 31
                Research Article
                Evidence-Based Healthcare
                Evidence-Based Healthcare/Clinical Decision-Making
                Evidence-Based Healthcare/Statistical Methodologies and Health Informatics



                Comment on this article