+1 Recommend
0 collections
      • Record: found
      • Abstract: found
      • Article: found
      Is Open Access

      The art and science of study identification: a comparative analysis of two systematic reviews


      Read this article at

          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.



          Systematic reviews (SRs) form the foundation for guidelines and evidence-based policy in medicine and public health. Although similar systematic reviews may include non-identical sets of studies, and it is recognized that different sets of studies may lead to different conclusions, little work has been published on why SR study cohorts differ.


          We took advantage of concurrent publication of two SRs on the same topic – prevention of child exposure to tobacco smoke - to understand why study cohorts differed in the two reviews. We identified all studies included in just one review, investigated validity of specified reasons for exclusions, and, using database records, explored reasons for study non-identification. We assessed review methods and discordancy, and attempted to assess whether changes in study cohorts would have changed conclusions.


          Sixty-one studies were included in the two reviews. Thirty-five studies were present in just one review; of these, twenty were identified and excluded by the parallel review.

          Omissions were due to: review scope (9 studies, 26 %), outcomes of interest not measured (8 studies, 23 %), exclusion of reports with inadequate reporting (6 studies, 17 %), mixed or unclear reasons (3 studies, 8 %), search strategies concerning filters, tagging, and keywords (3 studies, 8 %), search strategies regarding sources (PUBMED not searched) (2 studies, 6 %); discordant interpretation of same eligibility criteria (2 studies, 6 %), and non-identification due to non-specific study topic (2 studies, 6 %). Review conclusions differed, but were likely due to differences in synthesis methods, not differences in study cohorts.


          The process of study identification for SRs is part art and part science. While some differences are due to differences in review scope, outcomes measured, or reporting practices, others are caused by search methods or discrepancies in reviewer interpretations. Different study cohorts may or may not be a cause of differing SR results. Completeness of SR study cohorts could be enhanced by 1 – independent identification of studies by at least two reviewers, as recommended by recent guidelines, 2 - searching PUBMED with free-text keywords in addition to MEDLINE to identify recent studies, and 3 - Using validated search filters.

          Electronic supplementary material

          The online version of this article (doi:10.1186/s12874-016-0118-2) contains supplementary material, which is available to authorized users.

          Related collections

          Most cited references51

          • Record: found
          • Abstract: found
          • Article: not found

          Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses.

          The Quality of Reporting of Meta-analyses (QUOROM) conference was convened to address standards for improving the quality of reporting of meta-analyses of clinical randomised controlled trials (RCTs). The QUOROM group consisted of 30 clinical epidemiologists, clinicians, statisticians, editors, and researchers. In conference, the group was asked to identify items they thought should be included in a checklist of standards. Whenever possible, checklist items were guided by research evidence suggesting that failure to adhere to the item proposed could lead to biased results. A modified Delphi technique was used in assessing candidate items. The conference resulted in the QUOROM statement, a checklist, and a flow diagram. The checklist describes our preferred way to present the abstract, introduction, methods, results, and discussion sections of a report of a meta-analysis. It is organised into 21 headings and subheadings regarding searches, selection, validity assessment, data abstraction, study characteristics, and quantitative data synthesis, and in the results with "trial flow", study characteristics, and quantitative data synthesis; research documentation was identified for eight of the 18 items. The flow diagram provides information about both the numbers of RCTs identified, included, and excluded and the reasons for exclusion of trials. We hope this report will generate further thought about ways to improve the quality of reports of meta-analyses of RCTs and that interested readers, reviewers, researchers, and editors will use the QUOROM statement and generate ideas for its improvement.
            • Record: found
            • Abstract: found
            • Article: not found

            Rationale for systematic reviews.

            C D Mulrow (1994)
            Systematic literature reviews including meta-analyses are invaluable scientific activities. The rationale for such reviews is well established. Health care providers, researchers, and policy makers are inundated with unmanageable amounts of information; they need systematic reviews to efficiently integrate existing information and provide data for rational decision making. Systematic reviews establish whether scientific findings are consistent and can be generalised across populations, settings, and treatment variations, or whether findings vary significantly by particular subsets. Meta-analyses in particular can increase power and precision of estimates of treatment effects and exposure risks. Finally, explicit methods used in systematic reviews limit bias and, hopefully, will improve reliability and accuracy of conclusions.
              • Record: found
              • Abstract: found
              • Article: not found

              A guide to interpreting discordant systematic reviews.

              Systematic reviews are becoming prominent tools to guide health care decisions. As the number of published systematic reviews increases, it is common to find more than 1 systematic review addressing the same or a very similar therapeutic question. Despite the promise for systematic reviews to resolve conflicting results of primary studies, conflicts among reviews are now emerging. Such conflicts produce difficulties for decision-makers (including clinicians, policy-makers, researchers and patients) who rely on these reviews to help them make choices among alternative interventions when experts and the results of trials disagree. The authors provide an adjunct decision tool--a decision algorithm--to help decision-makers select from among discordant reviews.

                Author and article information

                972-50-875-1502 , rosenl@post.tau.ac.il
                BMC Med Res Methodol
                BMC Med Res Methodol
                BMC Medical Research Methodology
                BioMed Central (London )
                24 February 2016
                24 February 2016
                : 16
                : 24
                [ ]Deparment of Health Promotion, School of Public Health, Sackler Faculty of Medicine, Tel Aviv University, Ramat Aviv, 69978 Israel
                [ ]Gitter-Smolarz Library of Life Sciences and Medicine, Tel Aviv University, Ramat Aviv, 69978 Israel
                © Rosen and Suhami. 2016

                Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

                : 8 October 2015
                : 2 February 2016
                Funded by: FundRef http://dx.doi.org/10.13039/100005640, Flight Attendant Medical Research Institute;
                Award ID: 072086_YCSA
                Award Recipient :
                Research Article
                Custom metadata
                © The Author(s) 2016

                systematic reviews,meta-analysis,evidence-based decision making,electronic searching,tobacco smoke exposure,tobacco control


                Comment on this article