Epidemiology and Reporting Characteristics of Systematic Reviews of Biomedical Research: A Cross-Sectional Study

Abstract

Background

Systematic reviews (SRs) can help decision makers interpret the deluge of published biomedical literature. However, a SR may be of limited use if the methods used to conduct the SR are flawed, and reporting of the SR is incomplete. To our knowledge, since 2004 there has been no cross-sectional study of the prevalence, focus, and completeness of reporting of SRs across different specialties. Therefore, the aim of our study was to investigate the epidemiological and reporting characteristics of a more recent cross-section of SRs.

Methods and Findings

We searched MEDLINE to identify potentially eligible SRs indexed during the month of February 2014. Citations were screened using prespecified eligibility criteria. Epidemiological and reporting characteristics of a random sample of 300 SRs were extracted by one reviewer, with a 10% sample extracted in duplicate. We compared characteristics of Cochrane versus non-Cochrane reviews, and the 2014 sample of SRs versus a 2004 sample of SRs. We identified 682 SRs, suggesting that more than 8,000 SRs are being indexed in MEDLINE annually, corresponding to a 3-fold increase over the last decade. The majority of SRs addressed a therapeutic question and were conducted by authors based in China, the UK, or the US; they included a median of 15 studies involving 2,072 participants. Meta-analysis was performed in 63% of SRs, mostly using standard pairwise methods. Study risk of bias/quality assessment was performed in 70% of SRs but was rarely incorporated into the analysis (16%). Few SRs (7%) searched sources of unpublished data, and the risk of publication bias was considered in less than half of SRs. Reporting quality was highly variable; at least a third of SRs did not report use of a SR protocol, eligibility criteria relating to publication status, years of coverage of the search, a full Boolean search logic for at least one database, methods for data extraction, methods for study risk of bias assessment, a primary outcome, an abstract conclusion that incorporated study limitations, or the funding source of the SR. Cochrane SRs, which accounted for 15% of the sample, had more complete reporting than all other types of SRs. Reporting has generally improved since 2004, but remains suboptimal for many characteristics.

Conclusions

An increasing number of SRs are being published, and many are poorly conducted and reported. Strategies are needed to help reduce this avoidable waste in research.

Abstract

In a cross-sectional manuscript analysis, David Moher and colleagues score the prevalence, quality of conduct and completeness of reporting among systematic reviews published across medical disciplines in 2014.

Author Summary

Why Was This Study Done?

• Decisions in health care, such as which treatment to recommend or which test to order, should be based on evidence from all available research studies, rather than the results of the largest or most recent study.

• Systematic reviews, which explicitly use methods to identify, select, critically appraise, and synthesize the results of all existing studies of a given question, are considered the highest level of evidence for decision makers.

• We wanted to know how many systematic reviews of biomedical research are being published, what questions they are addressing, and how well the methods are reported, since information of this sort has not been collected since 2004.

What Did the Researchers Do and Find?

• We looked for all systematic reviews added to the main bibliographic database for biomedical literature during one month (February 2014), and recorded the characteristics of these reviews.

• We found 682 systematic reviews—a 3-fold increase over the last decade—that addressed a wide range of topics.

• In many cases, important aspects of the methods used were not reported (for example, at least a third of the reviews did not report how they searched for studies or how they assessed the quality of the included studies), unpublished data was rarely sought, and at least a third of the reviews used statistical methods discouraged by leading organizations that have developed guidance for systematic reviews (for example, Cochrane and the Institute of Medicine).

What Do These Findings Mean?

• We conclude that systematic reviews have become increasingly popular and that, similar to a decade ago, the quality of conduct and reporting varies widely; therefore, readers should not accept the findings of systematic reviews uncritically.

• We recommend a number of strategies to improve the value of systematic reviews, such as the development of software to facilitate better reporting, certified training for journal editors in how to implement the use of reporting guidelines such as PRISMA ( http://prisma-statement.org/), and formal training of biomedical researchers in research design and analysis.