+1 Recommend
0 collections
      • Record: found
      • Abstract: found
      • Article: not found

      Towards automatic recognition of scientifically rigorous clinical research evidence.

      Journal of the American Medical Informatics Association : JAMIA
      Artificial Intelligence, Bayes Theorem, Biomedical Research, standards, Evidence-Based Practice, Information Storage and Retrieval, methods, MEDLINE, ROC Curve

      Read this article at

          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.


          The growing numbers of topically relevant biomedical publications readily available due to advances in document retrieval methods pose a challenge to clinicians practicing evidence-based medicine. It is increasingly time consuming to acquire and critically appraise the available evidence. This problem could be addressed in part if methods were available to automatically recognize rigorous studies immediately applicable in a specific clinical situation. We approach the problem of recognizing studies containing useable clinical advice from retrieved topically relevant articles as a binary classification problem. The gold standard used in the development of PubMed clinical query filters forms the basis of our approach. We identify scientifically rigorous studies using supervised machine learning techniques (Naïve Bayes, support vector machine (SVM), and boosting) trained on high-level semantic features. We combine these methods using an ensemble learning method (stacking). The performance of learning methods is evaluated using precision, recall and F(1) score, in addition to area under the receiver operating characteristic (ROC) curve (AUC). Using a training set of 10,000 manually annotated MEDLINE citations, and a test set of an additional 2,000 citations, we achieve 73.7% precision and 61.5% recall in identifying rigorous, clinically relevant studies, with stacking over five feature-classifier combinations and 82.5% precision and 84.3% recall in recognizing rigorous studies with treatment focus using stacking over word + metadata feature vector. Our results demonstrate that a high quality gold standard and advanced classification methods can help clinicians acquire best evidence from the medical literature.

          Related collections

          Author and article information


          Artificial Intelligence,Bayes Theorem,Biomedical Research,standards,Evidence-Based Practice,Information Storage and Retrieval,methods,MEDLINE,ROC Curve


          Comment on this article