0
views
0
recommends
+1 Recommend
0 collections
    0
    shares
      • Record: found
      • Abstract: found
      • Article: not found

      Local large language models for privacy-preserving accelerated review of historic echocardiogram reports

      Read this article at

      ScienceOpenPublisher
      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          Objectives

          The study developed framework that leverages an open-source Large Language Model (LLM) to enable clinicians to ask plain-language questions about a patient’s entire echocardiogram report history. This approach is intended to streamline the extraction of clinical insights from multiple echocardiogram reports, particularly in patients with complex cardiac diseases, thereby enhancing both patient care and research efficiency.

          Materials and Methods

          Data from over 10 years were collected, comprising echocardiogram reports from patients with more than 10 echocardiograms on file at the Mount Sinai Health System. These reports were converted into a single document per patient for analysis, broken down into snippets and relevant snippets were retrieved using text similarity measures. The LLaMA-2 70B model was employed for analyzing the text using a specially crafted prompt. The model’s performance was evaluated against ground-truth answers created by faculty cardiologists.

          Results

          The study analyzed 432 reports from 37 patients for a total of 100 question-answer pairs. The LLM correctly answered 90% questions, with accuracies of 83% for temporality, 93% for severity assessment, 84% for intervention identification, and 100% for diagnosis retrieval. Errors mainly stemmed from the LLM’s inherent limitations, such as misinterpreting numbers or hallucinations.

          Conclusion

          The study demonstrates the feasibility and effectiveness of using a local, open-source LLM for querying and interpreting echocardiogram report data. This approach offers a significant improvement over traditional keyword-based searches, enabling more contextually relevant and semantically accurate responses; in turn showing promise in enhancing clinical decision-making and research by facilitating more efficient access to complex patient data.

          Related collections

          Most cited references10

          • Record: found
          • Abstract: not found
          • Article: not found

          Jumping NLP Curves: A Review of Natural Language Processing Research [Review Article]

            Bookmark
            • Record: found
            • Abstract: not found
            • Article: not found
            Is Open Access

            ChatGPT: friend or foe?

              Bookmark
              • Record: found
              • Abstract: not found
              • Article: not found

              Cosine similarity measures for intuitionistic fuzzy sets and their applications

              Jun Ye (2011)
                Bookmark

                Author and article information

                Contributors
                (View ORCID Profile)
                (View ORCID Profile)
                (View ORCID Profile)
                Journal
                Journal of the American Medical Informatics Association
                Oxford University Press (OUP)
                1067-5027
                1527-974X
                April 30 2024
                April 30 2024
                Article
                10.1093/jamia/ocae085
                8c4506e9-6483-456c-8d4d-50dc4b33e020
                © 2024

                https://academic.oup.com/pages/standard-publication-reuse-rights

                History

                Comments

                Comment on this article