Assessing the Quality of Patient Education Materials on Cardiac Catheterization From Artificial Intelligence Chatbots: An Observational Cross-Sectional Study

Abstract

Background

Health literacy empowers patients to participate in their own healthcare. Personal health literacy is one’s ability to find, understand, and use information/resources to make well-informed health decisions. Artificial intelligence (AI) has become a source for the acquisition of health-related information through large language model (LLM)-driven chatbots. Assessment of the readability and quality of health information produced by these chatbots has been the subject of numerous studies to date. This study seeks to assess the quality of patient education materials on cardiac catheterization produced by AI chatbots.

Methodology

We asked a set of 10 questions about cardiac catheterization to four chatbots: ChatGPT (OpenAI, San Francisco, CA), Microsoft Copilot (Microsoft Corporation, Redmond, WA), Google Gemini (Google DeepMind, London, UK), and Meta AI (Meta, New York, NY). The questions and subsequent answers were utilized to make patient education materials on cardiac catheterization. The quality of these materials was assessed using two validated instruments for patient education materials: DISCERN and the Patient Education Materials Assessment Tool (PEMAT).

Results

The overall DISCERN scores were 4.5 for ChatGPT, 4.4 for Microsoft Copilot and Google Gemini, and 3.8 for Meta AI. ChatGPT, Microsoft Copilot, and Google Gemini tied for the highest reliability score at 4.6, while Meta AI had the lowest with 4.2. ChatGPT had the highest quality score at 4.4, while Meta AI had the lowest with 3.4. ChatGPT and Google Gemini had Understandability scores of 100%, while Meta AI had the lowest with 82%. ChatGPT, Microsoft Copilot, and Google Gemini all had Actionability scores of 75%, while Meta AI had one of 50%.

Conclusions

ChatGPT produced the most reliable and highest quality materials, followed closely by Google Gemini. Meta AI produced the lowest quality materials. Given the easy accessibility that chatbots provide patients and the high-quality responses that we obtained, they could be a reliable source for patients to obtain information about cardiac catheterization.