Comparison of the Accuracy, Completeness, Reproducibility, and Consistency of Different AI Chatbots in Providing Nutritional Advice: An Exploratory Study

There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

Abstract

Background: The use of artificial intelligence (AI) chatbots for obtaining healthcare advice is greatly increased in the general population. This study assessed the performance of general-purpose AI chatbots in giving nutritional advice for patients with obesity with or without multiple comorbidities. Methods: The case of a 35-year-old male with obesity without comorbidities (Case 1), and the case of a 65-year-old female with obesity, type 2 diabetes mellitus, sarcopenia, and chronic kidney disease (Case 2) were submitted to 10 different AI chatbots on three consecutive days. Accuracy (the ability to provide advice aligned with guidelines), completeness, and reproducibility (replicability of the information over the three days) of the chatbots’ responses were evaluated by three registered dietitians. Nutritional consistency was evaluated by comparing the nutrient content provided by the chatbots with values calculated by dietitians. Results: Case 1: ChatGPT 3.5 demonstrated the highest accuracy rate (67.2%) and Copilot the lowest (21.1%). ChatGPT 3.5 and ChatGPT 4.0 achieved the highest completeness (both 87.3%), whereas Gemini and Copilot recorded the lowest scores (55.6%, 42.9%, respectively). Reproducibility was highest for Chatsonic (86.1%) and lowest for ChatGPT 4.0 (50%) and ChatGPT 3.5 (52.8%). Case 2: Overall accuracy was low, with no chatbot achieving 50% accuracy. Completeness was highest for ChatGPT 4.0 and Claude (both 77.8%), and lowest for Copilot (23.3%). ChatGPT 4.0 and Pi Ai showed the lowest reproducibility. Major inconsistencies regarded the amount of protein recommended by most chatbots, which suggested simultaneously to both reduce and increase protein intake. Conclusions: General-purpose AI chatbots exhibited limited accuracy, reproducibility, and consistency in giving dietary advice in complex clinical scenarios and cannot replace the work of an expert dietitian.

Related collections

Most cited references 30

Record: found
Abstract: found
Article: found

Is Open Access

ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns

Malik Sallam (2023)

ChatGPT is an artificial intelligence (AI)-based conversational large language model (LLM). The potential applications of LLMs in health care education, research, and practice could be promising if the associated valid concerns are proactively examined and addressed. The current systematic review aimed to investigate the utility of ChatGPT in health care education, research, and practice and to highlight its potential limitations. Using the PRIMSA guidelines, a systematic search was conducted to retrieve English records in PubMed/MEDLINE and Google Scholar (published research or preprints) that examined ChatGPT in the context of health care education, research, or practice. A total of 60 records were eligible for inclusion. Benefits of ChatGPT were cited in 51/60 (85.0%) records and included: (1) improved scientific writing and enhancing research equity and versatility; (2) utility in health care research (efficient analysis of datasets, code generation, literature reviews, saving time to focus on experimental design, and drug discovery and development); (3) benefits in health care practice (streamlining the workflow, cost saving, documentation, personalized medicine, and improved health literacy); and (4) benefits in health care education including improved personalized learning and the focus on critical thinking and problem-based learning. Concerns regarding ChatGPT use were stated in 58/60 (96.7%) records including ethical, copyright, transparency, and legal issues, the risk of bias, plagiarism, lack of originality, inaccurate content with risk of hallucination, limited knowledge, incorrect citations, cybersecurity issues, and risk of infodemics. The promising applications of ChatGPT can induce paradigm shifts in health care education, research, and practice. However, the embrace of this AI chatbot should be conducted with extreme caution considering its potential limitations. As it currently stands, ChatGPT does not qualify to be listed as an author in scientific articles unless the ICMJE/COPE guidelines are revised or amended. An initiative involving all stakeholders in health care education, research, and practice is urgently needed. This will help to set a code of ethics to guide the responsible use of ChatGPT among other LLMs in health care and academia.

0 comments Cited 719 times – based on 0 reviews      Review now

Record: found
Abstract: not found
Article: not found

Large language models in medicine

Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan … (2023)

0 comments Cited 613 times – based on 0 reviews      Review now

Record: found
Abstract: found
Article: not found

KDOQI Clinical Practice Guideline for Nutrition in CKD: 2020 Update

T Alp Ikizler, Jerrilynn Burrowes, Laura D Byham-Gray … (2020)

The National Kidney Foundation's Kidney Disease Outcomes Quality Initiative (KDOQI) has provided evidence-based guidelines for nutrition in kidney diseases since 1999. Since the publication of the first KDOQI nutrition guideline, there has been a great accumulation of new evidence regarding the management of nutritional aspects of kidney disease and sophistication in the guidelines process. The 2020 update to the KDOQI Clinical Practice Guideline for Nutrition in CKD was developed as a joint effort with the Academy of Nutrition and Dietetics (Academy). It provides comprehensive up-to-date information on the understanding and care of patients with chronic kidney disease (CKD), especially in terms of their metabolic and nutritional milieu for the practicing clinician and allied health care workers. The guideline was expanded to include not only patients with end-stage kidney disease or advanced CKD, but also patients with stages 1-5 CKD who are not receiving dialysis and patients with a functional kidney transplant. The updated guideline statements focus on 6 primary areas: nutritional assessment, medical nutrition therapy (MNT), dietary protein and energy intake, nutritional supplementation, micronutrients, and electrolytes. The guidelines primarily cover dietary management rather than all possible nutritional interventions. The evidence data and guideline statements were evaluated using Grading of Recommendations, Assessment, Development and Evaluation (GRADE) criteria. As applicable, each guideline statement is accompanied by rationale/background information, a detailed justification, monitoring and evaluation guidance, implementation considerations, special discussions, and recommendations for future research.