How well do different chatbots respond to multiple myeloma treatment guidelines?

Abstract

Recent publications in Leukemia offer further affirmation that daratumumab is a cornerstone of first-line therapy for transplant-ineligible patients with newly diagnosed multiple myeloma (NDMM) [1, 2]. These articles are a subgroup analysis and long-term follow-up of the MAIA trial. This raises the question of whether the advent of new therapies in multiple myeloma and the substantial amount of information now available, will enable the chatbots most commonly used to provide accurate answers aligned with management guidelines. There is a significant risk that these models may produce hallucinations (incorrect or confusing outputs), amplify biases and misinformation, and exhibit deficiencies in reasoning abilities [3]. Therefore, we wanted to evaluate artificial intelligence (AI)-based chatbots as tools to aid in understanding and using guidelines for treating NDMM.

Templates were used to create different diagnosis descriptions for NDMM, standard/high-risk transplant candidates, or non-candidates. Prompts were input to the GPT-4o model via the ChatGPT (OpenAI) interface, Gemini 1.5 Flash and Gemini 1.5 Pro (Google), Copilot (Microsoft), OpenEvidence, and Claude 3.5 Sonnet. The outputs of the chatbots were evaluated against two reliable sources: the 2021 National Comprehensive Cancer Network (NCCN) guidelines and the 2021 guidelines from the Spanish Myeloma Group on multiple myeloma [4, 5]. This was done due to the variability of the chatbot’s cutoff dates at the start of the study. Questions related to Spanish guidelines were asked in both Spanish and English. Feedback with treatment options, clinical trials, and management strategies for multiple myeloma or examples of correct answers was not permitted. Concordance was defined as the level of consistency of chatbot results with treatment guidelines and recommendations. For this, we used a score from 0 to 2, where zero meant the chatbot did not agree with the guidelines, one meant it was partly right, and two meant it was correct. Three certified hematologists and one hematology trainee analyzed the concordance of the chatbot outputs with both guidelines. Agreement between each evaluator’s responses was also assessed. Reliability was also evaluated with Kendall’s W (ranges between 0 and 1, and values close to 1 indicate a strong association; values close to 0 indicate a weak or null association). Institutional review board approval was not needed since human participants were not involved. Data were analyzed between November 1, 2024, and January 31, 2025, using Google application spreadsheets and Excel (version 16.74) and SPSS (version 25).