Luis A Hernández-Flores, José B López-Martínez, Jesús J Rosales-de-la-Rosa, Daniel Aillaud-De-Uriarte, Sergio Contreras-Garduño, Rubén Cortés-González
{"title":"Assessment of Challenging Oncologic Cases: A Comparative Analysis Between ChatGPT, Gemini, and a Multidisciplinary Tumor Board.","authors":"Luis A Hernández-Flores, José B López-Martínez, Jesús J Rosales-de-la-Rosa, Daniel Aillaud-De-Uriarte, Sergio Contreras-Garduño, Rubén Cortés-González","doi":"10.1002/jso.28121","DOIUrl":null,"url":null,"abstract":"<p><strong>Introduction: </strong>Since its introduction in 2022, public-access conversational AI, exemplified by ChatGPT and Gemini, has been increasingly utilized in medical decision-making, though its impact is questionable. This study aims to evaluate its efficacy in assessing complex oncologic cases compared to a multidisciplinary tumor board (MTB) comprising experts from various specialties.</p><p><strong>Methods: </strong>A 2-year retrospective analysis was conducted on 98 oncological cases at a reference medical center in Mexico City. A MTB comprising surgical oncologists, medical oncologists, radio-oncologists, pathologists, among others, reviewed and discussed each case to determine management strategies. We evaluated four key decision points, dichotomized as either affirmative or negative: the need for new imaging studies, radiation therapy, chemotherapy, and surgery. Comprehensive medical documentation accompanied each case. We then compared AI's decisions with those of the MTB using the same criteria and conducted a Cohen's Kappa test to assess agreement.</p><p><strong>Results: </strong>Agreement between ChatGPT (4o) and Gemini (1.5 Flash), and the MTB ranged from none to slight for additional imaging studies (Gemini: κ = 0.100, p = 0.087; ChatGPT 4o: κ = 0.024, p = 0.592) and chemotherapy (Gemini: κ = 0.089, p = 0.316; ChatGPT 4o: κ = 0.336, p = 0.001). Moderate agreement was observed for decisions regarding surgery (Gemini: κ = 0.194, p = 0.046; ChatGPT 4o: κ = 0.467, p = < 0.001) and radiotherapy (Gemini: κ = 0.214, p = 0.012; ChatGPT 4o: κ = 0.525, p = < 0.001).</p><p><strong>Conclusions: </strong>Both ChatGPT and Gemini showed moderate agreement with the multidisciplinary tumor board on decisions regarding surgery and radiotherapy. ChatGPT also showed moderate agreement in chemotherapy, but further assessment is needed for other interventions. ChatGPT proved to be superior to Gemini in most key points. The potential of these public access AI in oncology warrants continued exploration to refine its utility in clinical practice.</p>","PeriodicalId":17111,"journal":{"name":"Journal of Surgical Oncology","volume":" ","pages":""},"PeriodicalIF":2.0000,"publicationDate":"2025-02-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Surgical Oncology","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1002/jso.28121","RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"ONCOLOGY","Score":null,"Total":0}
引用次数: 0
Abstract
Introduction: Since its introduction in 2022, public-access conversational AI, exemplified by ChatGPT and Gemini, has been increasingly utilized in medical decision-making, though its impact is questionable. This study aims to evaluate its efficacy in assessing complex oncologic cases compared to a multidisciplinary tumor board (MTB) comprising experts from various specialties.
Methods: A 2-year retrospective analysis was conducted on 98 oncological cases at a reference medical center in Mexico City. A MTB comprising surgical oncologists, medical oncologists, radio-oncologists, pathologists, among others, reviewed and discussed each case to determine management strategies. We evaluated four key decision points, dichotomized as either affirmative or negative: the need for new imaging studies, radiation therapy, chemotherapy, and surgery. Comprehensive medical documentation accompanied each case. We then compared AI's decisions with those of the MTB using the same criteria and conducted a Cohen's Kappa test to assess agreement.
Results: Agreement between ChatGPT (4o) and Gemini (1.5 Flash), and the MTB ranged from none to slight for additional imaging studies (Gemini: κ = 0.100, p = 0.087; ChatGPT 4o: κ = 0.024, p = 0.592) and chemotherapy (Gemini: κ = 0.089, p = 0.316; ChatGPT 4o: κ = 0.336, p = 0.001). Moderate agreement was observed for decisions regarding surgery (Gemini: κ = 0.194, p = 0.046; ChatGPT 4o: κ = 0.467, p = < 0.001) and radiotherapy (Gemini: κ = 0.214, p = 0.012; ChatGPT 4o: κ = 0.525, p = < 0.001).
Conclusions: Both ChatGPT and Gemini showed moderate agreement with the multidisciplinary tumor board on decisions regarding surgery and radiotherapy. ChatGPT also showed moderate agreement in chemotherapy, but further assessment is needed for other interventions. ChatGPT proved to be superior to Gemini in most key points. The potential of these public access AI in oncology warrants continued exploration to refine its utility in clinical practice.
期刊介绍：
The Journal of Surgical Oncology offers peer-reviewed, original papers in the field of surgical oncology and broadly related surgical sciences, including reports on experimental and laboratory studies. As an international journal, the editors encourage participation from leading surgeons around the world. The JSO is the representative journal for the World Federation of Surgical Oncology Societies. Publishing 16 issues in 2 volumes each year, the journal accepts Research Articles, in-depth Reviews of timely interest, Letters to the Editor, and invited Editorials. Guest Editors from the JSO Editorial Board oversee multiple special Seminars issues each year. These Seminars include multifaceted Reviews on a particular topic or current issue in surgical oncology, which are invited from experts in the field.
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号
