Artificial Intelligence and Gynecological Oncology: A Comparative Study of ChatGPT-Omni and Gemini-Pro Across Repeated Intervals with Case Scenario and Open-Ended Queries.
{"title":"Artificial Intelligence and Gynecological Oncology: A Comparative Study of ChatGPT-Omni and Gemini-Pro Across Repeated Intervals with Case Scenario and Open-Ended Queries.","authors":"Seckin Tuna Kaplan","doi":"10.1159/000545231","DOIUrl":null,"url":null,"abstract":"<p><p>supporting clinical decision-making, diagnosis, and treatment. The study aims to compare the performance of ChatGPT-4o (Omni) and Gemini-pro in answering clinical questions and case scenarios related to gynecological oncology and to assess the consistency of their long-term responses.</p><p><strong>Methods: </strong>A two-phase comparative analysis was conducted. 700 clinical questions (350 per model) were developed and categorized into open-ended and case-scenario questions. Three months later, the same set of questions was presented again to evaluate any changes in performance for accuracy, completeness, and guideline adherence.</p><p><strong>Results: </strong>Omni outperformed Gemini-pro across all question types (p=0.001). Omni achieved a mean score of 5.9 for the basic open-ended questions, higher than Gemini, which had 5.1 (p=0.001). It also maintained a clear advantage in complex, open-ended questions, scoring a mean of 5.6 than Gemini AI's 4.2 (p=0.001). Omni scored a mean of 5.7 for basic case scenarios, while Gemini AI lagged with a mean score of 5 (p=0.001). Omni showed a modest improvement in complex, open-ended queries, with an increase of 0.2 points (+3.57%) (p=0.001). Omni provided more accurate and comprehensive responses in guideline adherence than Gemini, particularly in complex cases requiring nuanced judgment and adherence to oncology protocols. Its responses aligned with the latest guidelines, including the American Society of Clinical Oncology and the National Comprehensive Cancer Network.</p><p><strong>Conclusions: </strong>Omni is a more reliable and consistent model for answering questions related to gynecological cancers than Gemini. The stability of Omni's performance over time highlights its potential as an effective tool for clinical applications requiring high accuracy and consistency.</p>","PeriodicalId":19543,"journal":{"name":"Oncology Research and Treatment","volume":" ","pages":"1-10"},"PeriodicalIF":2.0000,"publicationDate":"2025-03-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Oncology Research and Treatment","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1159/000545231","RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"ONCOLOGY","Score":null,"Total":0}
引用次数: 0
Abstract
supporting clinical decision-making, diagnosis, and treatment. The study aims to compare the performance of ChatGPT-4o (Omni) and Gemini-pro in answering clinical questions and case scenarios related to gynecological oncology and to assess the consistency of their long-term responses.
Methods: A two-phase comparative analysis was conducted. 700 clinical questions (350 per model) were developed and categorized into open-ended and case-scenario questions. Three months later, the same set of questions was presented again to evaluate any changes in performance for accuracy, completeness, and guideline adherence.
Results: Omni outperformed Gemini-pro across all question types (p=0.001). Omni achieved a mean score of 5.9 for the basic open-ended questions, higher than Gemini, which had 5.1 (p=0.001). It also maintained a clear advantage in complex, open-ended questions, scoring a mean of 5.6 than Gemini AI's 4.2 (p=0.001). Omni scored a mean of 5.7 for basic case scenarios, while Gemini AI lagged with a mean score of 5 (p=0.001). Omni showed a modest improvement in complex, open-ended queries, with an increase of 0.2 points (+3.57%) (p=0.001). Omni provided more accurate and comprehensive responses in guideline adherence than Gemini, particularly in complex cases requiring nuanced judgment and adherence to oncology protocols. Its responses aligned with the latest guidelines, including the American Society of Clinical Oncology and the National Comprehensive Cancer Network.
Conclusions: Omni is a more reliable and consistent model for answering questions related to gynecological cancers than Gemini. The stability of Omni's performance over time highlights its potential as an effective tool for clinical applications requiring high accuracy and consistency.
期刊介绍：
With the first issue in 2014, the journal ''Onkologie'' has changed its title to ''Oncology Research and Treatment''. By this change, publisher and editor set the scene for the further development of this interdisciplinary journal. The English title makes it clear that the articles are published in English – a logical step for the journal, which is listed in all relevant international databases. For excellent manuscripts, a ''Fast Track'' was introduced: The review is carried out within 2 weeks; after acceptance the papers are published online within 14 days and immediately released as ''Editor’s Choice'' to provide the authors with maximum visibility of their results. Interesting case reports are published in the section ''Novel Insights from Clinical Practice'' which clearly highlights the scientific advances which the report presents.
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号
