Navigating Gynecological Oncology with Different Versions of ChatGPT: A Transformative Breakthrough or the Next Black Box Challenge?

IF 2 4区医学 Q3 ONCOLOGY

Oncology Research and Treatment Pub Date : 2024-12-17 DOI:10.1159/000543173

Nur Dokuzeylül Güngör, Fatih Sinan Esen, Tolga Taşçı, Kağan Güngör, Kaan Cil

{"title":"Navigating Gynecological Oncology with Different Versions of ChatGPT: A Transformative Breakthrough or the Next Black Box Challenge?","authors":"Nur Dokuzeylül Güngör, Fatih Sinan Esen, Tolga Taşçı, Kağan Güngör, Kaan Cil","doi":"10.1159/000543173","DOIUrl":null,"url":null,"abstract":"Introduction: The study evaluates the performance of large language model (LLM) versions of ChatGPT-ChatGPT-3.5, ChatGPT-4, and ChatGPT-Omni-in addressing inquiries related to the diagnosis and treatment of gynecological cancers, including ovarian, endometrial, and cervical cancers.Methods: A total of 804 questions were equally distributed across four categories: True/False, Multiple-Choice, Open-Ended, and Case-scenario, with each question type representing varying levels of complexity. Performance was assessed using a six-point Likert scale, focusing on accuracy, completeness, and alignment with established clinical guidelines.Results: For True/False queries, ChatGPT-Omni achieved accuracy rates of 100% for easy, 98% for medium, and 97% for complicated questions, higher than ChatGPT-4 (94%, 90%, 85%) and ChatGPT-3.5 (90%, 85%, 80%) (p=0.041, 0.023, 0.014, respectively). In Multiple-Choice, ChatGPT-Omni maintained superior accuracy with 100% for easy, 98% for medium, and 93% for complicated queries, compared to ChatGPT-4 (92%, 88%, 80%) and ChatGPT-3.5 (85%, 80%, 70%) (p=0.035, 0.028, 0.011). For Open-Ended questions, ChatGPT-Omni had mean Likert scores of 5.8 for easy, 5.5 for medium, and 5.2 for complex levels, outperforming ChatGPT-4 (5.4, 5.0, 4.5) and ChatGPT-3.5 (5.0, 4.5, 4.0) (p=0.037, 0.026, 0.015). Similar trends were observed in Case-Scenario questions, where ChatGPT-Omni achieved scores of 5.6, 5.3, and 4.9 for easy, medium, and hard levels, respectively (p=0.017, 0.008, 0.012).Conclusions: ChatGPT-Omni exhibited superior performance in responding to clinical queries related to gynecological cancers, underscoring its potential utility as a decision-support tool and an educational resource in clinical practice.","PeriodicalId":19543,"journal":{"name":"Oncology Research and Treatment","volume":" ","pages":"1-16"},"PeriodicalIF":2.0000,"publicationDate":"2024-12-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Oncology Research and Treatment","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1159/000543173","RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"ONCOLOGY","Score":null,"Total":0}

引用次数: 0

Abstract

Introduction: The study evaluates the performance of large language model (LLM) versions of ChatGPT-ChatGPT-3.5, ChatGPT-4, and ChatGPT-Omni-in addressing inquiries related to the diagnosis and treatment of gynecological cancers, including ovarian, endometrial, and cervical cancers.

Methods: A total of 804 questions were equally distributed across four categories: True/False, Multiple-Choice, Open-Ended, and Case-scenario, with each question type representing varying levels of complexity. Performance was assessed using a six-point Likert scale, focusing on accuracy, completeness, and alignment with established clinical guidelines.

Results: For True/False queries, ChatGPT-Omni achieved accuracy rates of 100% for easy, 98% for medium, and 97% for complicated questions, higher than ChatGPT-4 (94%, 90%, 85%) and ChatGPT-3.5 (90%, 85%, 80%) (p=0.041, 0.023, 0.014, respectively). In Multiple-Choice, ChatGPT-Omni maintained superior accuracy with 100% for easy, 98% for medium, and 93% for complicated queries, compared to ChatGPT-4 (92%, 88%, 80%) and ChatGPT-3.5 (85%, 80%, 70%) (p=0.035, 0.028, 0.011). For Open-Ended questions, ChatGPT-Omni had mean Likert scores of 5.8 for easy, 5.5 for medium, and 5.2 for complex levels, outperforming ChatGPT-4 (5.4, 5.0, 4.5) and ChatGPT-3.5 (5.0, 4.5, 4.0) (p=0.037, 0.026, 0.015). Similar trends were observed in Case-Scenario questions, where ChatGPT-Omni achieved scores of 5.6, 5.3, and 4.9 for easy, medium, and hard levels, respectively (p=0.017, 0.008, 0.012).

Conclusions: ChatGPT-Omni exhibited superior performance in responding to clinical queries related to gynecological cancers, underscoring its potential utility as a decision-support tool and an educational resource in clinical practice.

查看原文本刊更多论文

求助全文

约1分钟内获得全文求助全文

来源期刊

Oncology Research and Treatment ONCOLOGY-

CiteScore

3.20

自引率

0.00%

发文量

期刊介绍： With the first issue in 2014, the journal ''Onkologie'' has changed its title to ''Oncology Research and Treatment''. By this change, publisher and editor set the scene for the further development of this interdisciplinary journal. The English title makes it clear that the articles are published in English – a logical step for the journal, which is listed in all relevant international databases. For excellent manuscripts, a ''Fast Track'' was introduced: The review is carried out within 2 weeks; after acceptance the papers are published online within 14 days and immediately released as ''Editor’s Choice'' to provide the authors with maximum visibility of their results. Interesting case reports are published in the section ''Novel Insights from Clinical Practice'' which clearly highlights the scientific advances which the report presents.