Accuracy of ChatGPT-4 Plus in Providing Information on Oral Cancer Management.

IF 2.9 3区医学 Q1 DENTISTRY, ORAL SURGERY & MEDICINE

Oral diseases Pub Date : 2025-10-03 DOI:10.1111/odi.70110

Busra Yilmaz, Emine Nur Kahraman, Michael T Brennan, Amardeep S Grewal, Aynur Aktas

{"title":"Accuracy of ChatGPT-4 Plus in Providing Information on Oral Cancer Management.","authors":"Busra Yilmaz, Emine Nur Kahraman, Michael T Brennan, Amardeep S Grewal, Aynur Aktas","doi":"10.1111/odi.70110","DOIUrl":null,"url":null,"abstract":"Objective: Artificial intelligence (AI)-driven large language models, such as Chat Generative Pre-Trained Transformer (ChatGPT)-4 Plus, are increasingly used for patient education and clinical decision support in oral oncology, although their accuracy in oral cancer (OC) management remains uncertain. This study evaluates the accuracy of ChatGPT-4 Plus responses to clinically relevant questions regarding OC diagnosis, treatment, recovery, and prevention.Methods: A cross-sectional study assessed 65 clinically relevant OC-related questions using a paid ChatGPT-4 Plus subscription without modifications. Three oral medicine specialists and one radiation oncologist rated accuracy on a four-point scoring system. Interrater reliability was measured with the intraclass correlation coefficient (ICC), and chi-square tests were used for comparisons.Results: Among 65 questions, 63% of responses were Score 1, with none rated as Score 4. Score 1 was most frequent in Recovery (72%), followed by Treatment (62%), Prevention (60%), and Diagnosis (55%). Scores 2 and 3 responses were highest in Diagnosis (45%). Recovery had significantly higher Score 1 responses than Diagnosis (p < 0.05), while other comparisons were not significant. ICC ranged from 0.85 to 0.93.Conclusions: ChatGPT-4 Plus provided accurate responses to clinically relevant OC-related questions, particularly regarding recovery. However, diagnostic inconsistencies highlight the need for clinician oversight before integrating AI into practice.","PeriodicalId":19615,"journal":{"name":"Oral diseases","volume":" ","pages":""},"PeriodicalIF":2.9000,"publicationDate":"2025-10-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Oral diseases","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1111/odi.70110","RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"DENTISTRY, ORAL SURGERY & MEDICINE","Score":null,"Total":0}

引用次数: 0

Abstract

Objective: Artificial intelligence (AI)-driven large language models, such as Chat Generative Pre-Trained Transformer (ChatGPT)-4 Plus, are increasingly used for patient education and clinical decision support in oral oncology, although their accuracy in oral cancer (OC) management remains uncertain. This study evaluates the accuracy of ChatGPT-4 Plus responses to clinically relevant questions regarding OC diagnosis, treatment, recovery, and prevention.

Methods: A cross-sectional study assessed 65 clinically relevant OC-related questions using a paid ChatGPT-4 Plus subscription without modifications. Three oral medicine specialists and one radiation oncologist rated accuracy on a four-point scoring system. Interrater reliability was measured with the intraclass correlation coefficient (ICC), and chi-square tests were used for comparisons.

Results: Among 65 questions, 63% of responses were Score 1, with none rated as Score 4. Score 1 was most frequent in Recovery (72%), followed by Treatment (62%), Prevention (60%), and Diagnosis (55%). Scores 2 and 3 responses were highest in Diagnosis (45%). Recovery had significantly higher Score 1 responses than Diagnosis (p < 0.05), while other comparisons were not significant. ICC ranged from 0.85 to 0.93.

Conclusions: ChatGPT-4 Plus provided accurate responses to clinically relevant OC-related questions, particularly regarding recovery. However, diagnostic inconsistencies highlight the need for clinician oversight before integrating AI into practice.

查看原文本刊更多论文

ChatGPT-4 Plus提供口腔癌管理信息的准确性。

目的：人工智能（AI）驱动的大型语言模型，如聊天生成预训练转换器(ChatGPT)-4 Plus，越来越多地用于口腔肿瘤学的患者教育和临床决策支持，尽管它们在口腔癌（OC）管理中的准确性仍不确定。本研究评估了ChatGPT-4 Plus在OC诊断、治疗、恢复和预防等临床相关问题上的准确性。方法：一项横断面研究评估了65个临床相关的oc相关问题，使用ChatGPT-4 Plus付费订阅，无需修改。三名口腔医学专家和一名放射肿瘤学家在四分制评分系统中对准确性进行了评分。用类内相关系数（ICC）衡量组间信度，比较采用卡方检验。结果：在65个问题中，63%的回答为1分，没有回答为4分。1分最常见于康复（72%），其次是治疗（62%）、预防（60%）和诊断（55%）。得分2和3的反应在诊断中最高（45%）。结论：ChatGPT-4 Plus对临床相关的oc相关问题提供了准确的回答，特别是关于康复的问题。然而，诊断的不一致性突出了在将人工智能应用于实践之前需要临床医生的监督。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Oral diseases 医学-牙科与口腔外科

CiteScore

7.60

自引率

5.30%

发文量

325

审稿时长

4-8 weeks

期刊介绍： Oral Diseases is a multidisciplinary and international journal with a focus on head and neck disorders, edited by leaders in the field, Professor Giovanni Lodi (Editor-in-Chief, Milan, Italy), Professor Stefano Petti (Deputy Editor, Rome, Italy) and Associate Professor Gulshan Sunavala-Dossabhoy (Deputy Editor, Shreveport, LA, USA). The journal is pre-eminent in oral medicine. Oral Diseases specifically strives to link often-isolated areas of dentistry and medicine through broad-based scholarship that includes well-designed and controlled clinical research, analytical epidemiology, and the translation of basic science in pre-clinical studies. The journal typically publishes articles relevant to many related medical specialties including especially dermatology, gastroenterology, hematology, immunology, infectious diseases, neuropsychiatry, oncology and otolaryngology. The essential requirement is that all submitted research is hypothesis-driven, with significant positive and negative results both welcomed. Equal publication emphasis is placed on etiology, pathogenesis, diagnosis, prevention and treatment.