74 - Diagnostic accuracy of ChatGPT-4 in recommending immunohistochemistry panels for salivary gland tumours

IF 3.5 Q1 DENTISTRY, ORAL SURGERY & MEDICINE

Journal of the World Federation of Orthodontists Pub Date : 2025-12-01 Epub Date: 2025-12-09 DOI:10.1016/j.ejwf.2025.07.078

Valentina Alvarez Silberberg , Maria Carolina Cuevas Nuñez , Rodrigo Diaz Castañeda , Alexandro Gudenschwager , Maria Fernandez-Figueras

{"title":"74 - Diagnostic accuracy of ChatGPT-4 in recommending immunohistochemistry panels for salivary gland tumours","authors":"Valentina Alvarez Silberberg , Maria Carolina Cuevas Nuñez , Rodrigo Diaz Castañeda , Alexandro Gudenschwager , Maria Fernandez-Figueras","doi":"10.1016/j.ejwf.2025.07.078","DOIUrl":null,"url":null,"abstract":"<div><h3>Purpose</h3><div>Salivary gland tumours pose a significant diagnostic challenge due to their histological diversity and overlapping pathological features. Immunohistochemistry (IHC) is a crucial tool in distinguishing these neoplasms, yet selecting the optimal IHC panel remains complex. Artificial intelligence (AI) applications, hold promise in improving diagnostic accuracy by recommending IHC markers. This study evaluates the diagnostic performance of ChatGPT-4 in recommending IHC markers for salivary gland tumours.</div></div><div><h3>Methods</h3><div>A free version of ChatGPT-4 was used to generate IHC panel recommendations for 21 salivary gland tumour types. A consensus panel of expert pathologists established reference IHC panels, serving as the gold standard for comparison. Chatbot responses were assessed using a structured scoring system measuring accuracy, completeness, and relevance. Cohen’s Kappa analysis quantified agreement between chatbot and pathologist recommendations. Sensitivity, specificity, and F1-scores evaluated diagnostic performance. Consistency across multiple chatbot queries was assessed using repeated measures ANOVA and Bland-Altman analysis. Performance was compared against a rule-based system following strict pathologist guidelines.</div></div><div><h3>Results</h3><div>The chatbot demonstrated moderate overall agreement (κ=0.53) with pathologists, with higher agreement for benign (κ=0.67) than malignant tumours (κ=0.40). Sensitivity was high (>0.80) for benign tumours but lower for malignant neoplasms. The chatbot frequently recommended unnecessary markers, reducing specificity (<0.50) for several malignancies. Stability analysis revealed variability in chatbot responses, particularly for complex tumour types. ChatGPT underperformed compared to the rule-based system, exhibiting higher rates of incorrect and missed markers.</div></div><div><h3>Conclusion</h3><div>AI-powered chatbots show potential in aiding IHC marker selection but currently lack the precision required for clinical implementation.</div></div>","PeriodicalId":43456,"journal":{"name":"Journal of the World Federation of Orthodontists","volume":"14 6","pages":"Page 424"},"PeriodicalIF":3.5000,"publicationDate":"2025-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of the World Federation of Orthodontists","FirstCategoryId":"1085","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S221244382500133X","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2025/12/9 0:00:00","PubModel":"Epub","JCR":"Q1","JCRName":"DENTISTRY, ORAL SURGERY & MEDICINE","Score":null,"Total":0}

引用次数: 0

Abstract

Purpose

Salivary gland tumours pose a significant diagnostic challenge due to their histological diversity and overlapping pathological features. Immunohistochemistry (IHC) is a crucial tool in distinguishing these neoplasms, yet selecting the optimal IHC panel remains complex. Artificial intelligence (AI) applications, hold promise in improving diagnostic accuracy by recommending IHC markers. This study evaluates the diagnostic performance of ChatGPT-4 in recommending IHC markers for salivary gland tumours.

Methods

A free version of ChatGPT-4 was used to generate IHC panel recommendations for 21 salivary gland tumour types. A consensus panel of expert pathologists established reference IHC panels, serving as the gold standard for comparison. Chatbot responses were assessed using a structured scoring system measuring accuracy, completeness, and relevance. Cohen’s Kappa analysis quantified agreement between chatbot and pathologist recommendations. Sensitivity, specificity, and F1-scores evaluated diagnostic performance. Consistency across multiple chatbot queries was assessed using repeated measures ANOVA and Bland-Altman analysis. Performance was compared against a rule-based system following strict pathologist guidelines.

Results

The chatbot demonstrated moderate overall agreement (κ=0.53) with pathologists, with higher agreement for benign (κ=0.67) than malignant tumours (κ=0.40). Sensitivity was high (>0.80) for benign tumours but lower for malignant neoplasms. The chatbot frequently recommended unnecessary markers, reducing specificity (<0.50) for several malignancies. Stability analysis revealed variability in chatbot responses, particularly for complex tumour types. ChatGPT underperformed compared to the rule-based system, exhibiting higher rates of incorrect and missed markers.

Conclusion

AI-powered chatbots show potential in aiding IHC marker selection but currently lack the precision required for clinical implementation.

查看原文本刊更多论文

ChatGPT-4在推荐唾液腺肿瘤免疫组化检查中的诊断准确性

目的唾液腺肿瘤由于其组织多样性和重叠的病理特征，对诊断提出了重大挑战。免疫组化（IHC）是鉴别这些肿瘤的重要工具，但选择最佳的免疫组化组仍然很复杂。人工智能（AI）应用有望通过推荐免疫组化标记物来提高诊断准确性。本研究评估ChatGPT-4在推荐唾液腺肿瘤IHC标记物方面的诊断性能。方法使用ChatGPT-4免费版本生成21种唾液腺肿瘤类型的免疫组化推荐。由病理学专家组成的共识小组建立了参考免疫组化小组，作为比较的金标准。聊天机器人的回答使用一个结构化的评分系统来评估准确性、完整性和相关性。科恩的Kappa分析量化了聊天机器人和病理学家建议之间的一致性。敏感性、特异性和f1评分评估诊断性能。使用重复测量ANOVA和Bland-Altman分析评估多个聊天机器人查询的一致性。在严格的病理学指导下，将表现与基于规则的系统进行比较。结果聊天机器人与病理学家表现出中等程度的总体一致性（κ=0.53），对良性肿瘤（κ=0.67）的一致性高于恶性肿瘤（κ=0.40）。良性肿瘤的敏感性较高（>0.80），而恶性肿瘤的敏感性较低。聊天机器人经常推荐不必要的标记物，降低了几种恶性肿瘤的特异性（<0.50）。稳定性分析揭示了聊天机器人反应的可变性，特别是对于复杂的肿瘤类型。与基于规则的系统相比，ChatGPT表现不佳，显示出更高的不正确和遗漏标记率。结论人工智能聊天机器人在帮助免疫组化标志物选择方面具有潜力，但目前缺乏临床实施所需的精度。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Journal of the World Federation of Orthodontists DENTISTRY, ORAL SURGERY & MEDICINE-

CiteScore

3.80

自引率

4.80%

发文量