Quality of Information Provided by Artificial Intelligence Chatbots Surrounding the Management of Vestibular Schwannomas: A Comparative Analysis Between ChatGPT-4 and Claude 2.
Daniele Borsetto, Egidio Sia, Patrick Axon, Neil Donnelly, James R Tysome, Lukas Anschuetz, Daniele Bernardeschi, Vincenzo Capriotti, Per Caye-Thomasen, Niels Cramer West, Isaac D Erbele, Sebastiano Franchella, Annalisa Gatto, Jeanette Hess-Erga, Henricus P M Kunst, John P Marinelli, Richard Mannion, Benedict Panizza, Franco Trabalzini, Rupert Obholzer, Luigi Angelo Vaira, Jerry Polesel, Fabiola Giudici, Matthew L Carlson, Giancarlo Tirelli, Paolo Boscolo-Rizzo
{"title":"Quality of Information Provided by Artificial Intelligence Chatbots Surrounding the Management of Vestibular Schwannomas: A Comparative Analysis Between ChatGPT-4 and Claude 2.","authors":"Daniele Borsetto, Egidio Sia, Patrick Axon, Neil Donnelly, James R Tysome, Lukas Anschuetz, Daniele Bernardeschi, Vincenzo Capriotti, Per Caye-Thomasen, Niels Cramer West, Isaac D Erbele, Sebastiano Franchella, Annalisa Gatto, Jeanette Hess-Erga, Henricus P M Kunst, John P Marinelli, Richard Mannion, Benedict Panizza, Franco Trabalzini, Rupert Obholzer, Luigi Angelo Vaira, Jerry Polesel, Fabiola Giudici, Matthew L Carlson, Giancarlo Tirelli, Paolo Boscolo-Rizzo","doi":"10.1097/MAO.0000000000004410","DOIUrl":null,"url":null,"abstract":"<p><strong>Objective: </strong>To examine the quality of information provided by artificial intelligence platforms ChatGPT-4 and Claude 2 surrounding the management of vestibular schwannomas.</p><p><strong>Study design: </strong>Cross-sectional.</p><p><strong>Setting: </strong>Skull base surgeons were involved from different centers and countries.</p><p><strong>Intervention: </strong>Thirty-six questions regarding vestibular schwannoma management were tested. Artificial intelligence responses were subsequently evaluated by 19 lateral skull base surgeons using the Quality Assessment of Medical Artificial Intelligence (QAMAI) questionnaire, assessing \"Accuracy,\" \"Clarity,\" \"Relevance,\" \"Completeness,\" \"Sources,\" and \"Usefulness.\"</p><p><strong>Main outcome measure: </strong>The scores of the answers from both chatbots were collected and analyzed using the Student t test. Analysis of responses grouped by stakeholders was performed with McNemar test. Stuart-Maxwell test was used to compare reading level among chatbots. Intraclass correlation coefficient was calculated.</p><p><strong>Results: </strong>ChatGPT-4 demonstrated significantly improved quality over Claude 2 in 14 of 36 (38.9%) questions, whereas higher-quality scores for Claude 2 were only observed in 2 (5.6%) answers. Chatbots exhibited variation across the dimensions of \"Accuracy,\" \"Clarity,\" \"Completeness,\" \"Relevance,\" and \"Usefulness,\" with ChatGPT-4 demonstrating a statistically significant superior performance. However, no statistically significant difference was found in the assessment of \"Sources.\" Additionally, ChatGPT-4 provided information at a significant lower reading grade level.</p><p><strong>Conclusions: </strong>Artificial intelligence platforms failed to consistently provide accurate information surrounding the management of vestibular schwannoma, although ChatGPT-4 achieved significantly higher scores in most analyzed parameters. These findings demonstrate the potential for significant misinformation for patients seeking information through these platforms.</p>","PeriodicalId":19732,"journal":{"name":"Otology & Neurotology","volume":" ","pages":"432-436"},"PeriodicalIF":1.9000,"publicationDate":"2025-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Otology & Neurotology","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1097/MAO.0000000000004410","RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2025/2/4 0:00:00","PubModel":"Epub","JCR":"Q3","JCRName":"CLINICAL NEUROLOGY","Score":null,"Total":0}
引用次数: 0
Abstract
Objective: To examine the quality of information provided by artificial intelligence platforms ChatGPT-4 and Claude 2 surrounding the management of vestibular schwannomas.
Study design: Cross-sectional.
Setting: Skull base surgeons were involved from different centers and countries.
Intervention: Thirty-six questions regarding vestibular schwannoma management were tested. Artificial intelligence responses were subsequently evaluated by 19 lateral skull base surgeons using the Quality Assessment of Medical Artificial Intelligence (QAMAI) questionnaire, assessing "Accuracy," "Clarity," "Relevance," "Completeness," "Sources," and "Usefulness."
Main outcome measure: The scores of the answers from both chatbots were collected and analyzed using the Student t test. Analysis of responses grouped by stakeholders was performed with McNemar test. Stuart-Maxwell test was used to compare reading level among chatbots. Intraclass correlation coefficient was calculated.
Results: ChatGPT-4 demonstrated significantly improved quality over Claude 2 in 14 of 36 (38.9%) questions, whereas higher-quality scores for Claude 2 were only observed in 2 (5.6%) answers. Chatbots exhibited variation across the dimensions of "Accuracy," "Clarity," "Completeness," "Relevance," and "Usefulness," with ChatGPT-4 demonstrating a statistically significant superior performance. However, no statistically significant difference was found in the assessment of "Sources." Additionally, ChatGPT-4 provided information at a significant lower reading grade level.
Conclusions: Artificial intelligence platforms failed to consistently provide accurate information surrounding the management of vestibular schwannoma, although ChatGPT-4 achieved significantly higher scores in most analyzed parameters. These findings demonstrate the potential for significant misinformation for patients seeking information through these platforms.
期刊介绍：
Otology & Neurotology publishes original articles relating to both clinical and basic science aspects of otology, neurotology, and cranial base surgery. As the foremost journal in its field, it has become the favored place for publishing the best of new science relating to the human ear and its diseases. The broadly international character of its contributing authors, editorial board, and readership provides the Journal its decidedly global perspective.
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号
