Evaluation of Large Language Models' Concordance With Guidelines on Olfaction

IF 1.6 4区医学 Q2 OTORHINOLARYNGOLOGY

Laryngoscope Investigative Otolaryngology Pub Date : 2025-03-22 DOI:10.1002/lio2.70130

Ariana L. Shaari, Anthony M. Saad, Disha Patil, Jennifer Yanik, Wayne D. Hsueh, Jean Anderson Eloy, Andrey Filimonov

{"title":"Evaluation of Large Language Models' Concordance With Guidelines on Olfaction","authors":"Ariana L. Shaari, Anthony M. Saad, Disha Patil, Jennifer Yanik, Wayne D. Hsueh, Jean Anderson Eloy, Andrey Filimonov","doi":"10.1002/lio2.70130","DOIUrl":null,"url":null,"abstract":"<div>\n \n \n <section>\n \n <h3> Objective</h3>\n \n <p>To assess the concordance of artificial intelligence (AI)-generated information with the 2022 International Consensus Statement on Allergy and Rhinology: Olfaction (ICAR-O).</p>\n </section>\n \n <section>\n \n <h3> Methods</h3>\n \n <p>Forty-two guidelines were extracted from the ICAR-O. Each guideline was converted into a question, which was presented to ChatGPT version 4.o and Google Gemini. Concordance was deemed an agreement between the AI response and the clinical recommendation. Credibility was granted if the AI platform provided a credible resource. Accuracy was graded on a Likert scale (0: entirely inaccurate information, 1: mix of accurate and inaccurate information, 2: entirely accurate information). Statistical analysis was performed.</p>\n </section>\n \n <section>\n \n <h3> Results</h3>\n \n <p>A total of 84 responses were generated. The mean accuracy of the ChatGPT and Gemini responses was 1.85 and 1.48 out of 2, respectively, indicating that the responses contained a mix of accurate and inaccurate information. ChatGPT responses were significantly more accurate than Gemini responses (<i>p</i> = 0.001). Of the ChatGPT responses, 78.57% (<i>N</i> = 33) were concordant with the ICAR-O guidelines and 100% (<i>N</i> = 42) cited a credible resource. Of the Gemini responses, 66.67% (<i>N</i> = 28) were concordant and 97.62% (<i>N</i> = 41) cited a credible resource. There were no significant differences in concordance (<i>p</i> = 0.22) or credibility (<i>p</i> = 0.31) between the AI platforms.</p>\n </section>\n \n <section>\n \n <h3> Conclusion</h3>\n \n <p>ChatGPT provided more accurate information than Gemini on olfaction. However, overall, both platforms did not consistently align with clinical guidelines. AI platforms require further evaluation before clinical implementation or use as educational adjuncts.</p>\n </section>\n \n <section>\n \n <h3> Level of Evidence</h3>\n \n <p>N/A.</p>\n </section>\n </div>","PeriodicalId":48529,"journal":{"name":"Laryngoscope Investigative Otolaryngology","volume":"10 2","pages":""},"PeriodicalIF":1.6000,"publicationDate":"2025-03-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/lio2.70130","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Laryngoscope Investigative Otolaryngology","FirstCategoryId":"3","ListUrlMain":"https://onlinelibrary.wiley.com/doi/10.1002/lio2.70130","RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"OTORHINOLARYNGOLOGY","Score":null,"Total":0}

引用次数: 0

Abstract

Objective

To assess the concordance of artificial intelligence (AI)-generated information with the 2022 International Consensus Statement on Allergy and Rhinology: Olfaction (ICAR-O).

Methods

Forty-two guidelines were extracted from the ICAR-O. Each guideline was converted into a question, which was presented to ChatGPT version 4.o and Google Gemini. Concordance was deemed an agreement between the AI response and the clinical recommendation. Credibility was granted if the AI platform provided a credible resource. Accuracy was graded on a Likert scale (0: entirely inaccurate information, 1: mix of accurate and inaccurate information, 2: entirely accurate information). Statistical analysis was performed.

Results

A total of 84 responses were generated. The mean accuracy of the ChatGPT and Gemini responses was 1.85 and 1.48 out of 2, respectively, indicating that the responses contained a mix of accurate and inaccurate information. ChatGPT responses were significantly more accurate than Gemini responses (p = 0.001). Of the ChatGPT responses, 78.57% (N = 33) were concordant with the ICAR-O guidelines and 100% (N = 42) cited a credible resource. Of the Gemini responses, 66.67% (N = 28) were concordant and 97.62% (N = 41) cited a credible resource. There were no significant differences in concordance (p = 0.22) or credibility (p = 0.31) between the AI platforms.

Conclusion

ChatGPT provided more accurate information than Gemini on olfaction. However, overall, both platforms did not consistently align with clinical guidelines. AI platforms require further evaluation before clinical implementation or use as educational adjuncts.