Investigating the Accuracy and Completeness of an Artificial Intelligence Large Language Model About Uveitis: An Evaluation of ChatGPT.

IF 2.6 4区医学 Q2 OPHTHALMOLOGY

Ocular Immunology and Inflammation Pub Date : 2024-11-01 Epub Date: 2024-02-23 DOI:10.1080/09273948.2024.2317417

Rayna F Marshall, Krishna Mallem, Hannah Xu, Jennifer Thorne, Bryn Burkholder, Benjamin Chaon, Paulina Liberman, Meghan Berkenstock

{"title":"Investigating the Accuracy and Completeness of an Artificial Intelligence Large Language Model About Uveitis: An Evaluation of ChatGPT.","authors":"Rayna F Marshall, Krishna Mallem, Hannah Xu, Jennifer Thorne, Bryn Burkholder, Benjamin Chaon, Paulina Liberman, Meghan Berkenstock","doi":"10.1080/09273948.2024.2317417","DOIUrl":null,"url":null,"abstract":"Purpose: To assess the accuracy and completeness of ChatGPT-generated answers regarding uveitis description, prevention, treatment, and prognosis.Methods: Thirty-two uveitis-related questions were generated by a uveitis specialist and inputted into ChatGPT 3.5. Answers were compiled into a survey and were reviewed by five uveitis specialists using standardized Likert scales of accuracy and completeness.Results: In total, the median accuracy score for all the uveitis questions (n = 32) was 4.00 (between \"more correct than incorrect\" and \"nearly all correct\"), and the median completeness score was 2.00 (\"adequate, addresses all aspects of the question and provides the minimum amount of information required to be considered complete\"). The interrater variability assessment had a total kappa value of 0.0278 for accuracy and 0.0847 for completeness.Conclusion: ChatGPT can provide relatively high accuracy responses for various questions related to uveitis; however, the answers it provides are incomplete, with some inaccuracies. Its utility in providing medical information requires further validation and development prior to serving as a source of uveitis information for patients.","PeriodicalId":19406,"journal":{"name":"Ocular Immunology and Inflammation","volume":" ","pages":"2052-2055"},"PeriodicalIF":2.6000,"publicationDate":"2024-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Ocular Immunology and Inflammation","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1080/09273948.2024.2317417","RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2024/2/23 0:00:00","PubModel":"Epub","JCR":"Q2","JCRName":"OPHTHALMOLOGY","Score":null,"Total":0}

引用次数: 0

Abstract

Purpose: To assess the accuracy and completeness of ChatGPT-generated answers regarding uveitis description, prevention, treatment, and prognosis.

Methods: Thirty-two uveitis-related questions were generated by a uveitis specialist and inputted into ChatGPT 3.5. Answers were compiled into a survey and were reviewed by five uveitis specialists using standardized Likert scales of accuracy and completeness.

Results: In total, the median accuracy score for all the uveitis questions (n = 32) was 4.00 (between "more correct than incorrect" and "nearly all correct"), and the median completeness score was 2.00 ("adequate, addresses all aspects of the question and provides the minimum amount of information required to be considered complete"). The interrater variability assessment had a total kappa value of 0.0278 for accuracy and 0.0847 for completeness.

Conclusion: ChatGPT can provide relatively high accuracy responses for various questions related to uveitis; however, the answers it provides are incomplete, with some inaccuracies. Its utility in providing medical information requires further validation and development prior to serving as a source of uveitis information for patients.

查看原文本刊更多论文

调查关于葡萄膜炎的人工智能大型语言模型的准确性和完整性：对 ChatGPT 的评估。

目的：评估 ChatGPT 生成的有关葡萄膜炎描述、预防、治疗和预后的答案的准确性和完整性：由葡萄膜炎专家生成 32 个葡萄膜炎相关问题，并输入 ChatGPT 3.5。五位葡萄膜炎专家使用标准化的李克特量表对问题的准确性和完整性进行了审核：总的来说，所有葡萄膜炎问题（n = 32）的准确性中位数为 4.00 分（介于 "正确多于错误 "和 "几乎全部正确 "之间），完整性中位数为 2.00 分（"充分，回答了问题的所有方面，并提供了被视为完整所需的最低信息量"）。在交互变异性评估中，准确性的总卡帕值为 0.0278，完整性的总卡帕值为 0.0847：结论：ChatGPT 可以为葡萄膜炎相关的各种问题提供准确性相对较高的回答；但是，它所提供的回答并不完整，存在一些不准确之处。它在提供医疗信息方面的实用性还需要进一步验证和开发，然后才能作为葡萄膜炎患者的信息来源。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Ocular Immunology and Inflammation 医学-眼科学

CiteScore

6.20

自引率

15.20%

发文量

285

审稿时长

6-12 weeks

期刊介绍： Ocular Immunology & Inflammation ranks 18 out of 59 in the Ophthalmology Category.Ocular Immunology and Inflammation is a peer-reviewed, scientific publication that welcomes the submission of original, previously unpublished manuscripts directed to ophthalmologists and vision scientists. Published bimonthly, the journal provides an international medium for basic and clinical research reports on the ocular inflammatory response and its control by the immune system. The journal publishes original research papers, case reports, reviews, letters to the editor, meeting abstracts, and invited editorials.