ChatGPT's Response Consistency: A Study on Repeated Queries of Medical Examination Questions.

IF 3 Q1 PSYCHOLOGY, CLINICAL

European Journal of Investigation in Health Psychology and Education Pub Date : 2024-03-08 DOI:10.3390/ejihpe14030043

Paul F Funk, Cosima C Hoch, Samuel Knoedler, Leonard Knoedler, Sebastian Cotofana, Giuseppe Sofo, Ali Bashiri Dezfouli, Barbara Wollenberg, Orlando Guntinas-Lichius, Michael Alfertshofer

{"title":"ChatGPT's Response Consistency: A Study on Repeated Queries of Medical Examination Questions.","authors":"Paul F Funk, Cosima C Hoch, Samuel Knoedler, Leonard Knoedler, Sebastian Cotofana, Giuseppe Sofo, Ali Bashiri Dezfouli, Barbara Wollenberg, Orlando Guntinas-Lichius, Michael Alfertshofer","doi":"10.3390/ejihpe14030043","DOIUrl":null,"url":null,"abstract":"(1) Background: As the field of artificial intelligence (AI) evolves, tools like ChatGPT are increasingly integrated into various domains of medicine, including medical education and research. Given the critical nature of medicine, it is of paramount importance that AI tools offer a high degree of reliability in the information they provide. (2) Methods: A total of n = 450 medical examination questions were manually entered into ChatGPT thrice, each for ChatGPT 3.5 and ChatGPT 4. The responses were collected, and their accuracy and consistency were statistically analyzed throughout the series of entries. (3) Results: ChatGPT 4 displayed a statistically significantly improved accuracy with 85.7% compared to that of 57.7% of ChatGPT 3.5 (p < 0.001). Furthermore, ChatGPT 4 was more consistent, correctly answering 77.8% across all rounds, a significant increase from the 44.9% observed from ChatGPT 3.5 (p < 0.001). (4) Conclusions: The findings underscore the increased accuracy and dependability of ChatGPT 4 in the context of medical education and potential clinical decision making. Nonetheless, the research emphasizes the indispensable nature of human-delivered healthcare and the vital role of continuous assessment in leveraging AI in medicine.","PeriodicalId":30631,"journal":{"name":"European Journal of Investigation in Health Psychology and Education","volume":"14 3","pages":"657-668"},"PeriodicalIF":3.0000,"publicationDate":"2024-03-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10969490/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"European Journal of Investigation in Health Psychology and Education","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.3390/ejihpe14030043","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"PSYCHOLOGY, CLINICAL","Score":null,"Total":0}

引用次数: 0

Abstract

(1) Background: As the field of artificial intelligence (AI) evolves, tools like ChatGPT are increasingly integrated into various domains of medicine, including medical education and research. Given the critical nature of medicine, it is of paramount importance that AI tools offer a high degree of reliability in the information they provide. (2) Methods: A total of n = 450 medical examination questions were manually entered into ChatGPT thrice, each for ChatGPT 3.5 and ChatGPT 4. The responses were collected, and their accuracy and consistency were statistically analyzed throughout the series of entries. (3) Results: ChatGPT 4 displayed a statistically significantly improved accuracy with 85.7% compared to that of 57.7% of ChatGPT 3.5 (p < 0.001). Furthermore, ChatGPT 4 was more consistent, correctly answering 77.8% across all rounds, a significant increase from the 44.9% observed from ChatGPT 3.5 (p < 0.001). (4) Conclusions: The findings underscore the increased accuracy and dependability of ChatGPT 4 in the context of medical education and potential clinical decision making. Nonetheless, the research emphasizes the indispensable nature of human-delivered healthcare and the vital role of continuous assessment in leveraging AI in medicine.

查看原文本刊更多论文

ChatGPT 的反应一致性：重复询问体检问题的研究。

(1) 背景：随着人工智能（AI）领域的发展，像 ChatGPT 这样的工具正越来越多地融入医学的各个领域，包括医学教育和研究。鉴于医学的关键性，人工智能工具所提供信息的高度可靠性至关重要。(2) 方法：在 ChatGPT 3.5 和 ChatGPT 4 中分别手动输入了三次共 n = 450 个医学考试问题，并收集了回复信息，对整个输入过程中的准确性和一致性进行了统计分析。(3) 结果：与 ChatGPT 3.5 的 57.7% 相比，ChatGPT 4 的准确率明显提高，达到 85.7%（p < 0.001）。此外，ChatGPT 4 的一致性更强，在所有轮次中正确回答率为 77.8%，比 ChatGPT 3.5 的 44.9% 有了明显提高（p < 0.001）。(4) 结论：研究结果强调了 ChatGPT 4 在医学教育和潜在临床决策中的准确性和可靠性。不过，这项研究强调了人类提供医疗保健的不可或缺性，以及持续评估在利用人工智能医疗方面的重要作用。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊