日本国家牙科考试gpt - 40和GPT-4表现的探索性评估

IF 2.3 Q3 DENTISTRY, ORAL SURGERY & MEDICINE
Masaki Morishita , Hikaru Fukuda , Shino Yamaguchi , Kosuke Muraoka , Taiji Nakamura , Masanari Hayashi , Izumi Yoshioka , Kentaro Ono , Shuji Awano
{"title":"日本国家牙科考试gpt - 40和GPT-4表现的探索性评估","authors":"Masaki Morishita ,&nbsp;Hikaru Fukuda ,&nbsp;Shino Yamaguchi ,&nbsp;Kosuke Muraoka ,&nbsp;Taiji Nakamura ,&nbsp;Masanari Hayashi ,&nbsp;Izumi Yoshioka ,&nbsp;Kentaro Ono ,&nbsp;Shuji Awano","doi":"10.1016/j.sdentj.2024.11.006","DOIUrl":null,"url":null,"abstract":"<div><h3>Background and Objectives</h3><div>Multiple large language models (LLMs) have been released since 2022, including OpenAI’s GPT-3.5 and GPT-4. The latest model, GPT-4o, introduced on May 13, 2024, significantly improves GPT-4. Previous studies have shown the potential of LLMs as educational tools in medical and dental exams. This study evaluates the accuracy of GPT-4 and GPT-4o responses for the Japanese National Dental Examination (JNDE) to assess their potential as educational tools for dental education.</div></div><div><h3>Materials and methods</h3><div>We obtained the dataset of the 117th JNDE, administered in January 2024, consisting of 360 questions. After excluding questions with images and inappropriate ones, 202 questions were selected. GPT-4 and GPT-4o were used to generate responses. Standardized prompts ensured consistent input. Data analysis used Qlik Sense® and GraphPad Prism, employing Fisher’s exact test.</div></div><div><h3>Results</h3><div>GPT-4o showed a significantly higher correct response rate (73.8%) than GPT-4 (63.3%). In the compulsory section, GPT-4o achieved 88.6% accuracy, significantly higher than GPT-4′s 74.3%. Though not statistically significant, the general section saw an improvement with GPT-4o (66.4%) over GPT-4 (58.0%).</div></div><div><h3>Conclusion</h3><div>GPT-4o significantly outperformed GPT-4 in accuracy for JNDE questions, suggesting its improved potential as an educational tool in dental education. Further studies are needed to evaluate GPT-4o’s capabilities with visual materials and in diverse question sets to fully ascertain its utility in educational settings.</div></div>","PeriodicalId":47246,"journal":{"name":"Saudi Dental Journal","volume":"36 12","pages":"Pages 1577-1581"},"PeriodicalIF":2.3000,"publicationDate":"2024-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"An exploratory assessment of GPT-4o and GPT-4 performance on the Japanese National Dental Examination\",\"authors\":\"Masaki Morishita ,&nbsp;Hikaru Fukuda ,&nbsp;Shino Yamaguchi ,&nbsp;Kosuke Muraoka ,&nbsp;Taiji Nakamura ,&nbsp;Masanari Hayashi ,&nbsp;Izumi Yoshioka ,&nbsp;Kentaro Ono ,&nbsp;Shuji Awano\",\"doi\":\"10.1016/j.sdentj.2024.11.006\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<div><h3>Background and Objectives</h3><div>Multiple large language models (LLMs) have been released since 2022, including OpenAI’s GPT-3.5 and GPT-4. The latest model, GPT-4o, introduced on May 13, 2024, significantly improves GPT-4. Previous studies have shown the potential of LLMs as educational tools in medical and dental exams. This study evaluates the accuracy of GPT-4 and GPT-4o responses for the Japanese National Dental Examination (JNDE) to assess their potential as educational tools for dental education.</div></div><div><h3>Materials and methods</h3><div>We obtained the dataset of the 117th JNDE, administered in January 2024, consisting of 360 questions. After excluding questions with images and inappropriate ones, 202 questions were selected. GPT-4 and GPT-4o were used to generate responses. Standardized prompts ensured consistent input. Data analysis used Qlik Sense® and GraphPad Prism, employing Fisher’s exact test.</div></div><div><h3>Results</h3><div>GPT-4o showed a significantly higher correct response rate (73.8%) than GPT-4 (63.3%). In the compulsory section, GPT-4o achieved 88.6% accuracy, significantly higher than GPT-4′s 74.3%. Though not statistically significant, the general section saw an improvement with GPT-4o (66.4%) over GPT-4 (58.0%).</div></div><div><h3>Conclusion</h3><div>GPT-4o significantly outperformed GPT-4 in accuracy for JNDE questions, suggesting its improved potential as an educational tool in dental education. Further studies are needed to evaluate GPT-4o’s capabilities with visual materials and in diverse question sets to fully ascertain its utility in educational settings.</div></div>\",\"PeriodicalId\":47246,\"journal\":{\"name\":\"Saudi Dental Journal\",\"volume\":\"36 12\",\"pages\":\"Pages 1577-1581\"},\"PeriodicalIF\":2.3000,\"publicationDate\":\"2024-12-01\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Saudi Dental Journal\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://www.sciencedirect.com/science/article/pii/S1013905224003092\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q3\",\"JCRName\":\"DENTISTRY, ORAL SURGERY & MEDICINE\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Saudi Dental Journal","FirstCategoryId":"1085","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S1013905224003092","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"DENTISTRY, ORAL SURGERY & MEDICINE","Score":null,"Total":0}
引用次数: 0

摘要

自2022年以来,已经发布了多个大型语言模型(llm),包括OpenAI的GPT-3.5和GPT-4。最新型号gpt - 40于2024年5月13日推出,大大改进了GPT-4。以前的研究表明,法学硕士作为医学和牙科考试的教育工具的潜力。本研究评估了日本国家牙科考试(JNDE)中GPT-4和gpt - 40反应的准确性,以评估其作为牙科教育教育工具的潜力。材料和方法我们获得了2024年1月第117届JNDE考试的数据集,包括360个问题。剔除带有图片和不合适的问题后,共选出202个问题。使用GPT-4和gpt - 40产生反应。标准化的提示确保了输入的一致性。数据分析使用Qlik Sense®和GraphPad Prism,采用Fisher精确检验。结果gpt - 40的正确率为73.8%,显著高于GPT-4的63.3%。在必修部分,gpt - 40的准确率达到了88.6%,明显高于GPT-4的74.3%。虽然没有统计学意义,但gpt - 40组(66.4%)优于GPT-4组(58.0%)。结论pt - 40在JNDE问题的正确率上明显优于GPT-4,提示其作为牙科教育工具的潜力有所提高。需要进一步的研究来评估gpt - 40在视觉材料和不同问题集方面的能力,以充分确定其在教育环境中的效用。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
An exploratory assessment of GPT-4o and GPT-4 performance on the Japanese National Dental Examination

Background and Objectives

Multiple large language models (LLMs) have been released since 2022, including OpenAI’s GPT-3.5 and GPT-4. The latest model, GPT-4o, introduced on May 13, 2024, significantly improves GPT-4. Previous studies have shown the potential of LLMs as educational tools in medical and dental exams. This study evaluates the accuracy of GPT-4 and GPT-4o responses for the Japanese National Dental Examination (JNDE) to assess their potential as educational tools for dental education.

Materials and methods

We obtained the dataset of the 117th JNDE, administered in January 2024, consisting of 360 questions. After excluding questions with images and inappropriate ones, 202 questions were selected. GPT-4 and GPT-4o were used to generate responses. Standardized prompts ensured consistent input. Data analysis used Qlik Sense® and GraphPad Prism, employing Fisher’s exact test.

Results

GPT-4o showed a significantly higher correct response rate (73.8%) than GPT-4 (63.3%). In the compulsory section, GPT-4o achieved 88.6% accuracy, significantly higher than GPT-4′s 74.3%. Though not statistically significant, the general section saw an improvement with GPT-4o (66.4%) over GPT-4 (58.0%).

Conclusion

GPT-4o significantly outperformed GPT-4 in accuracy for JNDE questions, suggesting its improved potential as an educational tool in dental education. Further studies are needed to evaluate GPT-4o’s capabilities with visual materials and in diverse question sets to fully ascertain its utility in educational settings.
求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
Saudi Dental Journal
Saudi Dental Journal DENTISTRY, ORAL SURGERY & MEDICINE-
CiteScore
3.60
自引率
0.00%
发文量
86
审稿时长
22 weeks
期刊介绍: Saudi Dental Journal is an English language, peer-reviewed scholarly publication in the area of dentistry. Saudi Dental Journal publishes original research and reviews on, but not limited to: • dental disease • clinical trials • dental equipment • new and experimental techniques • epidemiology and oral health • restorative dentistry • periodontology • endodontology • prosthodontics • paediatric dentistry • orthodontics and dental education Saudi Dental Journal is the official publication of the Saudi Dental Society and is published by King Saud University in collaboration with Elsevier and is edited by an international group of eminent researchers.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信