Performance of large language artificial intelligence models on solving restorative dentistry and endodontics student assessments.

IF 3.1 2区医学 Q1 DENTISTRY, ORAL SURGERY & MEDICINE

Clinical Oral Investigations Pub Date : 2024-10-07 DOI:10.1007/s00784-024-05968-w

Paul Künzle, Sebastian Paris

{"title":"Performance of large language artificial intelligence models on solving restorative dentistry and endodontics student assessments.","authors":"Paul Künzle, Sebastian Paris","doi":"10.1007/s00784-024-05968-w","DOIUrl":null,"url":null,"abstract":"Objectives: The advent of artificial intelligence (AI) and large language model (LLM)-based AI applications (LLMAs) has tremendous implications for our society. This study analyzed the performance of LLMAs on solving restorative dentistry and endodontics (RDE) student assessment questions.Materials and methods: 151 questions from a RDE question pool were prepared for prompting using LLMAs from OpenAI (ChatGPT-3.5,-4.0 and -4.0o) and Google (Gemini 1.0). Multiple-choice questions were sorted into four question subcategories, entered into LLMAs and answers recorded for analysis. P-value and chi-square statistical analyses were performed using Python 3.9.16.Results: The total answer accuracy of ChatGPT-4.0o was the highest, followed by ChatGPT-4.0, Gemini 1.0 and ChatGPT-3.5 (72%, 62%, 44% and 25%, respectively) with significant differences between all LLMAs except GPT-4.0 models. The performance on subcategories direct restorations and caries was the highest, followed by indirect restorations and endodontics.Conclusions: Overall, there are large performance differences among LLMAs. Only the ChatGPT-4 models achieved a success ratio that could be used with caution to support the dental academic curriculum.Clinical relevance: While LLMAs could support clinicians to answer dental field-related questions, this capacity depends strongly on the employed model. The most performant model ChatGPT-4.0o achieved acceptable accuracy rates in some subject sub-categories analyzed.","PeriodicalId":10461,"journal":{"name":"Clinical Oral Investigations","volume":"28 11","pages":"575"},"PeriodicalIF":3.1000,"publicationDate":"2024-10-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11458639/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Clinical Oral Investigations","FirstCategoryId":"88","ListUrlMain":"https://doi.org/10.1007/s00784-024-05968-w","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"DENTISTRY, ORAL SURGERY & MEDICINE","Score":null,"Total":0}

引用次数: 0

Abstract

Objectives: The advent of artificial intelligence (AI) and large language model (LLM)-based AI applications (LLMAs) has tremendous implications for our society. This study analyzed the performance of LLMAs on solving restorative dentistry and endodontics (RDE) student assessment questions.

Materials and methods: 151 questions from a RDE question pool were prepared for prompting using LLMAs from OpenAI (ChatGPT-3.5,-4.0 and -4.0o) and Google (Gemini 1.0). Multiple-choice questions were sorted into four question subcategories, entered into LLMAs and answers recorded for analysis. P-value and chi-square statistical analyses were performed using Python 3.9.16.

Results: The total answer accuracy of ChatGPT-4.0o was the highest, followed by ChatGPT-4.0, Gemini 1.0 and ChatGPT-3.5 (72%, 62%, 44% and 25%, respectively) with significant differences between all LLMAs except GPT-4.0 models. The performance on subcategories direct restorations and caries was the highest, followed by indirect restorations and endodontics.

Conclusions: Overall, there are large performance differences among LLMAs. Only the ChatGPT-4 models achieved a success ratio that could be used with caution to support the dental academic curriculum.

Clinical relevance: While LLMAs could support clinicians to answer dental field-related questions, this capacity depends strongly on the employed model. The most performant model ChatGPT-4.0o achieved acceptable accuracy rates in some subject sub-categories analyzed.

查看原文本刊更多论文

大语言人工智能模型在解决修复牙科和牙髓病学学生评估方面的表现。

目的：人工智能（AI）和基于大型语言模型（LLM）的人工智能应用（LLMAs）的出现对我们的社会有着巨大的影响。本研究分析了 LLMA 在解决口腔修复和牙髓病学（RDE）学生评估问题上的表现。材料和方法：从 RDE 问题库中准备了 151 个问题，使用 OpenAI (ChatGPT-3.5,-4.0 和 -4.0o) 和谷歌 (Gemini 1.0) 的 LLMA 进行提示。多选题被分为四个问题子类别，输入 LLMA 并记录答案以供分析。使用 Python 3.9.16 进行了 P 值和卡方统计分析：ChatGPT-4.0o 的总答案准确率最高，其次是 ChatGPT-4.0、Gemini 1.0 和 ChatGPT-3.5（分别为 72%、62%、44% 和 25%），除 GPT-4.0 模型外，其他 LLMA 之间均存在显著差异。直接修复和龋齿子类别的性能最高，其次是间接修复和根管治疗：总的来说，LLMA 之间的性能差异很大。只有 ChatGPT-4 模型达到了可谨慎用于支持牙科学术课程的成功率：临床相关性：虽然 LLMA 可以帮助临床医生回答与牙科领域相关的问题，但这种能力在很大程度上取决于所使用的模型。性能最好的 ChatGPT-4.0o 模型在分析的某些科目子类别中达到了可接受的准确率。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Clinical Oral Investigations 医学-牙科与口腔外科

CiteScore

6.30

自引率

5.90%

发文量

484

审稿时长

3 months

期刊介绍： The journal Clinical Oral Investigations is a multidisciplinary, international forum for publication of research from all fields of oral medicine. The journal publishes original scientific articles and invited reviews which provide up-to-date results of basic and clinical studies in oral and maxillofacial science and medicine. The aim is to clarify the relevance of new results to modern practice, for an international readership. Coverage includes maxillofacial and oral surgery, prosthetics and restorative dentistry, operative dentistry, endodontics, periodontology, orthodontics, dental materials science, clinical trials, epidemiology, pedodontics, oral implant, preventive dentistiry, oral pathology, oral basic sciences and more.