Performance assessment of artificial intelligence chatbots (ChatGPT-4 and Copilot) for sharing insights on 3D-printed orthodontic appliances: A cross-sectional study

IF 1.8 Q2 DENTISTRY, ORAL SURGERY & MEDICINE

International Orthodontics Pub Date : 2025-02-24 DOI:10.1016/j.ortho.2025.100992

Asma Muhammad Yousuf, Fizzah Ikram, Munnal Gulzar, Rashna Hoshang Sukhia, Mubassar Fida

{"title":"Performance assessment of artificial intelligence chatbots (ChatGPT-4 and Copilot) for sharing insights on 3D-printed orthodontic appliances: A cross-sectional study","authors":"Asma Muhammad Yousuf, Fizzah Ikram, Munnal Gulzar, Rashna Hoshang Sukhia, Mubassar Fida","doi":"10.1016/j.ortho.2025.100992","DOIUrl":null,"url":null,"abstract":"<div><h3>Objective</h3><div>To evaluate and compare the performance of OpenAI's ChatGPT-4 and Microsoft Copilot in providing information on 3D-printed orthodontic appliances, with a focus on the accuracy, completeness of the content, and response generation time.</div></div><div><h3>Methods</h3><div>This cross-sectional study proceeded in five stages. Initially, three orthodontists created a total of 125 questions concerning 3D printed orthodontic appliances of which 105 questions were finalized to be incorporated into the study by a panel of senior orthodontists. These questions were subsequently organized into 15 distinct domains. Both chatbots were presented with the questions under consistent conditions, using the same laptop and internet setup. A stopwatch was used to record response times. The responses were anonymized and evaluated by seven orthodontists with extensive experience, who scored accuracy and completeness based on standardized tools. Through discussion, evaluators reached a consensus on each score, ensuring reliability.</div></div><div><h3>Results</h3><div>Spearman's correlation revealed a moderate to strong negative correlation between accuracy and completeness for both chatbots (<em>p  </em>≤0.001), though response time differences were not statistically significant (<em>p</em> <em>=</em> <em>0.204</em>). Cohen's Kappa results implied little to no consistency between the two models on the assessed parameters (<em>p></em> <!-->0.05).</div></div><div><h3>Conclusion</h3><div>ChatGPT-4 outperformed Microsoft Copilot in accuracy and completeness, providing more precise and comprehensive information on 3D-printed orthodontic appliances demonstrating a greater ability to handle complex, and detailed requests in this area.</div></div>","PeriodicalId":45449,"journal":{"name":"International Orthodontics","volume":"23 3","pages":"Article 100992"},"PeriodicalIF":1.8000,"publicationDate":"2025-02-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"International Orthodontics","FirstCategoryId":"1085","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S1761722725000270","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"DENTISTRY, ORAL SURGERY & MEDICINE","Score":null,"Total":0}

引用次数: 0

Abstract

Objective

To evaluate and compare the performance of OpenAI's ChatGPT-4 and Microsoft Copilot in providing information on 3D-printed orthodontic appliances, with a focus on the accuracy, completeness of the content, and response generation time.

Methods

This cross-sectional study proceeded in five stages. Initially, three orthodontists created a total of 125 questions concerning 3D printed orthodontic appliances of which 105 questions were finalized to be incorporated into the study by a panel of senior orthodontists. These questions were subsequently organized into 15 distinct domains. Both chatbots were presented with the questions under consistent conditions, using the same laptop and internet setup. A stopwatch was used to record response times. The responses were anonymized and evaluated by seven orthodontists with extensive experience, who scored accuracy and completeness based on standardized tools. Through discussion, evaluators reached a consensus on each score, ensuring reliability.

Results

Spearman's correlation revealed a moderate to strong negative correlation between accuracy and completeness for both chatbots (p ≤ 0.001). The negative correlation observed between accuracy and completeness scores, particularly prominent in Copilot, indicates a trade-off between these qualities in some responses. Mann-Whitney U tests confirmed significant differences in accuracy and completeness between the chatbots (p ≤ 0.001), though response time differences were not statistically significant (p = 0.204). Cohen's Kappa results implied little to no consistency between the two models on the assessed parameters (p > 0.05).

Conclusion

ChatGPT-4 outperformed Microsoft Copilot in accuracy and completeness, providing more precise and comprehensive information on 3D-printed orthodontic appliances demonstrating a greater ability to handle complex, and detailed requests in this area.

查看原文本刊更多论文

求助全文

约1分钟内获得全文求助全文

来源期刊

International Orthodontics DENTISTRY, ORAL SURGERY & MEDICINE-

CiteScore

2.50

自引率

13.30%

发文量

审稿时长

26 days

期刊介绍： Une revue de référence dans le domaine de orthodontie et des disciplines frontières Your reference in dentofacial orthopedics International Orthodontics adresse aux orthodontistes, aux dentistes, aux stomatologistes, aux chirurgiens maxillo-faciaux et aux plasticiens de la face, ainsi quà leurs assistant(e)s. International Orthodontics is addressed to orthodontists, dentists, stomatologists, maxillofacial surgeons and facial plastic surgeons, as well as their assistants.