Performance of four chatbots versus orthodontists in answering multiple-choice questions in orthodontics and dentofacial orthopedics

Abstract

Background

The present study assessed the performance of four prominent chatbots (ChatGPT-4o, Grok 3, Gemini Advanced, and Claude 3.7 Sonnet) in answering orthodontics- and dentofacial orthopedics-related multiple-choice questions (MCQs).

Methods

One hundred MCQs were prepared by three orthodontic consultants. These questions, each with four potential answers, covered basic and advanced orthodontic knowledge and clinical case scenarios. The chatbots’ answers were assessed twice at a one-week interval. Three orthodontists’ answers were reported once on a Google Form. Performance was measured as the percentage of correct answers, while consistency was measured as the McNemar test and Cohen's Kappa. A P ≤ 0.05 was considered significant.

Results

The chatbots’ performance ranged from 72% to 84% and from 79% to 85% in the first and second rounds, respectively. Orthodontists’ performances ranged from 71% to 84%. The highest intrachatbot consistency was observed for Grok 3 (McNemar P = 1, and Kappa = 0.793), while the lowest was for Gemini Advanced and ChatGPT-4o (McNemar P = 0.001 and 0.167, respectively, and Kappa = 0.490 each). The interchatbot consistencies showed variability over time and were significantly different between Grok 3 and Claude 3.7 Sonnet (McNemar P = 0.109, and Kappa = 0.677) in the first round and between Gemini Advanced and Claude 3.7 Sonnet (McNemar P = 0.344, and Kappa = 0.647) in the second round. The consistencies among the orthodontists and between the orthodontists and chatbots were highly variable.

Conclusions

The performance of the evaluated chatbots in answering MCQs in orthodontics and dentofacial orthopedics is still below expectations, although it was slightly better than that of orthodontists. Substantial inconsistencies existed among the chatbots, thus making them unsuitable for completely replacing existing educational methods and strongly suggesting that further improvement, continuous updating, and assessments are needed.