Evaluating the Accuracy and Performance of ChatGPT-4o in Solving Japanese National Dental Technician Examination

Abstract

Introduction and aims

With advancements in artificial intelligence (AI), its role in education and examinations is expanding. This study aimed to evaluate the accuracy and response capability of ChatGPT-4o in solving questions from the Japanese National Dental Technician Examination (JNDTE). By analysing the AI’s answers to a variety of questions, including both text-based items and those containing visual information across multiple domains, we examined the practicality of using AI as a supportive tool for learning and assessment in dental education.

Materials and methods

A total of 400 JNDTE questions (2018-2022) were analysed using ChatGPT-4o. Standardized prompts were used, and statistical analysis was conducted using Fisher’s exact test.

Results

The AI demonstrated high accuracy in basic knowledge-based questions but lower accuracy in specialized topics requiring technical skills. Notably, accuracy decreased when visual elements (figures, tables, and photographs) were present. The highest accuracy was observed in ‘Dental materials, laboratory equipment, and processing technology’ (84.1%), while the lowest was in ‘Orthodontic technology’ (46.2%) and ‘Pediatric dental technology’ (50.0%). The AI particularly struggled with interpreting radiographic images and procedural workflows.

Conclusions

ChatGPT-4o demonstrated reasonable proficiency with JNDTE-related knowledge, suggesting its potential application as a supplementary tool in dental education. However, its limitations in processing visual and technical content highlight the need for advancements in AI-driven image recognition and simulation-based learning.

Clinical Relevance

The findings of this study highlight the potential application of AI models like ChatGPT-4o as a supplementary tool in dental technician education and examination preparation. Enhancing AI-driven image recognition and simulation-based learning could improve its effectiveness in dental education and clinical training, ultimately contributing to better learning outcomes and skill development for future dental technicians.