Performance of a virtual assistant based on ChatGPT-4 in the diagnosis of syndromes with orofacial manifestations.

Abstract

Objective: To evaluate the performance of the virtual assistant "Syndromic Diseases and Orofacial Features" (SDOF), developed based on the Generative Pre-trained Transformer 4 model, in formulating diagnostic hypotheses and recommendations for syndromes with orofacial manifestations.

Study design: Twenty-six anonymized, previously diagnosed clinical cases, including clinical features and images, were selected. The assistant was trained using scientific references and configured to generate diagnostic hypotheses and suggest complementary exams. The responses were evaluated by two oral diagnosis specialists based on criteria such as accuracy, completeness, relevance, and comprehensibility. Statistical analysis was performed using RStudio software to calculate means and standard deviations.

Results: The SDOF correctly identified 96.2% of the cases, with 80.8% being the first diagnostic hypothesis and 15.4% being the second. In only one case (3.8%), the correct diagnosis was presented as the third hypothesis. The assistant performed best in the criteria "Relevance," "Practicality," and "Readability," while "Completeness" and "Up-to-dateness" scored the lowest. Despite the high accuracy rate, the assistant failed to mention all diagnostic steps in 7.69% of the cases.

Conclusions: The SDOF demonstrated significant potential to assist in the diagnosis of orofacial syndromes, with promising accuracy rates. However, the tool still requires professional supervision and improvements in completeness and up-to-dateness.