Evaluating the quality and empathy of responses to patient questions on the Korean Academy of Periodontology's online question and answer section: a cross-sectional study comparing periodontists and an AI-powered chatbot.

Abstract

Purpose: This study aimed to evaluate and compare the responses of an artificial intelligence (AI)-powered chatbot and professional periodontists to patient queries in periodontology and implantology, using the Korean Academy of Periodontology's (KAP) online question and answer (Q&A) section.

Methods: In this comparative cross-sectional study, we analyzed 219 patient-submitted periodontal and implant knowledge questions from the KAP online Q&A section. A panel of 10 evaluators-5 periodontists and 5 laypersons-rated both the periodontist's and the AI chatbot's responses using standardized scales. We applied the t-test and Spearman correlation coefficients to compare response quality, empathy, consistency, and evaluator preferences.

Results: Ten evaluators judged the AI chatbot's responses to be significantly superior in quality and empathy compared to periodontist replies. A higher proportion of periodontist responses fell below acceptable quality ("very poor" or "poor") than chatbot responses (28.7% vs. 15.0%; P<0.001), and more chatbot replies were rated "empathetic" or "very empathetic" (62.5% vs. 42.8%; P<0.001). Overall response consistency was deemed satisfactory at 64.2%, with no significant difference in consistency or preference between periodontist and lay evaluators.

Conclusions: AI-powered chatbots can deliver more accurate and empathetic answers than human periodontists, suggesting their potential role as consultation assistants merits further investigation. The high intraclass correlation coefficient values (0.79-0.93) indicate a high level of agreement among evaluators in both the periodontist and lay evaluator groups, thus confirming the reliability and robustness of the study's assessment methodology.