Comparing dentist and chatbot answers to dental questions for quality and empathy

Abstract

Background

Integration of large language models (LLMs) into health care, particularly in patient communication, is a growing trend. This study evaluated the effectiveness of LLM chatbots in addressing dental patient queries compared with responses from human dentists on a public online forum.

Methods

In January 2024, 20 patient questions and responses were randomly sampled from Reddit’s dental advice community. We assessed the quality and empathy of ChatGPT-generated responses (Version GPT-3.5, OpenAI) by 9 blinded dentists. The dentists were selected from a dental faculty pool familiar with reading and assessing written communication. The evaluators rated the information quality of the responses on a Likert scale (very poor, 1; poor, 2; acceptable, 3; good, 4; very good, 5) and empathy (not empathetic, 1; slightly empathetic, 2; moderately empathetic, 3; empathetic, 4; very empathetic, 5). Subsequently, they selected the best response (dentist or artificial intelligence). Nine blinded dentists rated 20 responses to the online inquiries, providing 180 potential responses.

Results

The results indicated that the LLM chatbots’ responses were rated as higher quality and exhibited higher levels of empathy than human responses. Among 179 responses (1 was missing) to the question about whether the response was better from ChatGPT or the dentist, 167 (93.3%) responses indicated ChatGPT and 12 (6.7%) indicated dentist (P < .001).

Conclusions

Although subjective variations in assessing quality and empathy may exist, this study suggests that LLM chatbot responses show higher quality and empathy than online dentist responses. The use of LLM chatbots by dentists can enhance patient communication in dental practice owing to their efficiency, empathy, and quality. Further research is needed to determine the full potential of artificial intelligence in dentistry.