When AI Sounds Supportive: Comparing Conversational AI and Therapist Responses to Mental Health Disclosures

Abstract

Objectives

To compare artificial intelligence (AI) and therapist responses to real-world mental health (MH) disclosures and evaluate differences in empathy, safety practices and contextual usefulness.

Methods

We analysed 937 unique MH disclosures and 2749 therapist responses using the Counsel Chat dataset. For each disclosure, we generated a standardised AI response using a large language model. Responses were evaluated across empathy, safety and contextual usefulness measures using automated metrics, blinded human ratings of 120 therapist-AI response pairs, mixed effects regression models, and qualitative analyses of disagreement cases.

Results

AI responses scored higher than therapist responses on automated empathy (β = 0.30; p < 0.001), human-rated empathy (β = 0.38; p < 0.001), emotional validation (β = 0.40; p < 0.001) and automated safety (β = 0.25; p < 0.001). AI safety advantages were largest for self-harm or suicide-related disclosures. Therapists scored higher on specificity/personalisation (β = −0.62; p < 0.001) and helpfulness (β = −0.23; p = 0.013); overall quality did not differ significantly.

Conclusions

As conversational AI becomes an increasingly common source of MH support, AI responses may offer strengths in empathic communication and explicit safety-oriented guidance, whereas therapist responses were rated more highly for personalisation and contextual usefulness. These findings highlight different strengths across AI and therapist responses while underscoring the need for caution in interpreting response quality as clinical effectiveness. They also point to a potential role for conversational AI as an accessible first point of support within broader MH care systems, particularly where timely professional support is limited.