Artificial intelligence vs. human expert: Licensed mental health clinicians' blinded evaluation of AI-generated and expert psychological advice on quality, empathy, and perceived authorship

IF 4.1 2区医学 Q1 HEALTH CARE SCIENCES & SERVICES

Internet Interventions-The Application of Information Technology in Mental and Behavioural Health Pub Date : 2025-06-03 DOI:10.1016/j.invent.2025.100841

Ludwig Franke Föyen , Emma Zapel , Mats Lekander , Erik Hedman-Lagerlöf , Elin Lindsäter

{"title":"Artificial intelligence vs. human expert: Licensed mental health clinicians' blinded evaluation of AI-generated and expert psychological advice on quality, empathy, and perceived authorship","authors":"Ludwig Franke Föyen , Emma Zapel , Mats Lekander , Erik Hedman-Lagerlöf , Elin Lindsäter","doi":"10.1016/j.invent.2025.100841","DOIUrl":null,"url":null,"abstract":"<div><h3>Background</h3><div>The use of artificial intelligence for psychological advice shows promise for enhancing accessibility and reducing costs, but it remains unclear whether AI-generated advice can match the quality and empathy of experts.</div></div><div><h3>Method</h3><div>In a blinded, comparative cross-sectional design, licensed psychologists and psychotherapists assessed the quality, empathy, and authorship of psychological advice, which was either AI-generated or authored by experts.</div></div><div><h3>Results</h3><div>AI-generated responses were rated significantly more favorable for emotional (OR = 1.79, 95 % CI [1.1, 2.93], <em>p</em> = .02) and motivational empathy (OR = 1.84, 95 % CI [1.12, 3.04], <em>p</em> = .02). Ratings for scientific quality (<em>p</em> = .10) and cognitive empathy (<em>p</em> = .08) were comparable to expert advice. Participants could not distinguish between AI- and expert-authored advice (<em>p</em> = .27), but <em>perceived</em> expert authorship was associated with more favorable ratings across these measures (ORs for perceived AI vs. perceived expert ranging from 0.03 to 0.15, all <em>p</em> < .001). For overall preference, AI-authored advice was favored when assessed blindly based on its actual source (<em>β</em> = 6.96, <em>p</em> = .002). Nevertheless, advice <em>perceived</em> as expert-authored was also strongly preferred (<em>β</em> = 6.26, <em>p</em> = .001), with 93.55 % of participants preferring the advice they believed came from an expert, irrespective of its true origin.</div></div><div><h3>Conclusions</h3><div>AI demonstrates potential to match expert performance in asynchronous written psychological advice, but biases favoring perceived expert authorship may hinder its broader acceptance. Mitigating these biases and evaluating AI's trustworthiness and empathy are important next steps for safe and effective integration of AI in clinical practice.</div></div>","PeriodicalId":48615,"journal":{"name":"Internet Interventions-The Application of Information Technology in Mental and Behavioural Health","volume":"41 ","pages":"Article 100841"},"PeriodicalIF":4.1000,"publicationDate":"2025-06-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Internet Interventions-The Application of Information Technology in Mental and Behavioural Health","FirstCategoryId":"3","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S2214782925000429","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"HEALTH CARE SCIENCES & SERVICES","Score":null,"Total":0}

引用次数: 0

Abstract

Background

The use of artificial intelligence for psychological advice shows promise for enhancing accessibility and reducing costs, but it remains unclear whether AI-generated advice can match the quality and empathy of experts.

Method

In a blinded, comparative cross-sectional design, licensed psychologists and psychotherapists assessed the quality, empathy, and authorship of psychological advice, which was either AI-generated or authored by experts.

Results

AI-generated responses were rated significantly more favorable for emotional (OR = 1.79, 95 % CI [1.1, 2.93], p = .02) and motivational empathy (OR = 1.84, 95 % CI [1.12, 3.04], p = .02). Ratings for scientific quality (p = .10) and cognitive empathy (p = .08) were comparable to expert advice. Participants could not distinguish between AI- and expert-authored advice (p = .27), but perceived expert authorship was associated with more favorable ratings across these measures (ORs for perceived AI vs. perceived expert ranging from 0.03 to 0.15, all p < .001). For overall preference, AI-authored advice was favored when assessed blindly based on its actual source (β = 6.96, p = .002). Nevertheless, advice perceived as expert-authored was also strongly preferred (β = 6.26, p = .001), with 93.55 % of participants preferring the advice they believed came from an expert, irrespective of its true origin.

Conclusions

AI demonstrates potential to match expert performance in asynchronous written psychological advice, but biases favoring perceived expert authorship may hinder its broader acceptance. Mitigating these biases and evaluating AI's trustworthiness and empathy are important next steps for safe and effective integration of AI in clinical practice.

查看原文本刊更多论文

人工智能与人类专家：持牌心理健康临床医生对人工智能生成的和专家心理建议的质量、同理心和感知作者身份的盲法评估

人工智能在心理咨询方面的应用有望提高可及性并降低成本，但目前尚不清楚人工智能生成的建议是否能与专家的质量和同理心相媲美。方法采用盲法比较横断面设计，持牌心理学家和心理治疗师评估了人工智能生成或专家撰写的心理建议的质量、同理心和作者身份。结果人工智能产生的反应在情感共情（OR = 1.79, 95% CI [1.1, 2.93], p = 0.02）和动机共情（OR = 1.84, 95% CI [1.12, 3.04], p = 0.02）方面显著更有利。科学质量评分（p = 0.10）和认知同理心评分（p = 0.08）与专家建议相当。参与者无法区分人工智能和专家撰写的建议（p = 0.27），但感知到的专家撰写与这些措施中更有利的评分相关(感知到的人工智能与感知到的专家的or范围从0.03到0.15，所有p <；措施)。就总体偏好而言，人工智能撰写的建议在基于其实际来源进行盲目评估时更受青睐（β = 6.96, p = 0.002）。然而，被认为是专家撰写的建议也被强烈偏爱（β = 6.26, p = .001）， 93.55%的参与者更喜欢他们认为来自专家的建议，而不管其真实来源如何。结论：ai在异步书面心理咨询中具有与专家表现相匹配的潜力，但对专家作者的偏见可能会阻碍其被广泛接受。减轻这些偏见，评估人工智能的可信度和同理心，是将人工智能安全有效地整合到临床实践中的重要下一步。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Internet Interventions-The Application of Information Technology in Mental and Behavioural Health Medicine-Health Informatics

CiteScore

6.50

自引率

9.30%

发文量

审稿时长

6 weeks

期刊介绍： Official Journal of the European Society for Research on Internet Interventions (ESRII) and the International Society for Research on Internet Interventions (ISRII). The aim of Internet Interventions is to publish scientific, peer-reviewed, high-impact research on Internet interventions and related areas. Internet Interventions welcomes papers on the following subjects: • Intervention studies targeting the promotion of mental health and featuring the Internet and/or technologies using the Internet as an underlying technology, e.g. computers, smartphone devices, tablets, sensors • Implementation and dissemination of Internet interventions • Integration of Internet interventions into existing systems of care • Descriptions of development and deployment infrastructures • Internet intervention methodology and theory papers • Internet-based epidemiology • Descriptions of new Internet-based technologies and experiments with clinical applications • Economics of internet interventions (cost-effectiveness) • Health care policy and Internet interventions • The role of culture in Internet intervention • Internet psychometrics • Ethical issues pertaining to Internet interventions and measurements • Human-computer interaction and usability research with clinical implications • Systematic reviews and meta-analysis on Internet interventions