Psychological First Aid by AI: Proof-of-Concept and Comparative Performance of ChatGPT-4 and Gemini in Different Disaster Scenarios.

IF 2.5 3区心理学 Q2 PSYCHOLOGY, CLINICAL

Journal of Clinical Psychology Pub Date : 2025-05-09 DOI:10.1002/jclp.23808

Jun Tat Tan, Rick Kye Gan, Carlos Alsua, Mark Peterson, Ricardo Úbeda Sales, Ann Zee Gan, José Antonio Cernuda Martínez, Pedro Arcos González

{"title":"Psychological First Aid by AI: Proof-of-Concept and Comparative Performance of ChatGPT-4 and Gemini in Different Disaster Scenarios.","authors":"Jun Tat Tan, Rick Kye Gan, Carlos Alsua, Mark Peterson, Ricardo Úbeda Sales, Ann Zee Gan, José Antonio Cernuda Martínez, Pedro Arcos González","doi":"10.1002/jclp.23808","DOIUrl":null,"url":null,"abstract":"Objective: This study aimed to evaluate the performance and proof-of-concept of psychological first aid (PFA) provided by two AI chatbots, ChatGPT-4 and Gemini.Methods: A mixed-method cross-sectional analysis was conducted using validated PFA scenarios from the Institute for Disaster Mental Health. Five scenarios representing different disaster contexts were selected. Data were collected by prompting both chatbots to perform PFA based on these scenarios. Quantitative performance was assessed using the PFA principles of Look, Listen, and Link, with scores assigned using IFRC's PFA scoring template. Qualitative analysis involved content analysis for AI hallucination, coding responses, and thematic analysis to identify key subthemes and themes.Results: ChatGPT-4 outperformed Gemini, achieving an overall score of 90% (CI: 86%-93%) compared to Gemini's 73% (CI: 67%-79%), a statistically significant difference (p = 0.01). In the Look domain, ChatGPT-4 scored higher (p = 0.02), while both performed equally in the Listen and Link domain. The content analysis of AI hallucinations reveals that ChatGPT-4 has a relative frequency of 18.4% (CI: 12%-25%), while Gemini exhibits a relative frequency of 50.0% (CI: 26.6%-71.3%), (p < 0.01). Five themes emerged from the qualitative analysis: Look, Listen, Link, Professionalism, Mental Health, and Psychosocial Support.Conclusion: ChatGPT-4 demonstrated superior performance in providing PFA compared to Gemini. While AI chatbots show potential as supportive tools for PFA providers, concerns regarding AI hallucinations highlight the need for cautious implementation. Further research is necessary to enhance the reliability and safety of AI-assisted PFA, particularly by eliminating hallucinations, and to integrate the current advances in voice-based chatbot functionality.","PeriodicalId":15395,"journal":{"name":"Journal of Clinical Psychology","volume":" ","pages":""},"PeriodicalIF":2.5000,"publicationDate":"2025-05-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Clinical Psychology","FirstCategoryId":"102","ListUrlMain":"https://doi.org/10.1002/jclp.23808","RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"PSYCHOLOGY, CLINICAL","Score":null,"Total":0}

引用次数: 0

Abstract

Objective: This study aimed to evaluate the performance and proof-of-concept of psychological first aid (PFA) provided by two AI chatbots, ChatGPT-4 and Gemini.

Methods: A mixed-method cross-sectional analysis was conducted using validated PFA scenarios from the Institute for Disaster Mental Health. Five scenarios representing different disaster contexts were selected. Data were collected by prompting both chatbots to perform PFA based on these scenarios. Quantitative performance was assessed using the PFA principles of Look, Listen, and Link, with scores assigned using IFRC's PFA scoring template. Qualitative analysis involved content analysis for AI hallucination, coding responses, and thematic analysis to identify key subthemes and themes.

Results: ChatGPT-4 outperformed Gemini, achieving an overall score of 90% (CI: 86%-93%) compared to Gemini's 73% (CI: 67%-79%), a statistically significant difference (p = 0.01). In the Look domain, ChatGPT-4 scored higher (p = 0.02), while both performed equally in the Listen and Link domain. The content analysis of AI hallucinations reveals that ChatGPT-4 has a relative frequency of 18.4% (CI: 12%-25%), while Gemini exhibits a relative frequency of 50.0% (CI: 26.6%-71.3%), (p < 0.01). Five themes emerged from the qualitative analysis: Look, Listen, Link, Professionalism, Mental Health, and Psychosocial Support.

Conclusion: ChatGPT-4 demonstrated superior performance in providing PFA compared to Gemini. While AI chatbots show potential as supportive tools for PFA providers, concerns regarding AI hallucinations highlight the need for cautious implementation. Further research is necessary to enhance the reliability and safety of AI-assisted PFA, particularly by eliminating hallucinations, and to integrate the current advances in voice-based chatbot functionality.

查看原文本刊更多论文

人工智能的心理急救：ChatGPT-4和Gemini在不同灾难场景中的概念验证和比较性能。

目的：本研究旨在评估两种人工智能聊天机器人ChatGPT-4和Gemini提供的心理急救（PFA）的性能和概念验证。方法：采用混合方法进行横断面分析，使用来自灾难心理健康研究所的经过验证的PFA情景。选择了代表不同灾难背景的五个场景。通过提示两个聊天机器人基于这些场景执行PFA来收集数据。使用PFA的“看、听和联系”原则对定量表现进行评估，并使用IFRC的PFA评分模板分配分数。定性分析包括AI幻觉的内容分析、编码反应和主题分析，以确定关键的子主题和主题。结果：ChatGPT-4的表现优于Gemini，总分为90% (CI: 86%-93%)， Gemini为73% (CI: 67%-79%)，差异有统计学意义（p = 0.01）。在Look领域，ChatGPT-4得分更高（p = 0.02），而两者在Listen和Link领域的表现相同。AI幻觉的内容分析显示，ChatGPT-4的相对频率为18.4% (CI: 12%-25%)，而Gemini的相对频率为50.0% （CI: 26.6%-71.3%）。(p结论：ChatGPT-4在提供PFA方面表现优于Gemini。虽然人工智能聊天机器人显示出作为PFA提供商支持工具的潜力，但对人工智能幻觉的担忧凸显了谨慎实施的必要性。需要进一步的研究来提高人工智能辅助PFA的可靠性和安全性，特别是通过消除幻觉，并整合当前基于语音的聊天机器人功能的进展。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Journal of Clinical Psychology PSYCHOLOGY, CLINICAL-

CiteScore

5.40

自引率

3.30%

发文量

177

期刊介绍： Founded in 1945, the Journal of Clinical Psychology is a peer-reviewed forum devoted to research, assessment, and practice. Published eight times a year, the Journal includes research studies; articles on contemporary professional issues, single case research; brief reports (including dissertations in brief); notes from the field; and news and notes. In addition to papers on psychopathology, psychodiagnostics, and the psychotherapeutic process, the journal welcomes articles focusing on psychotherapy effectiveness research, psychological assessment and treatment matching, clinical outcomes, clinical health psychology, and behavioral medicine.