Development and validation of a cognitive behavioral therapy for psychosis online training with automated feedback.

IF 2.6 2区 心理学 Q2 PSYCHOLOGY, CLINICAL
Psychotherapy Pub Date : 2025-03-01 DOI:10.1037/pst0000548
Sarah L Kopelovich, Rachel M Brian, Mike Tanana, Roisín Slevin, Brian Pace, Shannon K Stewart, Victoria Shepard, Dror Ben-Zeev, Scott A Baldwin, Christina S Soma, Sarah Stanco, Zac Imel
{"title":"Development and validation of a cognitive behavioral therapy for psychosis online training with automated feedback.","authors":"Sarah L Kopelovich, Rachel M Brian, Mike Tanana, Roisín Slevin, Brian Pace, Shannon K Stewart, Victoria Shepard, Dror Ben-Zeev, Scott A Baldwin, Christina S Soma, Sarah Stanco, Zac Imel","doi":"10.1037/pst0000548","DOIUrl":null,"url":null,"abstract":"<p><p>The accessibility of training and fidelity assessment is critical to implementing and sustaining empirically supported psychotherapies like cognitive behavioral therapy for psychosis (CBTp). We describe the development of an online CBTp training tool that incorporates behavioral rehearsal tasks to enable deliberate practice of cognitive and behavioral techniques for psychosis. The development process consisted of designing content, inclusive of didactics, client profiles, and learner prompts; constructing standardized performance tasks and metrics; collecting responses to learner prompts; establishing intraclass correlation (ICC) of responses among trained raters; and training a transformer-based machine learning (ML) model to meet or surpass human ICC. Authenticity ratings of each simulated client surpassed benchmarks. CBTp trainers (<i>n</i> = 12), clinicians (<i>n</i> = 78), and nonclinicians (<i>n</i> = 119) generated 3,958 unique verbal responses to 28 unique prompts (7 skills × 4 simulated clients), of which the coding team rated 1,961. Human ICC across all skills was high (mean ICC = 0.77). On average, there was a high correlation between ML and human ratings of fidelity (<i>r<sub>s</sub></i> = .74). Similarly, the average percentage of human agreement was high at 96% (range = 87%-102%), where values greater than 100 indicate that the ML model agreed with a human rater more than two human raters agreed with each other. Results suggest that it is possible to reliably measure discrete CBTp skills in response to simulated client vignettes while capturing expected variation in skill utilization across participants. These findings pave the way for a standardized, asynchronous training that incorporates automated feedback on learners' rehearsal of CBTp skills. (PsycInfo Database Record (c) 2025 APA, all rights reserved).</p>","PeriodicalId":20910,"journal":{"name":"Psychotherapy","volume":"62 1","pages":"1-11"},"PeriodicalIF":2.6000,"publicationDate":"2025-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11921910/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Psychotherapy","FirstCategoryId":"102","ListUrlMain":"https://doi.org/10.1037/pst0000548","RegionNum":2,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"PSYCHOLOGY, CLINICAL","Score":null,"Total":0}
引用次数: 0

Abstract

The accessibility of training and fidelity assessment is critical to implementing and sustaining empirically supported psychotherapies like cognitive behavioral therapy for psychosis (CBTp). We describe the development of an online CBTp training tool that incorporates behavioral rehearsal tasks to enable deliberate practice of cognitive and behavioral techniques for psychosis. The development process consisted of designing content, inclusive of didactics, client profiles, and learner prompts; constructing standardized performance tasks and metrics; collecting responses to learner prompts; establishing intraclass correlation (ICC) of responses among trained raters; and training a transformer-based machine learning (ML) model to meet or surpass human ICC. Authenticity ratings of each simulated client surpassed benchmarks. CBTp trainers (n = 12), clinicians (n = 78), and nonclinicians (n = 119) generated 3,958 unique verbal responses to 28 unique prompts (7 skills × 4 simulated clients), of which the coding team rated 1,961. Human ICC across all skills was high (mean ICC = 0.77). On average, there was a high correlation between ML and human ratings of fidelity (rs = .74). Similarly, the average percentage of human agreement was high at 96% (range = 87%-102%), where values greater than 100 indicate that the ML model agreed with a human rater more than two human raters agreed with each other. Results suggest that it is possible to reliably measure discrete CBTp skills in response to simulated client vignettes while capturing expected variation in skill utilization across participants. These findings pave the way for a standardized, asynchronous training that incorporates automated feedback on learners' rehearsal of CBTp skills. (PsycInfo Database Record (c) 2025 APA, all rights reserved).

基于自动反馈的精神病认知行为疗法在线培训的开发与验证。
培训的可及性和保真度评估对于实施和维持经验支持的心理治疗(如精神病认知行为治疗(CBTp))至关重要。我们描述了一种在线CBTp培训工具的开发,该工具结合了行为排练任务,以实现针对精神病的认知和行为技术的刻意练习。开发过程包括设计内容,包括教学、客户简介和学习者提示;构建标准化的绩效任务和指标;收集对学习者提示的回应;在经过训练的评分者之间建立反应的类内相关性(ICC);以及训练一个基于变压器的机器学习(ML)模型,以达到或超过人类的ICC。每个模拟客户端的真实性评级都超过了基准。CBTp培训师(n = 12)、临床医生(n = 78)和非临床医生(n = 119)对28个独特的提示(7种技能× 4个模拟客户)产生了3958个独特的口头反应,编码团队对其中的1961个进行了评分。人类所有技能的ICC都很高(平均ICC = 0.77)。平均而言,机器学习与人类的保真度评级之间存在高度相关性(rs = 0.74)。同样,人类同意的平均百分比很高,达到96%(范围= 87%-102%),其中值大于100表明ML模型与人类评分者的一致程度超过两个人类评分者的一致程度。结果表明,可以可靠地测量离散的CBTp技能,以响应模拟的客户场景,同时捕获参与者技能利用的预期变化。这些发现为标准化的异步训练铺平了道路,这种训练结合了学习者对CBTp技能的自动反馈。(PsycInfo Database Record (c) 2025 APA,版权所有)。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
Psychotherapy
Psychotherapy PSYCHOLOGY, CLINICAL-
CiteScore
4.60
自引率
12.00%
发文量
93
期刊介绍: Psychotherapy Theory, Research, Practice, Training publishes a wide variety of articles relevant to the field of psychotherapy. The journal strives to foster interactions among individuals involved with training, practice theory, and research since all areas are essential to psychotherapy. This journal is an invaluable resource for practicing clinical and counseling psychologists, social workers, and mental health professionals.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信