Natural language processing models for predicting treatment outcomes in internet-delivered cognitive behavioral therapy

IF 4.1 2区医学 Q1 HEALTH CARE SCIENCES & SERVICES

Internet Interventions-The Application of Information Technology in Mental and Behavioural Health Pub Date : 2025-10-10 DOI:10.1016/j.invent.2025.100879

Nils Hentati Isacsson , Lucía Gómez-Zaragozá , Fehmi Ben Abdesslem , Magnus Boman , Viktor Kaldo

{"title":"Natural language processing models for predicting treatment outcomes in internet-delivered cognitive behavioral therapy","authors":"Nils Hentati Isacsson , Lucía Gómez-Zaragozá , Fehmi Ben Abdesslem , Magnus Boman , Viktor Kaldo","doi":"10.1016/j.invent.2025.100879","DOIUrl":null,"url":null,"abstract":"<div><h3>Objective</h3><div>Predicting treatment outcome has the potential to enhance Internet-delivered Cognitive Behavioral Therapy (ICBT). One aspect of guided ICBT is patient-therapist interaction through written messages. With Natural language processing (NLP) these could be leveraged to predict outcome; however current evidence is limited. This study investigates the predictive accuracy of NLP models for treatment outcomes and evaluates whether NLP provides additional predictive value beyond symptom variables.</div></div><div><h3>Methods</h3><div>Patient-therapist messages from 6613 patients undergoing 12 weeks of treatment were used to train three types of NLP models: Term Frequency-Inverse Document Frequency (TF-IDF), Bidirectional Encoder Representations from transformers (BERT), and BERT for Longer Text (BELT). These were trained both with and without symptom variables from the initial treatment period to predict post-treatment symptoms. A dummy model was also used, and a linear regression model acted as a symptoms only benchmark. Multiple imputation addressed missing data, and nested cross-validation was used.</div></div><div><h3>Results</h3><div>The symptom only model performed best. Only BERT outperformed the dummy model, achieving a Root Mean Squared Error (RMSE) of 0.17 compared to RMSE of 0.18. Adding symptom variables to the BERT model significantly increased its accuracy, but not the RMSE metric. The best linear regression benchmark based on symptoms only had a BACC of 70 % (F<sub>1</sub>-score of 0.66) which outperformed the BERT model with 60 % (F<sub>1</sub>: 0.55) and the combined BERT plus symptoms model achieved 68 % (F<sub>1</sub>: 0.62).</div></div><div><h3>Conclusion</h3><div>These initial findings indicate a small predictive value from patient-therapist written message interaction but added no value beyond using only symptoms to predict post-treatment symptoms. Further research is needed to refine NLP-methods for use in psychological treatment, and more accurately assess the predictive potential of text-based interactions during ICBT.</div></div>","PeriodicalId":48615,"journal":{"name":"Internet Interventions-The Application of Information Technology in Mental and Behavioural Health","volume":"42 ","pages":"Article 100879"},"PeriodicalIF":4.1000,"publicationDate":"2025-10-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Internet Interventions-The Application of Information Technology in Mental and Behavioural Health","FirstCategoryId":"3","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S2214782925000806","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"HEALTH CARE SCIENCES & SERVICES","Score":null,"Total":0}

引用次数: 0

Abstract

Objective

Predicting treatment outcome has the potential to enhance Internet-delivered Cognitive Behavioral Therapy (ICBT). One aspect of guided ICBT is patient-therapist interaction through written messages. With Natural language processing (NLP) these could be leveraged to predict outcome; however current evidence is limited. This study investigates the predictive accuracy of NLP models for treatment outcomes and evaluates whether NLP provides additional predictive value beyond symptom variables.

Methods

Patient-therapist messages from 6613 patients undergoing 12 weeks of treatment were used to train three types of NLP models: Term Frequency-Inverse Document Frequency (TF-IDF), Bidirectional Encoder Representations from transformers (BERT), and BERT for Longer Text (BELT). These were trained both with and without symptom variables from the initial treatment period to predict post-treatment symptoms. A dummy model was also used, and a linear regression model acted as a symptoms only benchmark. Multiple imputation addressed missing data, and nested cross-validation was used.

Results

The symptom only model performed best. Only BERT outperformed the dummy model, achieving a Root Mean Squared Error (RMSE) of 0.17 compared to RMSE of 0.18. Adding symptom variables to the BERT model significantly increased its accuracy, but not the RMSE metric. The best linear regression benchmark based on symptoms only had a BACC of 70 % (F₁-score of 0.66) which outperformed the BERT model with 60 % (F₁: 0.55) and the combined BERT plus symptoms model achieved 68 % (F₁: 0.62).

Conclusion

These initial findings indicate a small predictive value from patient-therapist written message interaction but added no value beyond using only symptoms to predict post-treatment symptoms. Further research is needed to refine NLP-methods for use in psychological treatment, and more accurately assess the predictive potential of text-based interactions during ICBT.

查看原文本刊更多论文

网络认知行为治疗中预测治疗结果的自然语言处理模型

目的预测治疗结果有可能增强互联网提供的认知行为治疗（ICBT）。指导性ICBT的一个方面是通过书面信息进行患者与治疗师的互动。有了自然语言处理（NLP），这些可以用来预测结果；然而，目前的证据有限。本研究探讨了NLP模型对治疗结果的预测准确性，并评估NLP是否提供了除症状变量之外的其他预测价值。方法使用6613例接受12周治疗的患者的患者-治疗师信息来训练三种类型的NLP模型：Term Frequency- inverse Document Frequency （TF-IDF）、双向编码器表示（BERT）和BERT for Longer Text （BELT）。对这些患者进行了有或无症状变量的训练，以预测治疗后的症状。还使用了虚拟模型，并使用线性回归模型作为仅症状基准。多重输入解决了缺失数据，并使用嵌套交叉验证。结果单纯症状模型效果最好。只有BERT优于虚拟模型，实现了0.17的均方根误差（RMSE），而RMSE为0.18。将症状变量添加到BERT模型中可以显著提高其准确性，但不能提高RMSE度量。基于症状的最佳线性回归基准的BACC仅为70% (F1-score为0.66)，优于BERT模型的60% (F1: 0.55)， BERT +症状联合模型达到68% （F1: 0.62）。结论这些初步发现表明，患者-治疗师书面信息互动的预测价值很小，但除了仅使用症状来预测治疗后症状之外，没有任何价值。需要进一步的研究来完善nlp方法用于心理治疗，并更准确地评估ICBT期间基于文本的互动的预测潜力。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Internet Interventions-The Application of Information Technology in Mental and Behavioural Health Medicine-Health Informatics

CiteScore

6.50

自引率

9.30%

发文量

审稿时长

6 weeks

期刊介绍： Official Journal of the European Society for Research on Internet Interventions (ESRII) and the International Society for Research on Internet Interventions (ISRII). The aim of Internet Interventions is to publish scientific, peer-reviewed, high-impact research on Internet interventions and related areas. Internet Interventions welcomes papers on the following subjects: • Intervention studies targeting the promotion of mental health and featuring the Internet and/or technologies using the Internet as an underlying technology, e.g. computers, smartphone devices, tablets, sensors • Implementation and dissemination of Internet interventions • Integration of Internet interventions into existing systems of care • Descriptions of development and deployment infrastructures • Internet intervention methodology and theory papers • Internet-based epidemiology • Descriptions of new Internet-based technologies and experiments with clinical applications • Economics of internet interventions (cost-effectiveness) • Health care policy and Internet interventions • The role of culture in Internet intervention • Internet psychometrics • Ethical issues pertaining to Internet interventions and measurements • Human-computer interaction and usability research with clinical implications • Systematic reviews and meta-analysis on Internet interventions