Nils Hentati Isacsson , Kirsten Zantvoort , Erik Forsell , Magnus Boman , Viktor Kaldo
{"title":"Making the most out of timeseries symptom data: A machine learning study on symptom predictions of internet-based CBT","authors":"Nils Hentati Isacsson , Kirsten Zantvoort , Erik Forsell , Magnus Boman , Viktor Kaldo","doi":"10.1016/j.invent.2024.100773","DOIUrl":null,"url":null,"abstract":"<div><h3>Objective</h3><p>Predicting who will not benefit enough from Internet-Based Cognitive Behavioral (ICBT) Therapy early on can assist in better allocation of limited mental health care resources. Repeated measures of symptoms during treatment is the strongest predictor of outcome, and we want to investigate if methods that explicitly account for time-dependency are superior to methods that do not, with data from (a) only two pre-treatment timepoints and (b) the pre-treatment timepoints and three timepoints during initial treatment.</p></div><div><h3>Methods</h3><p>We use 1) commonly used time-independent methods (i.e., Linear Regression and Random Forest models) and 2) time-dependent methods (i.e., multilevel model regression, mixed-effects random forest, and a Long Short-Term Memory model) to predict symptoms during treatment, including the final outcome. This is done with symptom scores from 6436 ICBT patients from regular care, using robust multiple imputation and nested cross-validation methods.</p></div><div><h3>Results</h3><p>The models had a 14 %–12 % root mean squared error (RMSE) in predicting the post-treatment outcome, corresponding to a balanced accuracy of 67–74 %. Time-dependent models did not have higher accuracies. Using data for the initial treatment period (b) instead of only from before treatment (a) increased prediction results by 1.3 % percentage points (12 % to 10.7 %) RMSE and 6 % percentage points BACC (69 % to 75 %).</p></div><div><h3>Conclusion</h3><p>Training prediction models on only symptom scores of the first few weeks is a promising avenue for symptom predictions in treatment, regardless of which model is used. Further research is necessary to better understand the interaction between model complexity, dataset length and width, and the prediction tasks at hand.</p></div>","PeriodicalId":48615,"journal":{"name":"Internet Interventions-The Application of Information Technology in Mental and Behavioural Health","volume":null,"pages":null},"PeriodicalIF":3.6000,"publicationDate":"2024-09-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S2214782924000666/pdfft?md5=1990aa321535e76c003880576de54367&pid=1-s2.0-S2214782924000666-main.pdf","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Internet Interventions-The Application of Information Technology in Mental and Behavioural Health","FirstCategoryId":"3","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S2214782924000666","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"HEALTH CARE SCIENCES & SERVICES","Score":null,"Total":0}
引用次数: 0
Abstract
Objective
Predicting who will not benefit enough from Internet-Based Cognitive Behavioral (ICBT) Therapy early on can assist in better allocation of limited mental health care resources. Repeated measures of symptoms during treatment is the strongest predictor of outcome, and we want to investigate if methods that explicitly account for time-dependency are superior to methods that do not, with data from (a) only two pre-treatment timepoints and (b) the pre-treatment timepoints and three timepoints during initial treatment.
Methods
We use 1) commonly used time-independent methods (i.e., Linear Regression and Random Forest models) and 2) time-dependent methods (i.e., multilevel model regression, mixed-effects random forest, and a Long Short-Term Memory model) to predict symptoms during treatment, including the final outcome. This is done with symptom scores from 6436 ICBT patients from regular care, using robust multiple imputation and nested cross-validation methods.
Results
The models had a 14 %–12 % root mean squared error (RMSE) in predicting the post-treatment outcome, corresponding to a balanced accuracy of 67–74 %. Time-dependent models did not have higher accuracies. Using data for the initial treatment period (b) instead of only from before treatment (a) increased prediction results by 1.3 % percentage points (12 % to 10.7 %) RMSE and 6 % percentage points BACC (69 % to 75 %).
Conclusion
Training prediction models on only symptom scores of the first few weeks is a promising avenue for symptom predictions in treatment, regardless of which model is used. Further research is necessary to better understand the interaction between model complexity, dataset length and width, and the prediction tasks at hand.
期刊介绍:
Official Journal of the European Society for Research on Internet Interventions (ESRII) and the International Society for Research on Internet Interventions (ISRII).
The aim of Internet Interventions is to publish scientific, peer-reviewed, high-impact research on Internet interventions and related areas.
Internet Interventions welcomes papers on the following subjects:
• Intervention studies targeting the promotion of mental health and featuring the Internet and/or technologies using the Internet as an underlying technology, e.g. computers, smartphone devices, tablets, sensors
• Implementation and dissemination of Internet interventions
• Integration of Internet interventions into existing systems of care
• Descriptions of development and deployment infrastructures
• Internet intervention methodology and theory papers
• Internet-based epidemiology
• Descriptions of new Internet-based technologies and experiments with clinical applications
• Economics of internet interventions (cost-effectiveness)
• Health care policy and Internet interventions
• The role of culture in Internet intervention
• Internet psychometrics
• Ethical issues pertaining to Internet interventions and measurements
• Human-computer interaction and usability research with clinical implications
• Systematic reviews and meta-analysis on Internet interventions