Yvonne W Leung, Elise Wouterloot, Achini Adikari, Jinny Hong, Veenaajaa Asokan, Lauren Duan, Claire Lam, Carlina Kim, Kai P Chan, Daswin De Silva, Lianne Trachtenberg, Heather Rennie, Jiahui Wong, Mary Jane Esplen
{"title":"基于人工智能的共同协调人 (AICF) 用于检测和监控基于网络的癌症互助小组中的小组凝聚力结果：单臂试验研究。","authors":"Yvonne W Leung, Elise Wouterloot, Achini Adikari, Jinny Hong, Veenaajaa Asokan, Lauren Duan, Claire Lam, Carlina Kim, Kai P Chan, Daswin De Silva, Lianne Trachtenberg, Heather Rennie, Jiahui Wong, Mary Jane Esplen","doi":"10.2196/43070","DOIUrl":null,"url":null,"abstract":"<p><strong>Background: </strong>Commonly offered as supportive care, therapist-led online support groups (OSGs) are a cost-effective way to provide support to individuals affected by cancer. One important indicator of a successful OSG session is group cohesion; however, monitoring group cohesion can be challenging due to the lack of nonverbal cues and in-person interactions in text-based OSGs. The Artificial Intelligence-based Co-Facilitator (AICF) was designed to contextually identify therapeutic outcomes from conversations and produce real-time analytics.</p><p><strong>Objective: </strong>The aim of this study was to develop a method to train and evaluate AICF's capacity to monitor group cohesion.</p><p><strong>Methods: </strong>AICF used a text classification approach to extract the mentions of group cohesion within conversations. A sample of data was annotated by human scorers, which was used as the training data to build the classification model. The annotations were further supported by finding contextually similar group cohesion expressions using word embedding models as well. AICF performance was also compared against the natural language processing software Linguistic Inquiry Word Count (LIWC).</p><p><strong>Results: </strong>AICF was trained on 80,000 messages obtained from Cancer Chat Canada. We tested AICF on 34,048 messages. Human experts scored 6797 (20%) of the messages to evaluate the ability of AICF to classify group cohesion. Results showed that machine learning algorithms combined with human input could detect group cohesion, a clinically meaningful indicator of effective OSGs. After retraining with human input, AICF reached an F<sub>1</sub>-score of 0.82. AICF performed slightly better at identifying group cohesion compared to LIWC.</p><p><strong>Conclusions: </strong>AICF has the potential to assist therapists by detecting discord in the group amenable to real-time intervention. Overall, AICF presents a unique opportunity to strengthen patient-centered care in web-based settings by attending to individual needs.</p><p><strong>International registered report identifier (irrid): </strong>RR2-10.2196/21453.</p>","PeriodicalId":45538,"journal":{"name":"JMIR Cancer","volume":null,"pages":null},"PeriodicalIF":3.3000,"publicationDate":"2024-07-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11301110/pdf/","citationCount":"0","resultStr":"{\"title\":\"Artificial Intelligence-Based Co-Facilitator (AICF) for Detecting and Monitoring Group Cohesion Outcomes in Web-Based Cancer Support Groups: Single-Arm Trial Study.\",\"authors\":\"Yvonne W Leung, Elise Wouterloot, Achini Adikari, Jinny Hong, Veenaajaa Asokan, Lauren Duan, Claire Lam, Carlina Kim, Kai P Chan, Daswin De Silva, Lianne Trachtenberg, Heather Rennie, Jiahui Wong, Mary Jane Esplen\",\"doi\":\"10.2196/43070\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<p><strong>Background: </strong>Commonly offered as supportive care, therapist-led online support groups (OSGs) are a cost-effective way to provide support to individuals affected by cancer. One important indicator of a successful OSG session is group cohesion; however, monitoring group cohesion can be challenging due to the lack of nonverbal cues and in-person interactions in text-based OSGs. The Artificial Intelligence-based Co-Facilitator (AICF) was designed to contextually identify therapeutic outcomes from conversations and produce real-time analytics.</p><p><strong>Objective: </strong>The aim of this study was to develop a method to train and evaluate AICF's capacity to monitor group cohesion.</p><p><strong>Methods: </strong>AICF used a text classification approach to extract the mentions of group cohesion within conversations. A sample of data was annotated by human scorers, which was used as the training data to build the classification model. The annotations were further supported by finding contextually similar group cohesion expressions using word embedding models as well. AICF performance was also compared against the natural language processing software Linguistic Inquiry Word Count (LIWC).</p><p><strong>Results: </strong>AICF was trained on 80,000 messages obtained from Cancer Chat Canada. We tested AICF on 34,048 messages. Human experts scored 6797 (20%) of the messages to evaluate the ability of AICF to classify group cohesion. Results showed that machine learning algorithms combined with human input could detect group cohesion, a clinically meaningful indicator of effective OSGs. After retraining with human input, AICF reached an F<sub>1</sub>-score of 0.82. AICF performed slightly better at identifying group cohesion compared to LIWC.</p><p><strong>Conclusions: </strong>AICF has the potential to assist therapists by detecting discord in the group amenable to real-time intervention. Overall, AICF presents a unique opportunity to strengthen patient-centered care in web-based settings by attending to individual needs.</p><p><strong>International registered report identifier (irrid): </strong>RR2-10.2196/21453.</p>\",\"PeriodicalId\":45538,\"journal\":{\"name\":\"JMIR Cancer\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":3.3000,\"publicationDate\":\"2024-07-22\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11301110/pdf/\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"JMIR Cancer\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.2196/43070\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q2\",\"JCRName\":\"ONCOLOGY\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"JMIR Cancer","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.2196/43070","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"ONCOLOGY","Score":null,"Total":0}
引用次数: 0
摘要
背景：由治疗师主导的在线支持小组（OSG）是为癌症患者提供支持的一种经济有效的方式，通常作为支持性护理提供。OSG会议成功与否的一个重要指标是小组凝聚力；然而，由于基于文本的OSG缺乏非语言线索和人际互动，监测小组凝聚力可能具有挑战性。基于人工智能的共同主持人（AICF）旨在从对话中识别治疗结果并进行实时分析：本研究旨在开发一种方法来训练和评估 AICF 监测群体凝聚力的能力：AICF 采用文本分类方法提取对话中提及的群体凝聚力。人类评分员对数据样本进行了注释，并将其作为建立分类模型的训练数据。此外，还使用单词嵌入模型查找上下文相似的群组内聚表达，为注释提供进一步支持。AICF 的性能还与自然语言处理软件 Linguistic Inquiry Word Count (LIWC) 进行了比较：AICF 在从加拿大癌症聊天室获取的 80,000 条信息上进行了训练。我们对 34,048 条信息进行了 AICF 测试。人类专家对 6797 条（20%）信息进行了评分，以评估 AICF 对群体凝聚力进行分类的能力。结果表明，机器学习算法与人工输入相结合，可以检测出群体凝聚力，这是有效 OSGs 的一个具有临床意义的指标。在使用人工输入进行再训练后，AICF 的 F1 分数达到了 0.82。与LIWC相比，AICF在识别群体凝聚力方面的表现略胜一筹：AICF 有可能通过检测群体中的不和谐因素来协助治疗师进行实时干预。总之，AICF 提供了一个独特的机会，通过关注个人需求，在基于网络的环境中加强以患者为中心的护理：RR2-10.2196/21453。
Artificial Intelligence-Based Co-Facilitator (AICF) for Detecting and Monitoring Group Cohesion Outcomes in Web-Based Cancer Support Groups: Single-Arm Trial Study.
Background: Commonly offered as supportive care, therapist-led online support groups (OSGs) are a cost-effective way to provide support to individuals affected by cancer. One important indicator of a successful OSG session is group cohesion; however, monitoring group cohesion can be challenging due to the lack of nonverbal cues and in-person interactions in text-based OSGs. The Artificial Intelligence-based Co-Facilitator (AICF) was designed to contextually identify therapeutic outcomes from conversations and produce real-time analytics.
Objective: The aim of this study was to develop a method to train and evaluate AICF's capacity to monitor group cohesion.
Methods: AICF used a text classification approach to extract the mentions of group cohesion within conversations. A sample of data was annotated by human scorers, which was used as the training data to build the classification model. The annotations were further supported by finding contextually similar group cohesion expressions using word embedding models as well. AICF performance was also compared against the natural language processing software Linguistic Inquiry Word Count (LIWC).
Results: AICF was trained on 80,000 messages obtained from Cancer Chat Canada. We tested AICF on 34,048 messages. Human experts scored 6797 (20%) of the messages to evaluate the ability of AICF to classify group cohesion. Results showed that machine learning algorithms combined with human input could detect group cohesion, a clinically meaningful indicator of effective OSGs. After retraining with human input, AICF reached an F1-score of 0.82. AICF performed slightly better at identifying group cohesion compared to LIWC.
Conclusions: AICF has the potential to assist therapists by detecting discord in the group amenable to real-time intervention. Overall, AICF presents a unique opportunity to strengthen patient-centered care in web-based settings by attending to individual needs.
International registered report identifier (irrid): RR2-10.2196/21453.
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号
