Learning Frame-Level Classifiers for Video-Based Real-Time Assessment of Stroke Rehabilitation Exercises From Weakly Annotated Datasets

IF 5.2 2区 医学 Q2 ENGINEERING, BIOMEDICAL
Ana Rita Cóias;Min Hun Lee;Alexandre Bernardino;Asim Smailagic;Mariana Mateus;David Fernandes;Sofia Trapola
{"title":"Learning Frame-Level Classifiers for Video-Based Real-Time Assessment of Stroke Rehabilitation Exercises From Weakly Annotated Datasets","authors":"Ana Rita Cóias;Min Hun Lee;Alexandre Bernardino;Asim Smailagic;Mariana Mateus;David Fernandes;Sofia Trapola","doi":"10.1109/TNSRE.2025.3602548","DOIUrl":null,"url":null,"abstract":"Autonomous rehabilitation support solutions, such as virtual coaches, should provide real-time feedback to improve motor function and maintain patient engagement. However, fully annotated dataset collection for real-time exercise assessment is time-consuming and costly, posing a barrier to evaluating proposed methods. In this work, we present a novel framework that learns a frame-level classifier using weakly annotated videos for real-time assessment of compensatory motions in stroke rehabilitation exercises by generating pseudo-labels at a frame level. We consider three approaches: 1) a baseline approach that uses a source dataset to train a frame-level classifier, 2) a transfer learning approach that uses target dataset video-level labels and parameters learned from a source dataset with frame-level labels, and 3) a semi-supervised approach that leverages a target dataset video-level labels and a small set of frame-level labels. We intend to generalize to a weakly labeled target dataset with new exercises and patients. To validate the approach, we use two datasets annotated on compensatory motions: TULE, an existing video and frame-level labeled dataset of 15 post-stroke patients and three exercises, and SERE, a new dataset of 20 post-stroke patients and five exercises, created by the authors, with video-level labels and a small amount of frame-level labels. We show that a frame-level classifier trained on TULE does not generalize well on SERE (<inline-formula> <tex-math>${f}_{{1}} = {72}.{87}\\%$ </tex-math></inline-formula>), but our semi-supervised and transfer learning approaches achieve, respectively, <inline-formula> <tex-math>${f}_{{1}} = {78}.{93}\\%$ </tex-math></inline-formula> and <inline-formula> <tex-math>${f}_{{1}} = {80}.{47}\\%$ </tex-math></inline-formula>. Generating pseudo-labels leads to better frame-level classification results for the target dataset than training a classifier with the source dataset (baseline). Thus, the proposed approach can simplify the customization of virtual coaches to new patients and exercises with low data annotation efforts.","PeriodicalId":13419,"journal":{"name":"IEEE Transactions on Neural Systems and Rehabilitation Engineering","volume":"33 ","pages":"3334-3345"},"PeriodicalIF":5.2000,"publicationDate":"2025-08-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=11141498","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"IEEE Transactions on Neural Systems and Rehabilitation Engineering","FirstCategoryId":"5","ListUrlMain":"https://ieeexplore.ieee.org/document/11141498/","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"ENGINEERING, BIOMEDICAL","Score":null,"Total":0}
引用次数: 0

Abstract

Autonomous rehabilitation support solutions, such as virtual coaches, should provide real-time feedback to improve motor function and maintain patient engagement. However, fully annotated dataset collection for real-time exercise assessment is time-consuming and costly, posing a barrier to evaluating proposed methods. In this work, we present a novel framework that learns a frame-level classifier using weakly annotated videos for real-time assessment of compensatory motions in stroke rehabilitation exercises by generating pseudo-labels at a frame level. We consider three approaches: 1) a baseline approach that uses a source dataset to train a frame-level classifier, 2) a transfer learning approach that uses target dataset video-level labels and parameters learned from a source dataset with frame-level labels, and 3) a semi-supervised approach that leverages a target dataset video-level labels and a small set of frame-level labels. We intend to generalize to a weakly labeled target dataset with new exercises and patients. To validate the approach, we use two datasets annotated on compensatory motions: TULE, an existing video and frame-level labeled dataset of 15 post-stroke patients and three exercises, and SERE, a new dataset of 20 post-stroke patients and five exercises, created by the authors, with video-level labels and a small amount of frame-level labels. We show that a frame-level classifier trained on TULE does not generalize well on SERE ( ${f}_{{1}} = {72}.{87}\%$ ), but our semi-supervised and transfer learning approaches achieve, respectively, ${f}_{{1}} = {78}.{93}\%$ and ${f}_{{1}} = {80}.{47}\%$ . Generating pseudo-labels leads to better frame-level classification results for the target dataset than training a classifier with the source dataset (baseline). Thus, the proposed approach can simplify the customization of virtual coaches to new patients and exercises with low data annotation efforts.
从弱注释数据集学习基于视频的脑卒中康复训练实时评估的帧级分类器
自主康复支持解决方案,如虚拟教练,应该提供实时反馈,以改善运动功能并保持患者参与。然而,用于实时运动评估的完全注释数据集收集既耗时又昂贵,这对评估所提出的方法构成了障碍。在这项工作中,我们提出了一个新的框架,该框架使用弱注释视频学习帧级分类器,通过在帧级生成伪标签来实时评估卒中康复练习中的代偿运动。我们考虑了三种方法:1)使用源数据集训练帧级分类器的基线方法,2)使用从具有帧级标签的源数据集学习的目标数据集视频级标签和参数的迁移学习方法,以及3)利用目标数据集视频级标签和一小组帧级标签的半监督方法。我们打算推广到一个带有新运动和患者的弱标记目标数据集。为了验证该方法,我们使用了两个对代偿运动进行注释的数据集:TULE,一个现有的视频和帧级标记数据集,包含15名中风后患者和3个练习;SERE,一个由作者创建的新数据集,包含20名中风后患者和5个练习,具有视频级标签和少量帧级标签。我们证明了在TULE上训练的帧级分类器在SERE (${f}_{{1}} ={72})上不能很好地泛化。{87}\%$),但我们的半监督学习和迁移学习方法分别实现了${f}_{{1}} ={78}。{93}\%$和${f}_{{1}} ={80}。{47} \ % $。与使用源数据集(基线)训练分类器相比,生成伪标签会为目标数据集带来更好的帧级分类结果。因此,所提出的方法可以简化虚拟教练对新患者和练习的定制,并且数据注释的工作量很小。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
CiteScore
8.60
自引率
8.20%
发文量
479
审稿时长
6-12 weeks
期刊介绍: Rehabilitative and neural aspects of biomedical engineering, including functional electrical stimulation, acoustic dynamics, human performance measurement and analysis, nerve stimulation, electromyography, motor control and stimulation; and hardware and software applications for rehabilitation engineering and assistive devices.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信