A Two-Stream Deep-Learning Network for Heart Rate Estimation From Facial Image Sequence

IF 4.3 2区 综合性期刊 Q1 ENGINEERING, ELECTRICAL & ELECTRONIC
Wen-Nung Lie;Dao Q. Le;Po-Han Huang;Guan-Hao Fu;Anh Nguyen Thi Quynh;Quynh Nguyen Quang Nhu
{"title":"A Two-Stream Deep-Learning Network for Heart Rate Estimation From Facial Image Sequence","authors":"Wen-Nung Lie;Dao Q. Le;Po-Han Huang;Guan-Hao Fu;Anh Nguyen Thi Quynh;Quynh Nguyen Quang Nhu","doi":"10.1109/JSEN.2024.3483629","DOIUrl":null,"url":null,"abstract":"This article presents a deep-learning-based two-stream network to estimate remote Photoplethysmogram (rPPG) signal and hence derive the heart rate (HR) from an RGB facial video. Our proposed network employs temporal modulation blocks (TMBs) to efficiently extract temporal dependencies and spatial attention blocks on a mean frame to learn spatial features. Our TMBs are composed of two subblocks that can simultaneously learn overall and channelwise spatiotemporal features, which are pivotal for the task. Data augmentation (DA) in training and multiple redundant estimations for noise removal in testing were also designed to make the training more effective and the inference more robust. Experimental results show that the proposed temporal shift-channelwise spatio-temporal network (TS-CST Net) has reached competitive and even superior performances among the state-of-the-art (SOTA) methods on four popular datasets, showcasing our network’s learning capability.","PeriodicalId":447,"journal":{"name":"IEEE Sensors Journal","volume":"24 24","pages":"42343-42351"},"PeriodicalIF":4.3000,"publicationDate":"2024-10-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"IEEE Sensors Journal","FirstCategoryId":"103","ListUrlMain":"https://ieeexplore.ieee.org/document/10735090/","RegionNum":2,"RegionCategory":"综合性期刊","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"ENGINEERING, ELECTRICAL & ELECTRONIC","Score":null,"Total":0}
引用次数: 0

Abstract

This article presents a deep-learning-based two-stream network to estimate remote Photoplethysmogram (rPPG) signal and hence derive the heart rate (HR) from an RGB facial video. Our proposed network employs temporal modulation blocks (TMBs) to efficiently extract temporal dependencies and spatial attention blocks on a mean frame to learn spatial features. Our TMBs are composed of two subblocks that can simultaneously learn overall and channelwise spatiotemporal features, which are pivotal for the task. Data augmentation (DA) in training and multiple redundant estimations for noise removal in testing were also designed to make the training more effective and the inference more robust. Experimental results show that the proposed temporal shift-channelwise spatio-temporal network (TS-CST Net) has reached competitive and even superior performances among the state-of-the-art (SOTA) methods on four popular datasets, showcasing our network’s learning capability.
基于人脸图像序列的心率估计双流深度学习网络
本文提出了一种基于深度学习的双流网络来估计远程光电容积图(rPPG)信号,从而从RGB面部视频中得出心率(HR)。我们提出的网络采用时间调制块(TMBs)在平均帧上有效地提取时间依赖性和空间注意块来学习空间特征。我们的TMBs由两个子块组成,它们可以同时学习整体和通道的时空特征,这对任务至关重要。为了提高训练的有效性和推理的鲁棒性,还设计了训练中的数据增强(DA)和测试中的多重冗余估计去噪。实验结果表明,在四个流行的数据集上,所提出的时间移信道时空网络(TS-CST Net)在最先进的(SOTA)方法中达到了竞争甚至优越的性能,展示了我们的网络的学习能力。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
IEEE Sensors Journal
IEEE Sensors Journal 工程技术-工程:电子与电气
CiteScore
7.70
自引率
14.00%
发文量
2058
审稿时长
5.2 months
期刊介绍: The fields of interest of the IEEE Sensors Journal are the theory, design , fabrication, manufacturing and applications of devices for sensing and transducing physical, chemical and biological phenomena, with emphasis on the electronics and physics aspect of sensors and integrated sensors-actuators. IEEE Sensors Journal deals with the following: -Sensor Phenomenology, Modelling, and Evaluation -Sensor Materials, Processing, and Fabrication -Chemical and Gas Sensors -Microfluidics and Biosensors -Optical Sensors -Physical Sensors: Temperature, Mechanical, Magnetic, and others -Acoustic and Ultrasonic Sensors -Sensor Packaging -Sensor Networks -Sensor Applications -Sensor Systems: Signals, Processing, and Interfaces -Actuators and Sensor Power Systems -Sensor Signal Processing for high precision and stability (amplification, filtering, linearization, modulation/demodulation) and under harsh conditions (EMC, radiation, humidity, temperature); energy consumption/harvesting -Sensor Data Processing (soft computing with sensor data, e.g., pattern recognition, machine learning, evolutionary computation; sensor data fusion, processing of wave e.g., electromagnetic and acoustic; and non-wave, e.g., chemical, gravity, particle, thermal, radiative and non-radiative sensor data, detection, estimation and classification based on sensor data) -Sensors in Industrial Practice
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信