Remote heart rate measurement based on spatial–temporal self-attention

Remote photoplethysmography (rPPG), a non-contact technique for measuring heart rate, has gained significant traction due to its convenience and non-invasive nature. However, rPPG signals are inherently weak and exhibit regional variations across facial areas. Accurate heart rate estimation necessitates analysis of extended-duration video sequences (exceeding 100 frames). To address these challenges, this paper proposes a novel deep neural network (DNN) based on self-attention mechanism, named spatial–temporal self-attention network (STSA-Net). This convolution-free DNN primarily adopts a transformer architecture. Initially, differential processing is applied to the input video to accentuate frame-to-frame variations and amplify subtle rPPG signals. These differential frames are subsequently passed through a spatial self-attention encoding module. This module models spatial dependencies within facial regions, allowing the network to focus on informative areas while suppressing irrelevant noise. Following spatial encoding, the features are processed by a temporal self-attention module, which captures long-term dependencies across frames using transformer-based techniques. The proposed method is rigorously evaluated through comprehensive experiments, including ablation studies, intra-database evaluations, and cross-database comparisons, using three benchmark databases: UBFC-RPPG, PURE, and MAHNOB-HCI. Our model demonstrates performance on par with state-of-the-art methods for remote heart rate measurement, achieving a MAE of 0.4 bpm on PURE, 0.48 bpm on UBFC, and 3.75 bpm on MAHNOB-HCI in intra-database experiments, and 1.36 bpm (UBFC$\to$ PURE) and 1.27 bpm (PURE$\to$ UBFC) in cross-database experiments. Additionally, an outlier in the PURE database is identified, and its cause and impact are analyzed.