Towards real-time sensor-based human activity recognition: a re-parameterized multidimensional feature communication fusion framework

Abstract

Sensor-based human activity recognition (HAR) has become a research hotspot due to its far-reaching applications including health monitoring, smart home, and sports tracking. Deep learning, particularly convolutional neural networks (CNNs) with automatic feature extraction capabilities, has demonstrated outstanding recognition accuracy. However, some current deep learning-based HAR methods require substantial parameters and computation resources to pursue high accuracy, resulting in expensive resource and time costs. To handle the challenge, we propose a lightweight one-dimensional re-parameterization multidimensional feature communication fusion (ORepMFCF) framework for HAR, achieving an impressive trade-off between recognition accuracy and speed. The one-dimensional re-parameterization VGG (ORV) block is developed to decouple the trained multi-branch structure into a single path to improve the inference speed without compromising accuracy. The time-dependent information channel shuffle (TDICS) module is proposed to strengthen feature extraction and communication ability. The channel attention feature fusion (CAFF) module is presented to improve multidimensional feature fusion. The ORepMFCF achieves the accuracy of 98.60% on the WISDM, 89.38% on the UniMiB-SHAR, 86.43% on the OPPORTUNITY, and 96.26% on the MobiACT, respectively. The ORepMFCF requires fewer parameters and incurs less computational overhead with better recognition performance, which significantly surpasses the recent state-of-the-art HAR networks. The real-world inference time of the ORepMFCF is evaluated on a Raspberry Pi and a smartphone, demonstrating its effectiveness and practicality.