VirtualHAR: Virtual Sensing Device and Correlation-Based Learning Approach for Multiwearable Sensing Device-Based Human Activity Recognition

Human activity recognition (HAR) is a prominent research direction in ubiquitous computing. Current state-of-the-art HAR models achieve great success by learning the correlations between the regions of the body parts by using the attached sensing devices for feature extraction. However, explicitly computing the correlations between whole body parts and whole sub-body parts, which is crucial for extracting discriminatory features for some activities, has not been investigated due to the lack of sensing devices that capture the movements of the whole (sub-)body parts. This study proposes an effective yet lightweight VirtualHAR framework, which automatically models correlations between the whole body parts, whole sub-body parts, and regions based on the concept of virtual sensing devices. The VirtualHAR framework mainly encompasses three modules. The backbone feature extraction (BEF) module extracts the features from a physical sensing device, based on which the Multipurpose Correlations Learning module constructs virtual sensing devices for body parts and sub-body parts and then exploits the explicit correlations between body parts, sub-body parts as well as in regions by using their attached physical sensing devices. Finally, the global aggregation (GA) module learns the GA representation for each physical sensing device by collecting the learned correlated representation from each virtual sensing device and physical sensing device. Comprehensive experiments on benchmark HAR datasets and a resource-constrained device confirm that VirtualHAR outperforms SOTA models in recognition performance and computational complexity. Through thorough quantitative and qualitative analysis, we validate the proposed VirtualHAR framework’s effectiveness and efficiency.