Multimodal Cross-Attention Mechanism-Based Algorithm for Elderly Behavior Monitoring and Recognition

Abstract

In contrast to the general population, behavior recognition among the elderly poses increased specificity and difficulty, rendering the reliability and usability aspects of safety monitoring systems for the elderly more challenging. Hence, this study proposes a multi-modal perception-based solution for an elderly safety monitoring recognition system. The proposed approach introduces a recognition algorithm based on multi-modal cross-attention mechanism, innovatively incorporating complex information such as scene context and voice to achieve more accurate behavior recognition. By fusing four modalities, namely image, skeleton, sensor data, and audio, we further enhance the accuracy of recognition. Additionally, we introduce a novel human-robot interaction mode, where the system associates directly recognized intentions with robotic actions without explicit commands, delivering a more natural and efficient elderly assistance paradigm. This mode not only elevates the level of safety monitoring for the elderly but also facilitates a more natural and efficient caregiving approach. Experimental results demonstrate significant improvement in recognition accuracy for 11 typical elderly behaviors compared to existing methods.