Human Activity Recognition From Motion and Acoustic Sensors Using Contrastive Learning

Abstract

In this paper, we formulate human activity recognition as a downstream task of pretrained multimodal contrastive learning (MCL) models and break the convention of the one-modality-to-one-modality contrastive paradigm by allowing the models to have more than one source modality. Different from the prevailing assumption in MCL that one source modality and one target modality are the counterparts of each other, this work considers the possibility where it takes multiple source modalities with complementary information to match up to a target modality. In particular, we leverage a large-scale pretrained audio-language contrastive model and extend it to accepting IMU and audio input. The experiment results indicate the superiority of using complementary source modalities over using any source modality alone with 10.3% to 35.0% performance gain.