Visible light human activity recognition driven by generative language model

Abstract

Visible light-based indoor Human Activity Recognition (HAR) rises as a promising approach, due to its ability to provide indoor illumination, privacy protection, and serve sensing purposes. However, current visible light HAR methods are primarily focused on classification of individual human activity, which falls short of naturally representing and contextual relations. In this paper, we extend the challenge to a cross-modal alignment task between visible light signals and textual descriptions, proposing a framework that leverages generative large language models (LLMs) to decode visible light feature representations into human activity descriptions through sequence-to-sequence modeling. We implement a prototype system of our method and build up a custom dataset. Experiments in real indoor space demonstrate that our method achieves effective natural language level HAR from visible light sensing system, it promotes the information fusion between visible light and natural language, bringing the intelligent physical information systems towards realistic application with the integration of the generative LLMs.