Fine-grained activities recognition with coarse-grained labeled multi-modal data

Abstract

Fine-grained human activities recognition focuses on recognizing event- or action-level activities, which enables a new set of Internet-of-Things (IoT) applications such as behavior analysis. Prior work on fine-grained human activities recognition relies on supervised sensing, which makes the fine-grained labeling labor-intensive and difficult to scale up. On the other hand, it is much more practical to collect coarse-grained label at the level of activity of daily living (e.g., cooking, working), especially for real-world IoT systems. In this paper, we present a framework that learns fine-grained human activities recognition with coarse-grained labeled and a small amount of fine-grained labeled multi-modal data. Our system leverages the implicit physical knowledge on the hierarchy of the coarse- and fine-grained labels and conducts data-driven hierarchical learning that take into account the coarse-grained supervised prediction for fine-grained semi-supervised learning. We evaluated our framework and CFR-TSVM algorithm on the data gathered from real-world experiments. Results show that our CFR-TSVM achieved an 81% recognition accuracy over 10 fine-grained activities, which reduces the prediction error of the semi-supervised learning baseline TSVM by half.