A Thermal Camera-based Activity Recognition Using Discriminant Skeleton Features and RNN

Recognizing human activities from sensor data is one of the key areas of image processing, computer vision, and pattern recognition researches today. The target of human activity recognition (HAR) is usually to detect and analyze distinguished activities from the data acquired via different sensors (e.g. thermal cameras). This work proposes a HAR approach from videos recorded via a thermal camera. Skeletons of human bodies are extracted from thermal frames using an opensource deep convolutional neural network (CNN)-based approach named OpenPose. It is generally applied on videos of typical color cameras. However, this work adopts OpenPose on thermal images to extract useful features so that the HAR system can be deployed in environments with low lights as well. Once skeletons of human silhouettes are obtained from the thermal images, robust spatiotemporal features are extracted followed by discriminant analysis. Finally, the discriminant features are fed into a deep recurrent neural network (RNN) for activity training and recognition. The proposed HAR method can be applied to monitor the users such as elderly in both bright and dark environments to prolong their independent life, unlike other typical color cameras which are generally applied in bright environments.