Human-object Interaction Recognition Using Multitask Neural Network

Human behavior recognition is a popular research area in the field of computer vision and has been studied due to its important applications such as visual surveillance and video retrieval. In this paper, we propose a new approach for recognizing human-object interaction actions based on multitask 2D convolutional neural network, which combines human body motion, human hand motion and object recognition network. By using RGBD camera and digital gloves, refined recognition of human body and hand movements are collected and learned. In addition, a new object recognition network based on YOLOv3 is introduced which increases the accuracy of predicting human-object interaction labels. We designed eight representative actions and built our own data set containing body and accurate hand motions. In our experiment, the accuracy of recognizing interactive actions reached 93%, which shows the correctness and effectiveness of the multitasking framework we propose.