Learn Dynamic Facial Motion Representations Using Transformer Encoder

Abstract

Human face analysis is an essential topic in visual computing. Many of our daily applications, such as face-priority auto focus in camera, face-based identity verification, and TikTok stickers, are unattainable without face analysis techniques. In the past ten years, face-related visual computing tasks like face detection, face recognition, and facial expression classification have improved drastically in performance, benefiting from the rapid development of deep learning theory. This work explores how to model dynamic facial motion using a learning-based method. Our proposed model takes video clips containing customized facial motion as input and generates a uni-size vector (the embedding) as the output. We have inspected two different encoders–recurrent neural networks and transformers to extract the temporal features from the video clip. We collected our own facial motion analysis dataset because there is no suitable datasets for our facial motion analysis task. Although our domain-specific dataset is small compared to the well-known public datasets for ordinary face-related tasks, we adopt a transfer learning approach, and a data augmentation method (random trimming) to help the model converge. The experimental results show that the transformer-based encoder performs better than the RNN baseline, and the best F1-score with our validation data is 0.889.