Multi-Level semantic and spatial hierarchy reasoning for 3D face reconstruction and dense alignment in unconstrained environments

3D face reconstruction and dense alignment tasks encounter significant challenges when the samples are in highly unconstrained environments, particularly with large poses, extreme expressions, occlusions and complex backgrounds. Although carefully designing the neural network architecture can enhance the representation ability, its performance is still unsatisfactory due to the absence of accurate facial semantic and spatial hierarchy information. To address this challenge, we propose an approach that uses neural networks to capture multi-level facial semantic and spatial hierarchy knowledge so as to guide the learning process. Specifically, our approach, referred to as Multi-Level Semantic and Spatial Hierarchy Reasoning Network (MSHRNet), leverages a point-to-space level progressive face structure loss function to precisely learn the semantic and spatial hierarchy knowledge of different facial parts. This knowledge is then injected into the backbone network through multi-level hierarchy knowledge matrices to incorporate structural reasoning knowledge, which can suppress the effects of large poses, extreme expressions, and occlusions. Moreover, we introduce a sample-to-dataset level data augmentation module that effectively yields rich and diverse semantic and spatial hierarchy information to inhibit occlusions and complex backgrounds while learning fine-grained local details. Extensive quantitative and qualitative experiments on benchmark datasets demonstrate that our MSHRNet outperforms the state-of-the-art methods in terms of both accuracy and computational complexity at the cost of little increase in the number of parameters. Codes and all data are publicly available at https://github.com/Ray-tju/MSHRNet.