Pyramidal structure-correlated refinement for robust face alignment

Abstract

Recent face alignment methods attempt to capture representations of facial landmarks and learn the correlation between them. However, they often ignore the consistency between local landmarks and the overall face shape, which may lead to the low-efficiency correlation learning between long-distance landmarks. Besides, due to the uncertain localization, these methods may capture invalid local cues of landmark representations. To resolve these issues, we propose a pyramidal structure-correlated refinement method that integrates a novel fusion interactor into a pyramidal refinement framework. Specifically we introduce a fusion interactor to aggregate local regression cues of landmark representations into a global representation and encode the facial structure information. The facial structure information is then allocated to local representations to compensate for missing contexts of landmarks, such as occluded parts. Unlike vanilla attention mechanisms, our fusion interactor performs indirect interaction to avoid inconsistent landmark contexts, and incurs tiny computational complexity burdens. Additionally, to obtain valid local cues of landmarks, we further introduce a pyramidal refinement framework with multi-scale feature maps, which can sample landmark representations from the feature maps of specific scales according to the uncertainty of sampling positions. It can also gradually regularize the global representation with correct multi-scale spatial contexts to constrain the overall face shape. Experiments on some popular benchmarks demonstrate the effectiveness and robustness of our proposed method, especially its notably low failure rates in challenging scenarios.