Latency Improvement Strategy for Temporally Stable Sequential 3DMM-based Face Expression Tracking

Abstract

2D image-based face tracking is a core feature for multiple AR/VR applications. The latest advancements in self-supervised 3DMM face reconstruction maintained high-accuracy analysis-by-synthesis tracking but were not designed for online inference settings with low latency performance. Recently, state-of-the-art models such as MICA [1] has demonstrated significant improvement in term of accuracy for the offline face construction task but the design is ill-suited for practical use cases due to their long processing time on low and middle-end hardware. The original workflow includes two analysis-by-synthesis stages: face shape reconstruction and face tracking. The shape reconstruction aims to regress a neutral 3DMM model from the input. Then the tracking process learns relevant parameters for expressions, eyes, mouth, etc. for a differentiable render to reconstruct the original photographic input. This study aims to propose a design for an interface to apply offline 3DMM face tracking into an online inference pipeline for facial analysis-based applications.