Emotion recognition by global and local feature fusion for people with facial defects

Aiming at the problem of improving network performance by ignoring imperfections and performing recognition based on localization, ignoring the correlation between features and thus encountering challenges in the face recognition task for individuals with facial defects, a method combining facial texture reconstruction with a two-channel emotion recognition system is proposed. First, a defect removal module is added in the feature processing stage to smooth the damaged facial region and refine the texture. An adaptive module is introduced to deal with the fuzzy boundary between normal skin and damaged regions. In addition, a local fine-grained feature extraction module is introduced to capture multi-location information subspace features. Finally, a dual-channel mechanism combining local and global features is adopted to focus on detailed local features of the undamaged region, supplemented by reconstructed global features for emotion recognition. Extensive experiments show that the method’s performance in this paper is 89.57% on RAF-DB, 89.93% on FERPlus, 64.6% on AffectNet-7, and 60.73% on AffectNet-8.