{"title":"FACET–VLM: Facial emotion learning with text-guided multiview fusion via vision-language model for 3D/4D facial expression recognition","authors":"Muzammil Behzad","doi":"10.1016/j.neucom.2025.131621","DOIUrl":null,"url":null,"abstract":"<div><div>Facial expression recognition (FER) in 3D and 4D domains presents a significant challenge in affective computing due to the complexity of spatial and temporal facial dynamics. Its success is crucial for advancing applications in human behavior understanding, healthcare monitoring, and human-computer interaction. In this work, we propose FACET–VLM, a vision–language framework for 3D/4D FER that integrates multiview facial representation learning with semantic guidance from natural language prompts. FACET–VLM introduces three key components: Cross-View Semantic Aggregation (CVSA) for view-consistent fusion, Multiview Text-Guided Fusion (MTGF) for semantically aligned facial emotions, and a multiview consistency loss to enforce structural coherence across views. Our model achieves state-of-the-art accuracy across multiple benchmarks, including BU-3DFE, Bosphorus, BU-4DFE, and BP4D-Spontaneous. We further extend FACET–VLM to 4D micro-expression recognition (MER) on the 4DME dataset, demonstrating strong performance in capturing subtle, short-lived emotional cues. FACET–VLM achieves up to 99.41 % accuracy on BU-4DFE and outperforms prior methods by margins as high as 15.12 % in cross-dataset evaluation on BP4D. The extensive experimental results confirm the effectiveness and substantial contributions of each individual component within the framework. Overall, FACET–VLM offers a robust, extensible, and high-performing solution for multimodal FER in both posed and spontaneous settings.</div></div>","PeriodicalId":19268,"journal":{"name":"Neurocomputing","volume":"657 ","pages":"Article 131621"},"PeriodicalIF":6.5000,"publicationDate":"2025-09-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Neurocomputing","FirstCategoryId":"94","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0925231225022933","RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE","Score":null,"Total":0}
引用次数: 0
Abstract
Facial expression recognition (FER) in 3D and 4D domains presents a significant challenge in affective computing due to the complexity of spatial and temporal facial dynamics. Its success is crucial for advancing applications in human behavior understanding, healthcare monitoring, and human-computer interaction. In this work, we propose FACET–VLM, a vision–language framework for 3D/4D FER that integrates multiview facial representation learning with semantic guidance from natural language prompts. FACET–VLM introduces three key components: Cross-View Semantic Aggregation (CVSA) for view-consistent fusion, Multiview Text-Guided Fusion (MTGF) for semantically aligned facial emotions, and a multiview consistency loss to enforce structural coherence across views. Our model achieves state-of-the-art accuracy across multiple benchmarks, including BU-3DFE, Bosphorus, BU-4DFE, and BP4D-Spontaneous. We further extend FACET–VLM to 4D micro-expression recognition (MER) on the 4DME dataset, demonstrating strong performance in capturing subtle, short-lived emotional cues. FACET–VLM achieves up to 99.41 % accuracy on BU-4DFE and outperforms prior methods by margins as high as 15.12 % in cross-dataset evaluation on BP4D. The extensive experimental results confirm the effectiveness and substantial contributions of each individual component within the framework. Overall, FACET–VLM offers a robust, extensible, and high-performing solution for multimodal FER in both posed and spontaneous settings.
期刊介绍:
Neurocomputing publishes articles describing recent fundamental contributions in the field of neurocomputing. Neurocomputing theory, practice and applications are the essential topics being covered.