Leveraging generative AI synthetic and social media data for content generalizability to overcome data constraints in vision deep learning

Abstract

Generalizing deep learning models across diverse content types is a persistent challenge in domains like facial emotion recognition (FER), where datasets often fail to reflect the wide range of emotional responses triggered by different stimuli. This study addresses the issue of content generalizability by comparing FER model performance between models trained on video data collected in a controlled laboratory environment, data extracted from a social media platform (YouTube), and synthetic data generated using Generative Adversarial Networks. The videos focus on facial reactions to advertisements, and the integration of these different data sources seeks to address underrepresented advertisement genres, emotional reactions, and individual diversity. Our FER models leverage Convolutional Neural Networks Xception architecture, which is fine-tuned using category based sampling. This ensures training and validation data represent diverse advertisement categories, while testing data includes novel content to evaluate generalizability rigorously. Precision–recall curves and ROC-AUC metrics are used to assess performance. Results indicate a 7% improvement in accuracy and a 12% increase in precision–recall AUC when combining real-world social media and synthetic data, demonstrating reduced overfitting and enhanced content generalizability. These findings highlight the effectiveness of integrating synthetic and real-world data to build FER systems that perform reliably across more diverse and representative content.