Emotion-Assisted multi-modal Personality Recognition using adversarial Contrastive learning

Multi-modal personality recognition integrates text, audio, and video information to accurately identify personality traits, offering significant value in fields like human–computer interaction. However, existing methods face feature extraction, noise removal, and modal alignment challenges. These issues impact recognition accuracy and model robustness. To address these issues, we propose an Emotion-Assisted multi-modal Personality Recognition using adversarial Contrastive learning (EAPRC). EAPRC leverages text, audio, and image data, incorporating emotional information to enhance recognition accuracy and robustness through adversarial training. The model reduces inter-modal noise using adversarial sample generation and employs joint class propagation contrastive learning to extract discriminative feature representations. For emotion-based assistance, EAPRC uses emotion feature-guided fusion and emotion score decision fusion to exploit the correlation between emotions and personality traits fully. It further improves the accuracy and stability of multi-modal personality recognition. Experimental results on the ChaLearn First Impressions and ELEA datasets demonstrate that EAPRC performs effectively, validating its capability in multi-modal personality recognition tasks.