Cross-modality-enhanced visual Scene Graph Generation

Abstract

Humans perceive scenes through multisensory cues, yet existing Scene Graph Generation (SGG) methods predominantly rely on visual input alone, neglecting the complementary information provided by auditory signals and cross-modal interactions. To overcome this limitation, we propose Audio-Enhanced Scene Graph Generation (AESGG), a novel framework that integrates audio cues to enhance both object detection and relation prediction. AESGG improves visual object proposals by incorporating aligned audio features, thereby reducing ambiguity in detection. It further employs a spatio-temporal transformer to model dynamic inter-object relationships over time. A self-supervised learning strategy is introduced to capture relation transitions across video frames effectively. To facilitate research in audio-visual scene understanding, we also present the VALM dataset. Experimental results demonstrate that AESGG consistently outperforms state-of-the-art baselines, achieving up to a 2.0 percentage point improvement in relation prediction metrics (R@50, PredCls, with constraints), reflecting its robust and generalizable performance gains.