Dual-level dynamic heterogeneous graph network for video question answering

Abstract

Recently, Video Question Answering (VideoQA) has garnered considerable research interest as a pivotal task within the realm of vision-language understanding. However, existing Video Question Answering datasets often lack sufficient entity and event information. Thus, the Vision Language Models (VLMs) struggle to complete intricate grounding and reasoning among multi-modal entities or events and heavily rely on language short-cut or irrelevant visual context. To address these challenges, we make improvements from both data and model perspectives. In terms of VideoQA data, we focus on supplementing the missing specific entities and events with the proposed event and entity augmentation strategies. Based on the augmented data, we propose a Dual-Level Dynamic Heterogeneous Graph Network (DDHG) for Video Question Answering. DDHG incorporates transformer layers to capture the dynamic temporal-spatial changes of visual entities. Then, DDHG establishes multi-modal semantic grounding ability between vision and text with entity-level and event-level heterogeneous graphs. Finally, the Dual-level Cross-modal Interaction Module integrates the dual-level features to predict correct answers. Our method not only significantly outperforms existing VideoQA models on two complex event-based benchmark datasets (Causal-VidQA and NExT-QA) but also demonstrates superior event content prediction ability over several state-of-the-art approaches.