Keyword-aware Multi-modal Enhancement Attention for Video Question Answering

Video question answering (VideoQA) is an intriguing topic in the field of visual language. Most of the current VideoQA models directly harness the global video information to answer questions. However, in VideoQA task, the answers associated with the questions merely appear in a few video contents, and other contents are invalid and redundant information. Therefore, VideoQA is vulnerable to be interfered by a large number of irrelevant contents. To address this challenge, we propose a Keyword-aware Multi-modal Enhancement Attention model for VideoQA. Specifically, a multi-factor keyword extraction (MFKE) algorithm is proposed to emphasize the crucial information in multimodal feature extraction. Furthermore, based on attention mechanisms, a keyword-aware enhancement attention (KAEA) module is designed to correlate the information associated with multiple modalities and fuse multimodal features. The experimental results on publicly available large VideoQA datasets, namely TVQA+ and LifeQA, demonstrate the effectiveness of our model.