Multimodal prompting and masking strategy for video-grounded dialogue

Abstract

Video-Grounded Dialogue (VGD) is a challenging vision-language task aimed at engaging in multi-turn dialogues with humans based on video and audio content. Despite significant progress in improving AI-generated responses has been made, several challenges remain: 1) A significant amount of computing resources and time are required during training; 2) Current dominant approaches, utilizing T5 or GPT2 as base models, exhibit limited ability to understand video and audio features due to their text-based pre-training paradigms; 3) Existing studies have not addressed the robustness of models in real-world scenarios where dialog history is often missing. To address these issues, we propose VPM, a Video-Grounded Dialogue framework employing prompt-based tuning and a masking strategy. Firstly, to reduce computation resources, inspired by prompt learning, we are the first to employ prompt-based tuning in Video-Grounded Dialogue task by using only 20 % of the training set while maintaining proximal accuracy. Secondly, to enhance the model’s understanding of video and audio, we propose a slicing-based visual mapping network, integrating learnable visual prompts and video-audio slice features sequentially through a series of operations. Finally, we put forward an exponentially masking strategy for dialogue history to improve cross-modal understanding and robustness. Extensive experiments validate the effectiveness of our proposed framework, achieving state-of-the-art performance on the AVSD@DSTC7 and AVSD@DSTC8 datasets.