Multimodal representation fusion method for dense video captioning

Abstract

Dense video captioning aims to locate multiple events from untrimmed videos and generate corresponding captions for each meaningful event. The application of multimodal information(e.g., video, audio) for dense video captioning has recently achieved great success. However, learning the information interactions between different modalities while achieving cross-modal feature alignment is highly challenging for an encoder. Recent studies of several multimodal tasks have shown that multimodal models benefit from shared and individual representations. Thus, in this paper, we propose a novel feature fusion module, which uses shared and individual modality representations to capture commonalities and complementary relationships between modalities. Moreover, the proposed model bridges the gap between shared modality representations, which helps to obtain deeper cross-modal associations for better feature interaction and alignment. Furthermore, to compensate for the limitation that different level proposal heads do not interact sufficiently during event detection, we propose a multilevel information interaction mechanism to dynamically adjust and fuse the information among different level proposal heads in the event detection module. Based on the ActivityNet Captions, subdatasets of ActivityNet Captions and YouCook2, we conducted comprehensive experiments to evaluate the performance of our proposed model. The experimental results show that our model achieves impressive performance compared with state-of-the-art methods.