Event-level multimodal feature fusion for audio–visual event localization

Abstract

Audio–visual event localization, by identifying audio–visual segments most relevant to semantics from long video sequences, has become a crucial prerequisite for applications such as video content understanding and editing. Albeit the rich visual and auditory information in video data greatly enhances the accuracy of event localization models, the challenges, however, such as visual ambiguity, occlusion, small-scale targets, and sparse auditory features, hinder the acquisition of temporally continuous video segments with semantic consistency for events. To this end, we propose an event localization model with event-level multimodal feature fusion strategy to encode the event semantics consistency from video data, thereby improving the event localization accuracy. In particular, a multimodal features and distribution consistency loss is devised to train the spatial attention based architecture, along with the supervised loss, to fuse the multi-modal attended features to achieve the semantic consistency. To further mitigate the detrimental impact of outliers, i.e., the segments with non-relevant semantics, we propose to learn adaptive continuity sampling parameters to construct segment content sets with consistent semantics to the video, the experimental results demonstrate the advantages of our model against the existing event localization counterparts.