Few-sample video captioning using pre-trained language model with gated bidirectional fusion

Video captioning generates sentences for describing the video content. Previous works mostly rely on a large number of video samples for training the model, but annotating them is very costly, thus limiting the widespread application of video captioning. This motivates us to explore the way of using only a few labeled samples to describe the video, and propose a few-sample video captioning method by adopting the Pre-trained language model with Gated Bidirectional Fusion (PGBF). In particular, we design a triple dynamic gating module that dynamically adjusts the contributions of appearance, motion, and text information to leverage the linguistic knowledge from pre-trained language model. Meanwhile, we develop a bidirectional fusion module to fuse appearance-text features and motion-text features to learn better cross-modal features. Moreover, we introduce a semantic contrastive loss to minimize the gap between visual features (i.e., appearance, motion, and the fused one) and text features (i.e., parsed nouns, verbs and whole sentence). Extensive experiments on three popular benchmarks demonstrate that our method achieves promising video captioning performance by using only a few training samples.