Dual dynamic transformer for image captioning

Abstract

The task of image captioning, widely acclaimed in the field of computer vision, aims to depict the content of an image, wielding a significant impact on people’s lives. Present methodologies for this task typically involve extracting global and local features to capture both overall and intricate details within images. However, the former, reliant on high-level, low-resolution grid features, when directly inputted into transformer encoders, may falter in establishing robust correlations between individual grids, thereby leading to the loss of relationship information between grid features. Additionally, the latter, utilizing region features derived from object detectors, may hinder transformers from comprehending the semantic relationships among regions, resulting in semantic information loss. To tackle these challenges, we introduce a novel Dual Dynamic Transformer (D${}^2$T) framework for image captioning, amalgamating the benefits of dynamic grid features and dynamic region features. Specifically, the Dynamic Pseudo-regions Grid (DPG) encoder enhances the strong correlation between grid features by grouping the attention of different grids and dynamically generating pseudo-regions, facilitating superior fusion with region features. Furthermore, the Dynamic Multi-Level Relation Region (DMR${}^2$) encoder augments the comprehension of semantic relationships among various region features through attention-based multi-level relations. In the encoding phase, to seamlessly integrate dynamic grid features and dynamic region features, we propose a feature fusion module for combining these two distinct feature types. Moreover, additional experiments conducted on the MSCOCO dataset demonstrate that our model achieves state-of-the-art performance without incurring additional parameter overhead.