From multi-scale grids to dynamic regions: Dual-relation enhanced transformer for image captioning

Abstract

The purpose of image captioning is to describe the visual content of an image in an accurate and natural sentence. Some previous methods adopt convolutional networks to encode grid-level features, whereas others use an object detector to extract region-level features. However, the spatial resolution of high-level grid features is typically low, thus capturing small-scale objects is challenging for such models. In addition, most region-based methods directly set the same number of regions to represent all images, failing to account for varying scene complexities across different images. They introduce noise in region relationship modeling and disrupt sentence reasoning. To address these issues, we propose a novel Dual-Relation Enhanced Transformer (DRET) model that complements the advantages of multi-scale grid and dynamic region features. In the encoding phase, we first apply multiple sampling strategies to generate multi-scale grid features, then design a novel multi-scale grid attention (MGA) encoder that learns the relationships between features at different scales. Meanwhile, a new dynamic region selection (DRS) encoder is devised to dynamically select an appropriate number of regions based on the scene complexity of each input image, effectively pruning redundant regions and enhancing correlations between selected regions. In the decoding stage, we combine the advantages of grid and region features, using a cross-modal adaptive gating (CAG) decoder that automatically determines the gate weights of the two visual features at each time step. Extensive experiments on MS-COCO and Flickr30K show that our model achieves better performance compared to current methods.