Multi-level semantic-aware transformer for image captioning

Abstract

Effective visual representation is crucial for image captioning task. Among the existing methods, the grid-based visual encoding methods take fragmented features extracted from the entire image as input, lacking the fine-grained semantic information focused on salient objects. To address this issue, we propose an effective method, namely Multi-Level Semantic-Aware Transformer (MLSAT) for image captioning, to simultaneously focus on contextual details and high-level semantic information centered on salient objects. First, to model the spatial correlations of grids and the semantic interactions of salient objects, we propose the Visual Content Guided Attention (VCGA), which adaptively embeds the relative position relationships of the grids into the visual features based on their visual content and is used as the attention layer of the encoder. Then, in order to enhance the visual representation, we propose the Multi-Level Semantic-Aware (MLSA) module which further models the fine-grained semantic information centered on salient objects. In this module, the primary semantic information is first extracted from the encoder by using the Semantic Information Extractor (SIE), then refined by the Semantic Refiner (SR) and adaptively integrated into the visual representation by the Visual-Semantic Fusion Block (V-SFB). Our MLSAT is extensively evaluated on the MS-COCO dataset and outperforms the state-of-the-art models, with 135.1% CIDEr (c40) on the official online testing server. The source code is available at https://github.com/XvZhao147/MLSAT