Graph Structural Attention and Increased Global Attention for Image Captioning

Abstract

Attention mechanism plays a significant role in the current encoder-decoder framework of image captioning. Nevertheless, many attention mechanisms only fuse textual feature and image feature once, failing to adequately integrate the feature between context and image. Furthermore, many image captioning networks based on scene graphs only consider the node information but ignore the structure, which is insufficient in grasping the spatial object relationship. To address the above problems, we propose structural attention and increased global attention. Two attentions select critical image features from image detail and global image. The increased global attention, focusing on global image features, enhances integration between text and image via fusing detailed image features into global attention. To better describe the relationship among image objects, our network allows for both the node information by content attention and the structure information by structural attention. Structural attention computes the similarity between the structure information of scene graph and local attention, building the image objects relationship differing from content attention. We evaluate the performance of our image captioning network in MS COCO and Visual Genome datasets. The results of the experiments show that our method achieves superior performance compared with the existing methods.