An Augmented Embedding Spaces approach for Text-based Image Captioning

Abstract

Scene text-based Image Captioning is the problem that generates caption for an input image using both contexts of image and scene text information. To improve the performance of this problem, in this paper, we propose two modules, Objects-augmented and Grid features augmentation, to enhance spatial location information and global information understanding in images based on M4C-Captioner architecture for text-based Image Captioning problems. Experimental results on the TextCaps dataset show that our method achieves superior performance compared with the M4C-Captioner baseline approach. Our highest result on the Standard Test set is 20.02% and 85.64% in the two metrics BLEU4 and CIDEr, respectively.