Visual Question Answering Focusing on Object Positional Relation with Capsule Network

Abstract

This paper presents a visual question answering (VQA) system focusing on object positional relation, which consists of Capsule Network and a recurrent neural language model. Grasping object positions in an image is necessary to understand the image and appropriately answer a question based on the image. Most related works employ state-of-the-art object recognition systems to detect objects in an image correctly and achieve higher accuracy for VQA datasets. However, It is difficult for the object recognition systems to extract enough object position from the image because of their architectures. The systems employ max-pooling to select representative features in an area of the image and the max-pooling tends to introduce position ambiguity. To overcome the drawback, we construct a VQA system with Capsule Network, which can capture object position information without max-pooling. For experiments, we choose only yes/no type questions from VQA dataset and the proposed method improves approximately 4% accuracy for the whole questions. Especially, the proposed method improves approximately 15% accuracy for questions including "next to" and "front of"