Comparative Study of Visual Question Answering Algorithms

Visual Question Answering (VQA) is a recent task that challenges algorithms to reason about the visual content of an image to be able to answer a natural language question. In this study, we compare the performance of state of the art VQA algorithms on different VQA benchmarks. Each benchmark is more effective at testing VQA algorithms on different levels. Some datasets challenge the algorithms to perform complex reasoning steps to arrive to an answer. Other datasets might challenge algorithms to retrieve external world knowledge to answer the posed questions. We categorize the algorithms by their main contributions into 4 categories. Firstly, the joint embedding approach which focuses on how to map the visual and textual data into a common embedding space. Secondly, attention based methods which focuses on relevant parts of the image or the question. Thirdly, compositional models which deal with composing a model from smaller modules. Finally, we introduce external-knowledge based algorithms which need external sources to be able to retrieve facts necessary to answer a question when those facts may not be present in the scene nor in the whole training data set. We also mention other algorithms that don’t specifically belong to the aforementioned categories, but offers performance competitive with the state of the art.