A Novel Approach to Scene Graph Vectorization

Abstract

In recent times due to the advancement in perceptual applications, focus in computer vision has been inclined towards tasks that require a significant level of semantic understanding of scenes. Combination of textual and visual information has resulted in a great improvement in performance on tasks like retrieval, captioning and visual q/a etc. In this regard, scene graphs have become a popular form of structural knowledge. But unlike Word embedding, general-purpose scene graph embedding has not been explored significantly. In this work, we propose a general-purpose scene graph embedding model that combines linguistic and graph processing techniques through a reconstruction based learning network to learn a low-dimensional data-driven vectorized embedding of scene graphs. Visualization of embedding of COCO dataset has shown to possess semantic separability and distance-based abstraction of scenes. When applied to a retrieval task and evaluated using Med-r and recall metric on COCO-stuff and VRD dataset, our model showed promising results.