Human-like scene graph generation and evaluation.

Abstract

Current methods that generate a scene graph of a given image create an overdense graph regarding the number of objects and relationships found in the image, making the graph less effective in describing what is relevant in the image and, consequently, in downstream tasks such as cross-modal retrieval and mining. In this work, we propose a novel method that generates a scene graph that reflects what humans find important when describing an image. During training, our scene generation method is guided by human-drafted captions describing the images, which we assume will focus on essential scene elements. This guidance is realized by properly designed loss functions. During inference, scene graphs are generated solely from images. We evaluate the resulting scene graphs by comparing them with the ground-truth scene graphs of Visual Genome that are created by humans. Evaluation is done with recall- and precision-oriented metrics and graph edit distances. In the first set of experiments, we benchmark existing scene graph generation models, then we add the newly proposed loss functions leading to improved performance, especially in terms of the graph edit distance. Extra experiments show that the correct recognition of unimportant background objects and their relationships is crucial when generating human-like scene graphs. The codebase is released on github: https://github.com/VSJMilewski/relevance_graphs.