Prototype-based multi-domain self-distillation for unbiased scene graph generation

Abstract

Scene Graph Generation (SGG) plays an important role in reinforcing visual image understanding. Existing methods often encounter difficulties in effectively representing implicit relationship features, which limits their capacity to distinguish between predicates. Meanwhile, these approaches are susceptible to imbalanced instance distributions, hindering the efficient training of fine-grained predicates. To address these problems, we propose a novel prototype-based multi-domain self-distillation training framework. Specifically, a Multi-Domain Fusion (MDF) module is introduced to improve predicate feature representation by integrating global contextual information and local spatial-frequency domain information. Then, a Prototype Generation Network (PGN) is designed for building the class prototypes, which consists of the design of different granularity predicates and loss functions. Furthermore, we design two different data balancing strategies under the guidance of class prototypes, which correspond to mining the in-distribution and out-of-distribution information of the original data, respectively. The experimental results demonstrate that the proposed method is superior to the existing methods on VG, GQA and Open Images V6 datasets, which makes it more applicable to generating unbiased scene graph models.