DGTM: Deriving Graph from transformer with Mamba for panoptic scene graph generation

Scene Graph Generation (SGG) transforms images into structured graph representations that encapsulate the objects, attributes, and relationships present within objects. Graph models boost visual content understanding and reasoning for image captioning, question answering, and HCI. Panoptic Scene Graph Generation (PSG) enhances the object detection task within scene graph generation by incorporating panoptic segmentation, thereby imposing greater demands on the model’s capacity to comprehend images. Existing approaches often rely on intricate modeling techniques to predict relationships between objects, while neglecting the inherent connections among object queries that are learned through multi-head self-attention in object detectors. This oversight not only leads to a significant increase in parameter count but also complicates model design and hinders transferability. This paper proposes a new single-stage panoptic scene graph generator called DGTM (Deriving Graph from transformer with Mamba). DGTM utilizes the by-products of multi-head self-attention layers in transformers, treats queries and keys as subjects and objects respectively to extract relationship information between objects. By introducing the Mamba module, multi-level and multi-scale feature information is integrated efficiently, empowering the model to better grasp intricate relationships. In addition, a Kolmogorov–Arnold Network (KAN) is incorporated to help the model better distinguish between subjects and objects, enriching feature representation. Experimental results show that DGTM achieves at least 25%, 15%, and 15% improvements in mR@20, mR@50, and mR@100 compared to the baseline, demonstrating notable enhancements in the precision and comprehensiveness of PSG.