Refactored Maskformer: Refactor localization and classification for improved universal image segmentation

The introduction of DEtection TRansformers (DETR) has marked a new era for universal image segmentation in computer vision. However, methods that use shared queries and attention layers for simultaneous localization and classification often encounter inter-task optimization conflicts. In this paper, we propose a novel architecture called Refactored Maskformer, which builds upon the Mask2Former through two key modifications: Decoupler and Reconciler. The Decoupler separates decoding pathways for localization and classification, enabling task-specific query and attention layer learning. Additionally, it employs a unified masked attention to confine the regions of interest for both tasks within the same object, along with a query Interactive-Attention layer to enhance task interaction. In the Reconciler module, we mitigate the optimization conflict issue by introducing localization supervised matching cost and task alignment learning loss functions. These functions aim to encourage high localization accuracy samples, while reducing the impact of high classification confidence samples with low localization accuracy on network optimization. Extensive experimental results demonstrate that our Refactored Maskformer achieves performance comparable to existing state-of-the-art models across all unified tasks, surpassing our baseline network, Mask2former, with 1.2% PQ on COCO, 6.8% AP on ADE20k, and 1.1% mIoU on Cityscapes. The code is available at https://github.com/leonzx7/Refactored-Maskformer.