ReAlign: Re-Aligning Intra-Class Misalignment for Few-Shot Segmentation

Abstract

Few-shot segmentation aims to segment unseen query objects using a few annotated support samples, where robustness against intra-class misalignment between support and query features is crucial. We introduce ReAlign, a method that corrects these discrepancies by realigning intra-class misalignments. Our approach is underpinned by two key observations. First, intra-class consistent knowledge and spatial association knowledge effectively mitigate class inconsistencies. To leverage these insights, we developed the Harmonized Debiasing Module, which employs multimodal approaches to extract this essential knowledge. Addressing the potential biases from limited sample sizes, this module incorporates multi-teacher distillation and compression techniques to enhance representation robustness. Second, we observe that query features with strong semantic signals exhibit enhanced self-matching properties, which help reduce intra-class misalignment. Additionally, exploiting the rich textual knowledge from visual-language pre-trained models optimizes alignment. Building on these, we propose the High-CLIP method, which refines existing vision-language pre-training frameworks by incorporating text features to extract high-confidence query features for autonomous decision-making, further reducing misalignment issues. ReAlign significantly outperforms existing methods, setting new benchmarks. It achieves mIoU scores of 71.6% and 76.4% for 1-shot and 5-shot segmentation, respectively, on PASCAL- $5^{i}$ .