Concept-guided domain generalization for semantic segmentation

Abstract

Recent domain generalization semantic segmentation methods are proposed to use vision foundation models (VFMs) for achieving superior performance in unseen domains. However, unlike human vision, which naturally adapts to recognize objects in different contexts, VFMs still suffer from the distribution shift problem. Based on this, a concept-guided domain generalization (CDG) approach is proposed for semantic segmentation. First, considering that humans can recognize objects in various environments after humans learn the conception of objects, a concept token learning module is proposed to learn the semantic concept token from semantic prototypes, which aims to exploit domain-invariant instance-aware knowledge. Second, when the recognition of objects is uncertain, humans recognize the objects by contextual information. Thus, a concept-contextual calibration strategy is proposed to generate concept-contextual relations by the semantic concepts to calibrate uncertain regions for refining final predictions. Extensive experiments demonstrate that the proposed approach achieves superior performance on multiple benchmarks. The code is released on GitHub: https://github.com/seabearlmx/CDG.