Segment Any Vehicle: Semantic and Visual Context Driven SAM and a Benchmark

Abstract

With the rapid advancement of autonomous driving, vehicle perception, particularly detection and segmentation, demands increasingly sophisticated algorithmic performance. Pre-trained large segmentation models, especially the Segment Anything Model (SAM), have sparked significant interest and inspired new research directions in artificial intelligence. However, SAM cannot be directly applied to fine-grained vehicle part segmentation tasks. Its text-prompted segmentation functionality is not publicly accessible, and the mask regions generated in its default mode lack semantic labels, limiting its utility in structured, category-specific segmentation tasks. To address these limitations, we propose SAV (Segment Anything Vehicle), a novel framework comprising three core components: a SAM-based encoder-decoder, a vehicle part knowledge graph, and a context sample retrieval encoding module. The knowledge graph explicitly models the spatial and geometric relationships among vehicle parts through structured ontology, effectively encoding prior structural knowledge. Meanwhile, the context retrieval module enhances segmentation by identifying and leveraging visually similar vehicle instances from training data, providing rich contextual priors for improved generalization. Furthermore, we introduce VehicleSeg10K, a new large-scale benchmark dataset for vehicle part segmentation containing 11,665 high-quality images and 97,471 annotated part instances across diverse scenes and viewpoints. We conduct comprehensive experiments on this dataset and two other benchmark datasets, evaluating multiple representative baseline methods to establish a solid foundation for future research and comparison. Both the dataset and source code are available at https://github.com/Event-AHU/SAV