CA-DA: Class-adaptive deformable attention for semantic segmentation in internet-enabled visual sensing networks

Abstract

Semantic segmentation is a fundamental component of Internet-enabled visual sensing networks, where distributed cameras, edge devices, and networked sensors collect large-scale visual data from complex environments. However, such data often exhibit severe pixel-level imbalance and category heterogeneity, with semantic classes differing in frequency, spatial extent, boundary complexity, and structural patterns. These characteristics make standard pixel-wise training prone to biased decision boundaries and insufficient representations for underrepresented categories. In this paper, we identify two key limitations in this setting: the misalignment between standard optimization objectives and class-balanced evaluation criteria, and the lack of class-adaptive feature aggregation under heterogeneous category statistics. To address them, we propose a unified category-aware prior-compensation framework that combines output-level calibration with sampling-level adaptive feature learning. Specifically, we adopt relative-prior logit adjustment to alleviate pixel-wise decision bias toward dominant classes. Meanwhile, we introduce a spatial-demand-aware class-adaptive deformable attention mechanism that modulates deformable offsets during training according to class frequency and class-wise spatial demand. This design encourages flexible contextual aggregation for underrepresented and spatially complex categories while maintaining stable aggregation for dominant ones. During inference, the offset predictor remains fully feature-driven and requires no ground-truth labels, pseudo-labels, or external class annotations. Experiments on Cityscapes and ADE20K show that the proposed method improves segmentation performance for underrepresented classes while maintaining competitive overall accuracy, achieving superior class-balanced performance over representative baselines.