Adaptive DETR: A framework with dynamic sampling points and feature-guided adaptive attention updates

Abstract

In recent years, DETR-based models have advanced object detection but still face key challenges: the encoder’s high complexity and limited adaptability, and the decoder’s slow convergence due to query initialization. We propose Adaptive DETR, a framework with dynamic sampling and adaptive feature encoding. First, we design an attention update strategy that computes weights based on image features, enhancing detection accuracy. Second, we enable dynamic adjustment of sampling points in deformable attention, improving adaptability in complex scenes. Finally, we optimize the decoder by performing attention between bounding-box and semantic queries during initialization, effectively injecting semantics, accelerating convergence, and improving localization. Experiments on COCO, UAVDT, VisDrone, and RSOD confirm that Adaptive DETR achieves superior accuracy and generalization with improved efficiency.