SEACount: Semantic-driven Exemplar query Attention framework for image boosting class-agnostic Counting in Internet of Things

Abstract

In the Internet of Things (IoT) environment, large amounts of visual data are continuously collected, providing a rich resource for intelligent surveillance and management. For the task of class-agnostic counting in images, this paper proposes the Semantic-driven Exemplar query Attention Counting (SEACount) framework, which aims to quickly adapt and count unseen classes of objects using a few-shot exemplars. This is critical for real-time monitoring and analyzing visual semantic information in IoT. Specifically, we introduce two new components to extend Object Detection with Transformers (DETR): the Exemplar Query Attention (EQA) and the Dynamic Reshaping Module (DRM). EQA injects exemplar queries with rich semantic information into the decoder, facilitating the global image response to exemplar targets and enhancing the exemplar-to-image similarity metrics. The DRM, instead of only utilizing decoder features, fuses them with image features to enhance local details, reduce noise interference, and reshape the feature maps required for predicting density maps. This approach efficiently captures exemplar-relevant targets in images and quickly adapts to new categories without fine-tuning. Experimental results demonstrate that our proposed SEACount framework significantly outperforms other state-of-the-art methods on the latest FSC-147 dataset. We release the code at https://github.com/lxinhui1109/SEACount.git.