A progressive attention network with transformer for multi-label image recognition

Abstract

Recent research typically improves the performance of multi-label image recognition by constructing higher-order pairwise label correlations. However, these methods lack the ability to effectively learn multi-scale features, which makes it difficult to distinguish small-scale objects. Moreover, most current attention-based methods to capture local salient features may ignore many useful non-salient features. To address the aforementioned issues, we propose a Transformer-based Progressive Attention Network (TPANet) for multi-label image recognition. Specifically, we first design a new adaptive multi-scale feature attention (AMSA) module to learn cross-scale features in multi-level features. Then, to excavate various useful object features, we introduce the transformer encoder to construct a semantic spatial attention (ESA) module and also propose a context-aware feature enhanced (CAFE) module. The former ESA module is used to discover complete object regions and capture discriminative features, and the latter CAFE module leverages object-local features to enhance pixel-level global features. The proposed TPANet model can generate more accurate object labels in three popular benchmark datasets (i.e., MS-COCO 2014, Pascal VOC 2007 and Visual Genome), and is competitive to state-of-the-art models (e.g., SST and FL-Tran, etc.).