A complementary object detection method via integrating CNN with Vision Transformer(ViT)

Abstract

The object detection method based on CNN is the mainstream method in the field of object detection because of its special structure of hierarchical and gradual extraction of local features, which can simultaneously consider low-level geometric features and high-level semantic features. However, this structure does not make full use of the global information of the image, resulting in classification errors when the features are limited or fuzzy. To solve the above problems, this paper explores a complementary feature extraction backbone via integrating CNN with Vision Transformer(ViT), and designs a shallow ViT structure to interact features of the proposals in a two-stage object detector with the image background feature to realize global modeling and feature alignment. In addition, according to the particularity of the supplementary structure, a segmented training strategy is designed. This strategy ensures that the model can extract the features together, and maximize the independence of their respective structures, giving full play to the advantages brought by different feature extraction methods. The model is verified on the COCO and PASCAL VOC Datasets. Through the experimental results and feature visualization analysis, it can be concluded that the mAP values are higher than CNN-based and ViT based detectors on the premise of adding limited parameters, which proves the effectiveness of the method.