SPAFPN: Wear a multi-scale feature fusion scarf around neck for real-time object detector

Abstract

Nowadays, the feature extraction module of the backbone network develops rapidly in the field of real-time object detection, while the multi-scale design for the neck iterates slowly. The traditional PAFPN is short in cross-scale propagation efficiency. However, many newly proposed multi-scale fusion methods, which make up for this drawback, are difficult to widely apply because of their complex fusion modules and unfriendly training. In this paper, we propose the Scarf Path Aggregation Feature Pyramid Network (SPAFPN), an advanced neck structure of multi-scale fusion for real-time object detection. SPAFPN adheres to the decentralized multi-scale fusion idea of ”Light Fusion, Heavy Decouple” while inheriting the concept of versatility and portability design. SPAFPN can promote cross-scale low-loss transfer of features and improve the performance of the model, which mainly consists of Pyramid Fusion and Multi-Concat modules. The experimental results on the MS COCO dataset show that SPAFPN-HG-N/S/M can achieve a 5.2%/3.2%/1.1% mAP improvement over YOLOv8, respectively. We also adopted some latest backbone networks to verify the scalability of SPAFPN. Specifically, SPAFPN combined with C2f and GELAN as backbone network modules both perform well. In addition, SPAFPN still maintains good performance when applied to the DETR-based real-time object detector. The code is available at https://github.com/ztbian-bzt/SPAFPN.