TAM-TR: Text-guided attention multi-modal transformer for object detection in UAV images

Object detection in unmanned aerial vehicles (UAV) imagery is crucial in many fields, such as maritime search and rescue, remote sensing mapping, urban management and agricultural monitoring. The diverse perspectives and altitudes of UAV images often result in significant variations in the appearance and dimensions of objects, and occlusions are found more frequently than in general scenes. The unique bird’s-eye view of drones makes it more difficult for existing object detection models to distinguish between similar objects. A text-guided attention multi-modal transformer network named TAM-TR is proposed to address the above challenges. A Bidirectional Text–Image Attention Path Aggregation Network (BTA-PAN) is proposed in TAM-TR. This network imitates the architecture of the classic algorithm Scale-Invariant Feature Transform (SIFT) and shows better scale adaptability. A novel Multi-modal encoder–decoder head (MEH) was proposed, which can simultaneously consider all input sequence positions to avoid the disappearance of features of occluded objects. An additional text-guided attention branch, combined with a large text model, was proposed to improve the TAM-TR’s classification accuracy. Additionally, a Rotation-invariant IOU (RIOU) loss function was proposed to eliminate the previous loss function’s rotational instability. The experiment demonstrated that the TAM-TR outperformed the baseline by 9.5% and achieves 39.7% mean Averaged Precision (mAP) on the Visdrone dataset. The code will be available at https://github.com/Xjh-UCAS/TAM-TR.