MBDBFormer: a multimodal bridge dual branch Transformer for person re-identification

Abstract

A key challenge in the person re-identification (ReID) task is the extraction of robust and discriminative pedestrian features. However, the sensitivity of RGB images in complex scenes to light, viewing angle differences and occlusion seriously affect the stability of feature extraction. To address the above problems, we propose a Multimodal Bridge Dual Branch Transformer (MBDBFormer) by combining CNN and Transformer. First, we use the luminance component in RGB and IHS with its frequency domain transformed low and high frequency components (I) as multimodal inputs for image preprocessing, so that the network takes into account both light adaptation and color information. Second, to effectively fuse the feature advantages of the two modalities, the preprocessed image information is input into a bridge branch network consisting of a multilayer down sampling network, and outputs one global and four local feature information through Transformer coding. Finally, using the dynamic allocation of attention weights, focusing on strengthening the feature expression of discriminative regions such as edges and textures, we designed the Gated Dynamic Attention and Feature Interaction Mechanism (GDFM), which establishes the long-range dependency between RGB and I feature, and achieves the complementary optimization of the two modal features. It enables the output fusion features to retain the rich color information of the RGB modality while incorporating the illumination robustness of the I modality. A number of experimental results show that our proposed method is better than the state-of-the-art method on the Market1501, DukeMTMC, MSMT17 generalized dataset and the Occluded-Duke occlusion dataset, which verifies the effectiveness of our method on the task of person re-identification.