CaPaT: Cross-Aware Paired-Affine Transformation for Multimodal Data Fusion Network

Abstract

This letter proposes a cross-aware paired-affine transformation (CaPaT) network for multimodal data fusion tasks. Unlike existing networks that employ weight-sharing or indirect interaction strategies, the CaPaT introduces a direct feature interaction paradigm that significantly improves the transfer efficiency of feature fusion while reducing the number of model parameters. Specifically, this letter, respectively, splits multimodal data along the channel domain. It synthesizes specific group channels and opposite residual channels as data pairs to generate refined features, achieving direct interaction among multimodal features. Next, a scaling attention module is conducted on the refined feature pair for confidence map generation. Then, this letter multiplies confidence maps by their corresponding feature pairs, determining a more reasonable and significant margin feature representation. Finally, a classifier is conducted on the transformation features to output the final class labels. Experimental results demonstrate that the CaPaT achieves superior classification performance with fewer parameters than state-of-the-art methods.