Multimodal Rumor Detection via Hierarchical CNN-Transformer Integration and Co-Adaptive Multimodal Fusion

Abstract

The widespread proliferation of misinformation on social media has raised significant concerns, emphasizing the urgent need for accurate multimodal rumor detection to ensure information authenticity. However, existing approaches remain limited by inadequate modeling of visual features, insufficient cross-modal semantic alignment, and rigid fusion strategies that fail to adapt to varying modality contributions. To address these challenges, we propose a novel framework: hierarchical co-adaptive multimodal model (HiCoMM). Specifically, to mitigate the limitation of existing visual encoders that fail to capture both local forgeries and global semantics, HiCoMM introduces a hierarchical visual encoder combining convolutional neural networks (CNNs) and Transformer layers to jointly extract fine-grained textures and holistic semantic information. Furthermore, to handle semantic misalignment and modality imbalance during multimodal integration, we design a co-adaptive multimodal fusion module. Within this module, a cross-modal multihead co-attention mechanism establishes precise alignment between visual patches and textual tokens to alleviate semantic mismatches, while a gating-based dynamic fusion mechanism adaptively adjusts the contribution of each modality according to sample-specific characteristics, thus overcoming the rigidity of static fusion strategies that often lead to modality imbalance. Experimental results on two real-world multimodal datasets demonstrate that HiCoMM consistently outperforms state-of-the-art methods.