An adaptive auto fusion with hierarchical attention for multimodal fake news detection

Abstract

The phenomenon of fake news often relies on diverse multimodal evidence to deceive readers and achieve widespread popularity. While existing fusion methods aim to enhance feature interaction, they typically rely on concatenation or attention mechanisms that struggle to model nuanced dynamics of multimodal information due to missing data and modality heterogeneity. To overcome these limitations, we propose an Adaptive Auto Fusion with Hierarchical Attention (AAFHA) framework for multimodal fake news detection. AAFHA integrates image captions directly into the fusion pipeline to strengthen cross-modal learning, unlike prior approaches that treat them as siloed inputs. We first design a multi-level interaction for text and captions by incorporating hierarchical encoding to capture both local and global dependencies, allowing the model to detect subtle cross-modal associations. Then, a sparse weighting technique, guided by hierarchical attention, further refines these interactions by dynamically allocating attention across modalities. This guided focus is implemented through a constrained SoftMax function, improving contextual alignment and reducing isolated feature modeling. To enable adaptive semantic integration, we introduce an Auto-Fusion module that supports dynamic end-to-end training. The model optimizes a learned similarity measure in a shared representation space, aligning textual, caption, and image features to adaptively capture semantic associations. Additionally, sparse training with contrastive loss is incorporated to preserve semantic consistency and enhance class separability during fusion. Experimental results demonstrate that AAFHA outperforms existing baselines, yielding accuracy improvements of 0.094%, 0.198%, and 0.001% on the PolitiFact, Gossip, and Pheme datasets, respectively. These findings demonstrate the model’s effectiveness in identifying multimodal fake news.