MS-SMF: A probabilistic–causal multi-layer framework for image–text matching

Abstract

Image–text matching faces significant challenges due to the complex many-to-many semantic relationships between modalities, and existing methods remain deficient in both uncertainty modeling and causal robustness. To address these issues, we propose a multi-layer structural semantic matching framework (MS-SMF) that enhances cross-modal representations from a probabilistic–causal perspective. First, our method employs graph convolutional networks augmented with pseudo-coordinates and syntactic dependencies to precisely align image regions with text tokens, encoding each aligned pair as a Gaussian distribution to explicitly quantify semantic uncertainty. Next, these local Gaussian distributions are aggregated via structure-aware similarity weights into global Gaussian features, capturing holistic semantic context. Finally, we simulate causal interventions by randomly permuting the global mixture weights to generate counterfactual semantic distributions, and impose a contrastive loss between the original and counterfactual embeddings to enforce both invariance and discriminability. By synergizing the “soft” robustness of Gaussian modeling with the “hard” discriminative power of causal contrast, MS-SMF substantially improves the stability and discriminative capacity of cross-modal representations under structural perturbations. Experimental results on the MSCOCO and Flickr30K datasets demonstrate that our approach outperforms state-of-the-art methods — achieving an rSum of 542.9 on Flickr30K, with image-to-text R@1 and text-to-image R@1 improved by 1.0% and 2.3%, respectively — validating its superior performance in complex and ambiguous matching scenarios.