CM-SC: Cross-modal spatial-channel attention network for image captioning

Abstract

In multi-modal reasoning tasks, modern models often encounter difficulties with capturing higher-order interactions and maintaining computational efficiency, particularly when processing long visual sequences. Conventional attention-based models for image captioning generally focus on first-order interactions, limiting their ability to capture intricate relationships between objects within an image. This constraint can hinder the generation of high-quality captions, as nuanced understanding and description of visual content require recognizing these deeper interconnections. In this work, we address these challenges by introducing a novel attention mechanism which is an innovative model tailored for cross-modal interactions that selectively utilizes both visual and textual information to generate fine-grained captions. Our attention mechanism, termed Cross-Modal Spatial-Channel (CM-SC), incorporates a flexible variant of cross-variance that effectively captures higher-order interactions within spatial and channel-wise attention distributions across different modalities. By stacking multiple CM-SC attention blocks, our approach facilitates second-order to potentially infinite-order feature interactions without requiring additional parameters. The attention module integrates seamlessly with both LSTM and Transformer frameworks. Notably, our approach reduces average computation time by 20.74% compared to the baseline model and improves performance metrics. Extensive experiments were conducted to validate the proposed framework on the Flickr30K and MSCOCO datasets. The results show that our approach performs competitively against many contemporary standard methods.