BPA-Net: Bidirectional Prototype Alignment for Image Captioning

Abstract

Image captioning aims to generate accurate textual descriptions for visual inputs. A major challenge is the semantic gap between continuous visual features and discrete linguistic concepts. Existing methods often rely on the unidirectional injection of external textual priors, which inevitably introduces semantic noise. Furthermore, CLIP-based image captioning methods lack direct supervision from ground-truth captions, leading to weak cross-modal alignment. To address these issues, we propose BPA-Net, a framework for bidirectional prototype alignment. Specifically, the Dual Prototype Extraction module independently projects visual features and ground-truth text tokens into a shared semantic space to form structured prototypes. Then, the Cross-Modal Prototype Alignment module enables bidirectional interaction and alignment between visual and textual prototypes guided by symmetric matching objectives. This approach effectively mitigates the noise inherent in external proxy texts and establishes robust semantic alignment. Extensive experiments on the MS-COCO, Flickr8k, and Flickr30k benchmarks demonstrate that BPA-Net achieves state-of-the-art performance. Furthermore, evaluations on the Nocaps dataset confirm its strong generalization capability for describing novel objects. The code is available at https://github.com/CarasuLee/BPA-Net