ClipCap+ +: An efficient image captioning approach via image encoder optimization and LLM fine-tuning

Abstract

ClipCap (CLIP prefix for image captioning), a leading image captioning model, exhibits limitations in recognizing images within specific domains. This study presents ClipCap+ +, an enhanced version of ClipCap that integrates key-value pair and residual connection modules. The key-value pair module implements a few-shot learning strategy by incorporating domain-specific knowledge, thereby improving the model's capability to recognize specialized image categories. The residual connection module optimizes the weight distribution between the pre-trained model and the key-value pair module, enhancing the model's transfer learning performance. During the inference phase, the model processes an input image through a multi-stage pipeline: (1) the visual encoder extracts image features to generate a hard visual prompt, (2) the key-value pair module dynamically constructs a domain-specific soft prompt, and (3) these complementary prompts are jointly fed into the large language model to synthesize the final image description. Extensive experiments on in-domain, near-domain, and cross-domain tasks show ClipCap+ + surpasses state-of-the-art models in accuracy, training efficiency, and generalization.