Refining visual token sequence for efficient image captioning

In practical applications, both accuracy and speed are critical for image captioning (IC) systems. Recently, transformer-based architectures have significantly advanced the field of IC; however, these improvements often come at the cost of increased computational complexity and slower inference speeds. In this paper, we conduct a comprehensive analysis of the computational overhead of IC models and find that the visual encoding process accounts for the majority of this overhead. Considering the redundancy in visual information — where many regions are irrelevant or provide low information for prediction — we propose a knowledge-injection-based visual token Reduction module. This module estimates the importance of each token and retains only a subset of them. To minimize visual semantic loss, we introduce token Fusion and Insertion modules that supplement visual semantics by reusing discarded tokens and capturing global semantics. Based on this, our visual token sequence refinement strategy, referred to as RFI, is deployed at specific positions in the visual backbone to hierarchically compress the visual token sequence, thereby reducing the overall computational overhead of the model at its source. Extensive experiments demonstrate the effectiveness of the proposed method, showing that it can accelerate model inference without sacrificing performance. Additionally, the method allows for flexible trade-offs between accuracy and speed under different settings.