MCoCa: Towards fine-grained multimodal control in image captioning

Controllable image captioning (CIC) models have traditionally focused on generating controlled descriptions using specific text styles. However, these approaches are limited as they rely solely on text control signals, which often fail to align with complex human intentions, such as selecting specific areas in images. To enhance multimodal interactivity, we propose to augment current CIC systems with diverse and joint visual-text controls. To achieve this, we first create a comprehensive Multimodal Controllable Image Captioning Corpus (MCoCa) dataset by leveraging language rewriting ability of GPT-3.5, containing 0.97M image-captions pairs along with 21 visual-text control signals. By training the visual and textual adapters equipped on the multimodal large language model with newly proposed instructional prompts on MCoCa, we observe emergent combinatory multimodal controllability and significant improvement in text controllability. We present exhaustive quantitative and qualitative results, benchmarking our trained model’s state-of-the-art zero-shot captioning performance on SentiCap and FlickrStyle10K in terms of both fidelity and controllability. For regional understanding ability of visual-controlled captioning, our method achieves obvious improvement compared with the baseline models.