Research on Digital Media Art for Image Caption Generation Based on Integrated Transformer Models in CLIP

Abstract

Digital media art has a wide application in the field of image caption generation. In digital media art exhibitions or online works displays, some complex image works may have multiple layers of meanings or abstract expressions, which can help viewers better understand the works. It can also serve as another auxiliary element besides sound, collaborating with visual elements to provide a richer experience for the audience. The purpose of picture captioning is to provide textual descriptions that correlate to input images. The CLIP paradigm is highly versatile to resolve vision-text difficulties. In the field of picture description, the standard Transformer architecture has also exhibited good effects, which uses an image encoder and a text decoder. Large parameter numbers and the demand for further data preprocessing are still significant difficulties. In order to replace the fundamental features of conventional multi-modal fusion models, we propose a New Multi-modal Fusion Attention module (NMFA), which efficiently decreases parameter sizes and computational complexity in half. Expanding upon this, we propose the Transformer Fusion CLIP (TFC) model, which minimizes parameter sizes and processing demands while getting remarkable assessment scores. Additionally, we strengthen the mechanism for cumulative points and reward sequence length to encourage the construction of larger sequences. Finally, we combine the enhanced beam search technique to further train the TFC model. Results from our testing on the MSCOCO dataset reveal that we have not only greatly improved the efficiency of the TFC model but also speeded up its runtime by eight times and reduced model parameters by over 50%.