Enriching visual feature representations for vision–language tasks using spectral transforms

Abstract

This paper presents a novel approach to enrich visual feature representations for vision–language tasks, such as image classification and captioning, by incorporating spectral transforms. Although spectral transforms have been widely utilized in signal processing, their application in deep learning has been relatively under-explored. We conducted extensive experiments on various transforms, including the Discrete Fourier Transform (DFT), Discrete Cosine Transform, Discrete Hartley Transform, and Hadamard Transform. Our findings highlight the effectiveness of the DFT, mainly when using the magnitude of complex outputs, in enriching visual features. The proposed method, validated on the MS COCO and Kylberg datasets, demonstrates superior performance compared to previous models, with a 4.8% improvement in CIDEr scores for image captioning tasks. Additionally, our approach enhances caption diversity by up to 3.1% and improves generation speed by up to 2% in Transformer models. These results underscore the potential of spectral feature enrichment in advancing vision–language tasks.