Frequency domain transfer learning for remote sensing visual question answering

Remote Sensing Visual Question Answering (RSVQA) aims to parse the content of remote sensing images through multimodal interaction to accurately extract scientific knowledge. Current RSVQA methods typically fine-tune pre-trained models on specific datasets, which overlooks the mining of complex structured information in remote sensing images that are highly coupled with color, scale, and semantics. Additionally, these methods lack adequate handling of overfitting issues caused by the high complexity and noise attributes of remote sensing data, resulting in predictions that are neither comprehensive nor accurate. To mitigate this issue, this paper proposes a Parameter-Efficient Transfer Learning (PETL) method based on the frequency domain. By leveraging Fourier Transform, it captures the intricate structural information of complex remote sensing and enhances the generalizability across domains, data, and models. The main contributions of this paper are as follows: 1) We introduce an efficient and stable X-PFA framework for Remote Sensing Visual Question Answering (RSVQA). Here, ‘X’ denotes pretrained VLP models, and ‘PFA’ stands for Primary Frequency Adapter, which performs a Fast Fourier Transform (FFT) over the intermediate spatial domain features at each layer to produce the corresponding frequency representation, including an amplitude component (encoding scene-perceptual style such as texture, color, scene contrast) and a phase component (encoding rich semantics). The PFA adapts to the specific dataset distribution by learning salient features in the frequency domain. 2) Our proposed framework demonstrates stable and excellent performance across various pre-trained models, significantly mitigating overfitting issues on small datasets. On average, accuracy improves by 0.62 %, and stability increases by 28.5 %. 3) The PFA contains only 17.2 million trainable parameters. Compared to full-parameter fine-tuning, our approach reduces training time by 54.9 %, resulting in substantial training cost savings.