Efficient Side-Tuning for Remote Sensing: A Low-Memory Fine-Tuning Framework

Fine-tuning pretrained models for remote sensing tasks often demands substantial computational resources. To reduce memory requirements and training costs, this article proposes a low-memory fine-tuning framework, called efficient side-tuning (EST), for remote sensing downstream tasks. EST attaches a parallel network to the backbone of the model, and only fine-tunes the parameters of the parallel network during the training phase. The proposed EST Block is the main component of the parallel network, which uses the multichannel adapter fusion module, gate layer and depthwise convolution to achieve feature selection and enhancement effects. In the evaluation, on six remote sensing datasets including object detection and semantic segmentation, EST achieved SOTA performance results using only less than 40$\%$ of the memory expenditure of full fine-tuning, which is better than all current parameter efficient fine-tuning methods. In addition, experiments on backbones of various sizes and classes show that the generalizability of EST is also reliable. EST thus offers a highly efficient and effective approach for efficient transfer learning in remote sensing, unlocking new possibilities for advanced remote sensing applications.