WaveletFT: Discrete wavelet transform for parameter-efficient fine-tuning

Recently, Low-rank adaptation (LoRA) has achieved significant popularity for fine-tuning foundational models owing to its ability to substantially reduce the number of trainable parameters and avoid additional inference costs. This reduction is achieved by introducing low-rank matrices A and B to represent weight update,defined as $\Delta W=\mathrm{AB}$. Nonetheless, the accuracy gap frequently remains between LoRA and full fine-tuning (FT). Additionally, LoRA encounters difficulties concerning storage, particularly when extensive customization adaptations or larger base models are involved. In this work, we aim to resemble the learning capacity of FT and further reduce trainable parameters by leveraging the robust expressiveness of the wavelet transform (WT). We present a novel approach, named WaveletFT, which treats $\Delta W$ as a matrix within the spatial domain and focuses on learning only a small subset of its coefficients. By employing the trained spectral coefficients, we utilize the inverse discrete WT to reconstruct $\Delta W$. Experimental results demonstrate that the proposed WaveletFT method offers comparable or superior performance with fewer parameters compared to LoRA across diverse tasks, such as natural language understanding, natural language generation, instruction tuning, image classification and text-to-image generation.