LoRA2: Multi-scale low-rank approximations for fine-tuning large language models

Abstract

Fine-tuning large language models (LLMs) with high parameter efficiency for downstream tasks has become a new paradigm. Low-Rank Adaptation (LoRA) significantly reduces the number of trainable parameters for fine-tuning. Although it has demonstrated commendable performance, updating parameters within a single scale may not be the optimal choice for complex downstream tasks. In this paper, we extend the LoRA to multiple scales, dubbed as LoRA${}^2$. We first combine orthogonal projection theory to train two low-dimensional LoRAs in two mutually orthogonal planes. By multiplying two LoRAs, a high-dimensional LoRA is obtained, forming a multi-scale LoRA. Then, we improve the importance score algorithm, significantly reducing the computation required for parameter sensitivity scoring. By pruning singular values with lower importance scores, thereby enhancing adaptability to various downstream tasks. Extensive experiments are conducted on two widely used pre-trained models to validate the effectiveness of LoRA${}^2$. Results show that it significantly reduces the number of trainable parameters to just 0.72% compared to full fine-tuning on the DeBERTa-V3-base model, while still delivering highly impressive performance. Our code is available here: https://github.com/Godz-z/LoRA-2.