MSLoRA: Meta-learned scaling for adaptive fine-tuning of LoRA

Abstract

Low rank adaptation (LoRA) methods have demonstrated strong capabilities in efficiently fine-tuning large models. However, existing LoRA-based approaches typically require manually setting the scaling factor, a process that involves extensive search efforts to find optimal values. To address this challenge, we first develop data-driven heuristic methods that automatically determine layer-wise scaling factors through either activation pattern analysis during forward propagation or gradient behavior monitoring during backward updates. However,their practical performance remains unsatisfactory in applications. Building upon these theoretical foundations, we present MSLoRA, a novel framework that reformulates scaling factor determination as a dynamic optimization problem in parameter-efficient fine-tuning. Our approach innovatively models scaling factors as self-adaptive meta-parameters whose optimal values emerge organically through the interplay between transformer architecture hierarchies and task-specific learning objectives. Extensive experiments conducted across both natural language understanding and generative tasks reveal that MSLoRA consistently outperforms baseline models. This highlights the effectiveness of MSLoRA’s dynamic, layer-specific adjustment mechanism in capturing the complex nature of task-specific activation patterns, making it a more robust and scalable solution for parameter-efficient fine-tuning of large models.