Comprehensive Weight Decomposition Analysis of Modern Parameter-Efficient Methods

Abstract

Large Language Models fine-tuning is an essential part of modern artificial intelligent systems that solve numerous tasks, such as natural language processing and computer vision. Among the various fine-tuning strategies, the most prominent approach for Large Language Model fine-tuning is Parameter-Efficient Fine-Tuning (PEFT), as it allows to achieve state-of-the-art performance on multiple tasks while minimizing computational resources and training time. Recently, an increasing number of PEFT methodologies have been developed, each asserting superiority based on performance metrics. However, a critical evaluation of how these methods align with the tuning dynamic of the full fine-tuning (FT) remains largely unexplored. This study focuses on bridging this gap by analyzing the learning behavior of such PEFT approaches as LoRA, LoRA+, AdaLoRA, DoRA, VeRA, PiSSA, LoKr and LoHa in comparison to FT. This work provides a comprehensive comparative analysis aimed at identifying which PEFT methods diverge significantly in weights update dynamic from the FT standard. The findings reveal insights into the underlying causes of these discrepancies, offering a deeper understanding of each method’s behavior and efficacy.