DCFT: Dependency-aware continual learning fine-tuning for sparse LLMs

Abstract

As the size of Large Language Models (LLMs) increasing, they exhibit enhanced capabilities in general intelligence but also present greater challenges in deployment. Consequently, compressing LLMs has become critically important. Among the various compression techniques, post-training pruning is highly favored by researchers due to its efficiency. However, this one-shot pruning approach often results in a significant deterioration of model performance. To mitigate this issue, we introduce Dependency-aware Continual learning Fine-Tuning (DCFT) for sparse LLMs. This method facilitates fine-tuning across sequential tasks without compromising the model’s sparsity. Initially, we revisit the inference process in LLMs from a novel perspective, treating two matrices that previously required independent optimization as a unified entity. This strategy involves introduces merely 0.011‰ additional parameters to achieve efficient fine-tuning. Furthermore, we re-evaluate the parameter fine-tuning process through the lens of matrix space mapping. By constraining the similarity of the mapping matrices, our approach enables the model to retain its performance on prior tasks while learning new ones. We tested our method on models from the LLaMA-V1/V2 families, with parameters ranging from 7B to 70B, and under various sparsity ratios and patterns (unstructured and N:M sparsity). The results consistently demonstrate outstanding performance.