GF-SVD: Global knowledge-infused singular value decomposition of large language models

Abstract

Singular Value Decomposition (SVD) provides an efficient solution for compressing and accelerating Large Language Models (LLMs) without retraining or specialized hardware. Despite its advantages, current SVD-based LLMs compression methods suffer from three critical limitations that degrade performance: (1) Cross-domain knowledge preservation is compromised, (2) Layer-isolated decomposition disrupts inter-layer information flow, and (3) Gradual knowledge erosion caused by aggressive truncation of singular values and corresponding vectors. To overcome these, we propose GF-SVD, a novel framework that integrates: (1) Hierarchical Knowledge Infusion: Enhances dataset diversity by integrating hierarchical knowledge to improve cross-domain generalization, (2) Global Information Integration: Captures inter-layer dependencies and broader context via weighted aggregation of multi-layer feature matrices, and (3) Knowledge-Enhanced Truncation and Updating: Truncates and updates weights with infused dataset to mitigate knowledge erosion. Extensive experiments demonstrate that GF-SVD surpasses existing SVD-based LLMs compression methods across diverse tasks, including knowledge-intensive question answering, complex reasoning, physical system, and mathematical problem-solving. Notably, GF-SVD can also improve inference speed by 2.36x on GPUs and 2.74x on CPUs at 60 % compression ratio.