Safe and effective post-fine-tuning alignment in large language models

Abstract

Fine-tuning is critical to customizing Large Language Models (LLMs) in various applications, but it inevitably disrupts the safety alignment of the models. Current alignment methods tackle harmful fine-tuning challenges but frequently compromise model usefulness, resulting in unsatisfactory downstream task performance. To address this issue, we propose a Safe and Effective post-fine-tuning Alignment (SEA) from a knowledge disentanglement perspective. SEA introduces a novel two-level pruning process that surgically removes harmful functionalities. We first propose a differential importance score to isolate harmful pathways at the parameter level, and then introduce a module-wise analysis to protect entangled modules, thereby robustly balancing safety and utility. Experimental results on Llama2, Gemma and Mistral demonstrate that SEA effectively mitigates safety risks while maintaining optimal fine-tuning accuracy. This work provides a practical solution to the safety-performance dilemma associated with harmful fine-tuning of LLMs.