Law LLM unlearning via interfere prompt, review output and update parameter: new challenges, method and baseline

Abstract

Law large language models (Law LLMs) often generate hallucinations, one of the reasons is due to memorizing sensitive, inaccurate, or outdated information, and unlearning such information has important research value, yet the legal domain lacks publicly available datasets and effective methods for LLM unlearning tasks and evaluations. This work innovatively proposes three unlearning tasks in legal field, providing new datasets for unlearning tasks. In addition, proposes a Law LLM unlearning method via loss adjustment with only need forgotten sequence (UNFS), providing a new baseline and unlearning method for the unlearning tasks. Further, after UNFS unlearning, proposing an inference method for Law LLMs that combines interfering input and reviewing output, reinforcing that Law LLMs avoid including erroneous information in the output. Designing a new metric for law LLM unlearning, the legal data memory evaluation method (LawME), LawME automatically judges the output quality of Law LLMs by comparing the content output by Law LLMs with the ground truth. Real-world dataset experiments and analyses validate UNFS’s effectiveness: on the three proposed legal unlearning datasets, UNFS’s accuracy decreases by 16.53 %, perplexity increased by 3.94, and AUC decreased by 16.09 %. On the retained datasets, UNFS’s accuracy only decreased by 0.02 %-0.26 %, and on the generalized task MMLU by only 0.07 %-0.15 %. These results demonstrate that UNFS has excellent unlearning performance and does not harm the performance on other data that do not participate in unlearning. Other experiments and analyses verified the validity of the proposed inference approach, LawME metrics.