Toward Efficient and Generalizable Text Dataset Distillation via a Dual-Agent Large Language Model Framework.

Abstract

Large-scale datasets impose substantial training costs on machine learning models. Dataset distillation addresses this issue by synthesizing compact datasets that can achieve performance comparable to that of the original data. However, text dataset distillation remains challenging: the discrete nature of text renders traditional gradient-matching methods ineffective, while embedding optimization approaches are often inefficient and exhibit limited generalization. To address these challenges, this paper proposes an LLM-native distillation framework based on dual-agent collaboration. Our framework decomposes the distillation process into two stages: selection and improvement. The Selector identifies high-quality samples through multi-dimensional scoring, while the Improver enhances data density and clarity under semantic consistency constraints and a candidate generation mechanism. The entire pipeline is automated through a self-iterative cycle of generation and selection, and leverages scoring signals to drive agent self-reward iteration, thereby avoiding differentiable optimization and trajectory matching. Experiments show that starting from a random subsampling, using only 2.5% of original data and only three iterations, the distilled dataset enables Llama-2-7B, Mistral-7B, and Qwen2.5-7B to match full-dataset performance on MMLU and Winogrande. It also improves training stability, convergence efficiency, and cross-model generalization, especially among architecturally similar models, while maintaining a competitive distillation cost of 143 GPU h. Overall, this study provides a practical solution for efficient, automated, and general-purpose text dataset distillation.