Privacy-enhanced data distillation with probability distribution matching

Abstract

Data distillation aims to condense the large-scale original dataset into a small-scale synthetic dataset while preserving as much data utility as possible. As one of the typical implementation mechanisms of data distillation, distribution matching works by aligning the feature distributions of synthetic and original samples, while avoiding the expensive computation and memory costs associated with other matching mechanisms. However, there still exist two primary limitations in distribution matching. On the one hand, distribution matching suffers from inadequate class discrimination, the synthetic samples within the same class may be misclassified as other classes due to the scattered feature distribution. On the other hand, distribution matching raises serious privacy concerns, as the synthetic dataset may inadvertently contain some sensitive information extracted from the original dataset. Taking this cue, we propose here a novel privacy-enhanced distribution matching-based data distillation algorithm. First, we design a probability distribution matching method with intra-class aggregation constraint and inter-class dispersion constraint based on symmetric Kullback-Leibler divergence to strengthen the performance of data distillation. Second, we design a dynamic noise perturbation method based on differential privacy to enhance data privacy guarantees while preserving higher sample quality. Extensive experiments demonstrate that our algorithm can achieve performance improvements of up to 4.5 % on the CIFAR10 dataset and 2.7 % on the SVHN dataset, compared to the state-of-the-art methods.