Iterative Data Distillation and Augmentation for Enhancing Deep Learning Performance With Limited Image Training Data

Deep learning models require large training datasets. Incorporating additional data into small training datasets can enhance the model's performance. However, acquiring additional data may sometimes be challenging or beyond one's control. In such situations, data augmentation becomes essential to overcome the limited supply of labeled data by generating new data that preserves the essential properties of the original dataset. The primary objective of our research is to develop an iterative data distillation and augmentation (IDDA) method that enlarges the size of a limited image training dataset while preserving its properties. At every iteration, our method distills a set of images from the training set of the previous iteration utilizing the kernel inducing point (KIP) method, and the union of the training and distilled sets creates the new training set. However, our experiments show that IDDA is computationally expensive, increasing processing time by approximately 17%–27$\%$ for MNIST and Fashion-MNIST, 31%–39$\%$ for CIFAR-10, and up to 48%–49$\%$ for CIFAR-100 compared to state-of-the-art augmentation methods, due to the additional step of applying KIP for image distillation. We have experimentally determined that for a few iterations the classification accuracy increases and then drops afterward. We validate the IDDA capabilities by comparing it with conventional augmenting methods and MixUp on the following publicly available image datasets: MNIST digit, Fashion-MNIST, CIFAR-10, and CIFAR-100. Our approach proves highly effective for very limited datasets, addressing the challenge of database expansion for improved performance of deep learning models.