G-Mix: A Generalized Mixup Learning Framework Toward Flat Minima

Deep neural networks (DNNs) have demonstrated promising results in various complex tasks. However, such DNN models face challenges related to over-parameterization, particularly in scenarios where training data are scarce. In response to these challenges and to improve the generalization capabilities of DNNs, the Mixup technique has emerged, which effectively addresses the limitations posed by over-parameterization. Nevertheless, it still produces suboptimal outcomes. Inspired by the successful sharpness-aware minimization (SAM) method, which establishes a connection between the sharpness of the training loss landscape and model generalization, we propose a new learning framework called Generalized-Mixup, which combines the strengths of Mixup and SAM for training DNN models. The theoretical analysis provided demonstrates how the developed G-Mix framework enhances generalization. Additionally, to further optimize DNN performance with the G-Mix framework, we introduce two novel algorithms: Binary G-Mix (BG-Mix) and Decomposed G-Mix (DG-Mix). These algorithms partition the training data into two subsets based on the sharpness-sensitivity of each example to address the issue of “manifold intrusion” in Mixup. Both theoretical explanations and experimental results reveal that the proposed BG-Mix and DG-Mix algorithms further enhance model generalization across multiple datasets and models, achieving state-of-the-art performance.