Mixer: Accelerating Deep Neural Network Training With Hybrid Optimizers

Optimization algorithms, or optimizers, are indispensable for accelerating training in deep learning. However, they often suffer from high computational complexity and inefficiency. In this work, we propose a novel approach called Mixer through combining both first-order and second-order optimization approaches to address these challenges. Mixer has three contributions: (1) introducing adaptive algorithm switching conditions, (2) presenting a gradient vector correction strategy to mitigate accuracy degradation, and (3) incorporating a lightweight second-order method using efficient matrix decomposition to reduce computation and memory overhead. We evaluate Mixer’s performance by training networks such as ResNet and BERT with real datasets, comparing them against several state-of-the-art methods. Results demonstrate Mixer outperforms others under the same convergence accuracy, saving 48.86% and 14.06% end-to-end training time compared with SGD and KAISA for ResNet-152, achieving up to 1.95× and 1.16× speedups, respectively. Besides, Mixer saves 30.93% and 22.63% training time, achieving 1.45× and 1.29× improvement over LAMB and KAISA for BERT-large. Simultaneously, Mixer saves at least 17.42% memory footprint over KAISA.