Convex-Concave Programming: An Effective Alternative for Optimizing Shallow Neural Networks

In this study, we address the challenges of non-convex optimization in neural networks (NNs) by formulating the training of multilayer perceptron (MLP) NNs as a difference of convex functions (DC) problem. Utilizing the basic convex–concave algorithm to solve our DC problems, we introduce two alternative optimization techniques, DC-GD and DC-OPT, for determining MLP parameters. By leveraging the non-uniqueness property of the convex components in DC functions, we generate strongly convex components for the DC NN cost function. This strong convexity enables our proposed algorithms, DC-GD and DC-OPT, to achieve an iteration complexity of $O\left(\log \left(\frac{1}{\varepsilon }\right)\right)$, surpassing that of other solvers, such as stochastic gradient descent (SGD), which has an iteration complexity of $O\left(\frac{1}{\varepsilon }\right)$. This improvement raises the convergence rate from sublinear (SGD) to linear (ours) while maintaining comparable total computational costs. Furthermore, conventional NN optimizers like SGD, RMSprop, and Adam are highly sensitive to the learning rate, adding computational overhead for practitioners in selecting an appropriate learning rate. In contrast, our DC-OPT algorithm is hyperparameter-free (i.e., it requires no learning rate), and our DC-GD algorithm is less sensitive to the learning rate, offering comparable accuracy to other solvers. Additionally, we extend our approach to a convolutional NN architecture, demonstrating its applicability to modern NNs. We evaluate the performance of our proposed algorithms by comparing them to conventional optimizers such as SGD, RMSprop, and Adam across various test cases. The results suggest that our approach is a viable alternative for optimizing shallow MLP NNs.