Solution Path Algorithm for Double Margin Support Vector Machines

Abstract

Data uncertainty is a challenging problem in machine learning. Distributionally robust optimization (DRO) can be used to model the data uncertainty. Based on DRO, a new support vector machines with double regularization terms and double margins can be derived. The proposed model can capture the data uncertainty in a probabilistic way and perform automatic feature selection for high dimensional data. We prove that the optimal solutions of this model change piecewise linearly with respect to the hyperparameters. Based on this property, we can derive the entire solution path by computing solutions only at the breakpoints. A solution path algorithm is proposed to efficiently identify the optimal solutions, thereby accelerating the hyperparameter tuning process. In computational efficiency experiments, the proposed solution path algorithm demonstrates superior performance compared to the CVXPY method and the Sequential Minimal Optimization (SMO) algorithm. Numerical experiments further confirm that the proposed model achieves robust performance even under noisy data conditions.