On the probability of feature selection in support vector classification

Abstract

Feature selection is important for classification problem, especially when the number of features is very large or noisiness is present in data. Support vector machine (SVM) with Lp regularization is a popular approach for feature selection. Many researches have devoted to develop efficient methods to solve the optimization problem in support vector machine. However, to our knowledge, there is still no formal proof or comprehensive mathematical understanding on how Lp regularization can bring feature selection. In this paper, we first show that feature selection depends not only the parameter p but also the data itself. If the feasible region generated from the data lies faraway relatively from the coordinates, then feature selection maybe impossible for any p. Otherwise, a small p can help to enhance the ability of feature selection of Lp-SVM. Then we provide a formula for computing the probabilities which measure the feature selection ability. The only assumption is that the optimal solutions of all possible classification problems distribute uniformly on the contour of the objective function. Based on this formula, we compute the probabilities for some popular p.