Classification Related Manifold Dimension Estimation with Restricted Boltzmann Machine

Abstract

To handle high dimensional variables in real world, especially multimedia data, dimension reduction techniques provide effective solutions for feature selection which makes the problem easy to deal in a lower dimension subspace. However, the primary problem with traditional dimension reduction method is to estimate intrinsic dimension of manifold supporting the raw data. Since all existing approaches of dimension reduction need to set the target dimension first. And for task of classification, the dimension we need maybe different from the intrinsic one, which means feature with intrinsic dimension may offer too much information for the classification task. In this paper, we propose a method to estimate classification related dimension with the help of RBM (Restricted Boltzmann Machine) and SVM classifiers. RBM is used for dimension reduction by mapping raw data to a low dimensional hidden space, and SVM classifier is used to test information preserving ability of this hidden space. An optimal dimension is selected as the minimum one that preserves all the classification related information of novel vectors. Further, a novel initialization strategy is proposed to speed up the training of RBM. These new methods are with low time consumption and memory cost compared to many other approaches of dimension estimation. Experiments and comparisons on several synthetic and realistic datasets show the superiority of the proposed method.