Feature selection based on rough diversity entropy

Abstract

Information entropy, as a powerful tool for measuring the uncertainty of information, is widely used in many fields such as communication, data compression, data mining and bioinformatics. However, the classical information entropy has two shortcomings, that is, information entropy cannot accurately measure the uncertainty of knowledge in some cases and the joint probability in information entropy is usually difficult to calculate for high-dimensional data. Additionally, uncertainty measure is the foundation of feature selection in granular computing. Inaccurate measures may lead to poor performance of feature selection methods. To address these issues, we propose a novel uncertainty measure called rough diversity entropy based on rough set theory. Rough diversity entropy can more accurately measure the uncertainty of knowledge compared with the classical information entropy. In this article, rough diversity entropy and its variants are first defined, and their related properties are studied. Next, a heuristic feature selection method based on the defined measures is put forward, and the corresponding algorithm is also designed. Finally, a series of experiments are executed to validate the effectiveness and rationality of the proposed method. The analysis results show that our proposed method has good performance compared with eight existing feature selection methods. Moreover, the proposed method improves the average accuracy of 15 datasets by 6.53% under four classifiers, and achieves an average feature reduction rate of up to 99.81%. We believe that the proposed method is an effective feature selection approach for classification learning.