Parallel attribute reduction algorithm based on simplified neighborhood matrix with Apache Spark

Abstract

As an important branch of rough set theory, neighborhood rough set theory effectively addresses the problem of information loss originated from discretization process. Nevertheless, the computational efficiency of existing parallel neighborhood algorithms remains limited. In this paper, a parallel attribute reduction algorithm based on simplified neighborhood matrix is proposed and implemented with Apache Spark. Firstly, we define a novel neighborhood matrix to describe the neighborhood relationships among objects; Next, the neighborhood matrix is divided into a simplified neighborhood matrix and a set of neighborhood information granules, which is referred to as neighborhood knowledge in this paper. On the basis, a parallel attribute reduction algorithm is proposed based on simplified neighborhood matrix. The new reduction algorithm utilizes Spark’s sorting technique to generate the simplified neighborhood matrix swiftly and employs Python’s interrupt capabilities to enhance computational efficiency. Theoretical analysis and experimental results show that the proposed algorithm keeps the consistency of neighborhood knowledge and exhibits excellent parallel performance. It improves computational efficiency by 93.2%, 69.1%, and 80.4% compared to the benchmark algorithms.