Processing All k-Nearest Neighbor Query on Large Multidimensional Data

Abstract

All k nearest neighbor (AkNN) query processing is a data processing problem which is important in many fields such as computer architecture, searching user information by coordinates, and city planning. Nowadays amount of data tends to grow in size and becomes huge. It is a major challenge that we need to face. Therefore, many traditional methods are no longer effective when dealing with the problem. Meanwhile, the method that processes distributed and parallel AkNN problem on MapReduce model and uses equal-cell-dividing technique is effective on multidimensional large dataset. However, when data is not equally distributed, the method becomes inefficient and even cannot be implemented. In this paper, we improve this method by applying a new cell-dividing technique. Instead of dividing the target space into cells which have the same size, we aim to divide it to cells in which the number of points are balanced, and there is not a cell that contains a large number of points. We also conduct experiments and compare the results produced by the old method and our method. Experimental results show that our method is more efficient and more stable.