DF-LSH: An efficient Double Filters Locality Sensitive Hashing for approximate nearest neighbor search

Abstract

Locality-sensitive hashing (LSH) is an effective randomized technique for high-dimensional approximate nearest neighbor search. In practice, LSH has been used in large-scale recommendation and information retrieval tasks. However, it often suffers from the problem of generating many false positives due to the randomness of hash functions, which significantly increases the query overhead needed to retrieve these false positives. To this end, recent LSH variants have proposed more rigorous search schemes or used more compact hash codes to identify eligible candidates. However, as the number of candidates increases, query performance often degrades. To effectively address this problem, we propose a Double Filters Locality Sensitive Hashing scheme, called DF-LSH, designed for efficient approximate nearest neighbor search in high-dimensional datasets. DF-LSH first learns data-aware hash functions for a Bloom filter to decrease false positives, and then leverages the geometrical nature of random projection for further filter. By employing these double filters, DF-LSH provides probability guarantees similar to that of standard LSH schemes for query performance. Experiment results demonstrate that DF-LSH achieves superior query performance in both accuracy and efficiency over various high-dimensional datasets, with an average query time reduction of up to around 45x compared against previous LSH techniques while maintaining identical query accuracy.