Faster and space efficient indexing for locality sensitive hashing

Abstract

This work suggests faster and space-efficient index construction algorithms for LSH for Euclidean distance (a.k.a. E2LSH) and cosine similarity (a.k.a. SRP). The index construction step of these LSHs relies on grouping data points into several bins of hash tables based on their hashcode. To generate an m-dimensional hashcode of the d-dimensional data point, these LSHs first project the data point onto a d-dimensional random Gaussian vector and then discretise the resulting inner product. The time and space complexity of both E2LSH and SRP for computing an m-sized hashcode of a d-dimensional vector is $O\left(md\right)$, which becomes impractical for large values of m and d. To overcome this problem, we propose two alternative LSH hashcode generation algorithms both for Euclidean distance and cosine similarity, namely, CS-E2LSH, HCS-E2LSH and CS-SRP, HCS-SRP, respectively. CS-E2LSH and CS-SRP are based on count sketch [1] and HCS-E2LSH and HCS-SRP utilize higher-order count sketch [2]. These proposals significantly reduce the hashcode computation time from $O\left(md\right)$ to $O\left(d\right)$. Additionally, both CS-E2LSH and CS-SRP reduce the space complexity from $O\left(md\right)$ to $O\left(d\right)$; and HCS-E2LSH, HCS-SRP reduce the space complexity from $O\left(md\right)$ to $O\left(N\sqrt[N]{d}\right)$ respectively, where $N\ge 1$ denotes the size of the input/reshaped tensor. Our proposals are backed by strong mathematical guarantees, and we validate their performance through simulations on various real-world datasets.