Generating Permutations Using Hash Tables

Abstract

Given a set of $N$ distinct values, the operation of shuffling those elements and creating a random order is used in a wide range range of applications, including (but not limited to) statistical analysis, machine learning, games, and bootstrapping. The operation of shuffling elements is equivalent to generating a random permutation and applying the permutation. For example, the random permutation of an input allows splitting into two or more subsets without bias. This operation is repeated for machine learning applications when both a train and test data set are needed. In this paper we describe a new method for creating random permutations that is scalable, efficient, and simple. We show that the operation of generating a random permutation shares traits with building a hashtable. Our method uses a fairly new hash table, called HashGraph, to generate the permutation. HashGraph's unique data-structure ensures easy generation and retrieval of the permutation. HashGraph is one of the fastest known hash-tables for the GPU and also outperforms many leading CPU hash-tables. We show the performance of our new permutation generation scheme using both Python and CUDA versions of HashGraph. Our CUDA implementation is roughly 10% faster than our Python implementation. In contrast to the shuffle operation in NVIDIA's Thrust and cuPy frameworks, our new permutation generation algorithm is 2.6 x and 1.73 x faster, respectively and up to 150 x faster than numPy.