Fast Permutation Architecture on Encrypted Data for Secure Neural Network Inference

Abstract

Recently, the secure neural network inference, an organic combination of the homomorphic encryption (HE) and the deep neural network (DNN), has attracted much attention. Nevertheless, the large number computations, brought by the HE scheme, form the bottleneck for real-time applications. A significant portion of the network is the permutation (Perm), which is mainly made up of the number theoretic transform (NTT). In this paper, for the first time, we propose an efficient architecture for the Perm by incorporating algorithmic transformations and architectural level optimizations. First, the core butterfly unit (BU) of NTT is optimized, which reduces the multiplication operations by about 30% compared with the original BU. Then, based on the optimization, a highly parallelized architecture is devised for the Perm. The operations in different modules are well managed by a merging strategy to balance the data path and reduce the memory access. The proposed architecture is synthesized under the TSMC 28-nm CMOS technology. The experimental results show that for the ciphertext size of 2048×60 bits, the proposed design achieves a 7.54x speedup compared to the implementation on an Intel(R) Core(TM) i7-6850K 3.60Hz CPU. Moreover, we apply eight Perm engines to the 1D convolution, which shows a 17.25x speedup over the software implementation.