Optimized homomorphic linear computation in privacy-preserving CNN inference

Machine Learning as a Service (MLaaS) provides robust solutions for deploying deep learning inference in cloud environments. However, it also raises serious privacy concerns regarding user data and proprietary model parameters. Numerous hybrid cryptographic protocols that integrate homomorphic encryption (HE) and garbled circuits (GC) have been proposed to enable secure inference with low latency. In these protocols, the homomorphic evaluation of linear operations remains the primary performance bottleneck and warrants further optimization. In this work, we propose novel optimizations for HE-based linear computations within the hybrid cryptographic framework for secure neural network inference. Specifically, we devise two efficient strategies for homomorphic matrix-vector multiplication and convolution. For matrix-vector multiplication, we introduce a grouped diagonal extraction technique that encodes the weight matrix more compactly and enables configurable ciphertext rotation reuse, while for homomorphic convolution, we present a group-wise combine-and-merge evaluation method. Both methods significantly reduce the number of required ciphertext rotations. Our approach achieves up to a $3.9\times$ speedup in matrix-vector multiplication and a $2.9\times$ improvement in convolution over state-of-the-art (SOTA) solutions. The HE-GC hybrid secure convolutional neural networks (CNN) inference framework incorporating these enhancements yields speedups of $2.5\times$ on widely used ResNets deep learning architectures.