PCSR: Enabling Cross-Modal Semantic Retrieval With Privacy Preservation

Abstract

Cross-modal semantic retrieval systems face significant privacy risks due to storing plaintext data on cloud servers. We propose PCSR, a privacy-preserving framework enabling semantic search directly on encrypted high-dimensional data. It consists of three essential modules: a cross-modal encoder, an approximate nearest neighbor (ANN) search algorithm, and an encryption algorithm. Specifically, we utilize CLIP, a deep neural network model, to extract features of images and texts. We design two ANN search methods for high-dimensional feature vectors by utilizing the space partitioning technique and Singular Value Decomposition algorithms, respectively. Furthermore, we employ adapted Random Matrix Multiplication (RMM) for efficient and secure vector similarity computations. Our rigorous security analysis demonstrates that our proposed schemes are secure. We conduct experiments on four datasets and systematically compare the performance of different encrypted retrieval methods. The superior performance validates the feasibility and efficiency of our proposed schemes.