Image retrieval based on multimodality neural network and local sensitive hash

Abstract

With the rapid development of deep convolutional neural networks, the use of deep convolutional neural networks to extract features instead of manual features has become one of the current research hotspots. However, deep convolutional neural network can not understand image features well, and there is a “semantic gap”. Contrastive Language-Image PreTraining (CLIP) model is a pre-training neural network model based on matching image and text. Use the pre-trained CLIP model to extract the high-dimensional feature vector of the image data set to be retrieved, and the Local Sensitive Hash (LSH) algorithm was used to extract the retrieval speed to complete the retrieval task based on the image content and text. Experimental results show that compared with other content-based image retrieval algorithms, the proposed algorithm can also understand the text information in the image to complete the retrieval task, and has a wider retrieval range.