Heterogeneous Collaborative Refining for Real-Time End-to-End Image-Text Retrieval System

Abstract

The image-text retrieval task currently suffers from high search latency due to the cost of image feature extraction and semantic alignment calculation. We propose a real-time image-text retrieval system for edge-end servers with low-power AI accelerator cards. The procedure is conspicuously sped up by selectively placing part of the deep learning calculation on accelerator devices with a heterogeneous collaborative computation scheme. We also design a lightweight GCN optimization method, which directly transfers the correlation between the image detection areas in projection to reduce computational redundancy. Our other contributions include performance analyses of models with different weights for industrial reference in practical applications. It is the first GCN-based image-text retrieval system to perform a real-time end-to-end search to the best of our knowledge. Experiments show that the system can process 20 image-to-text retrievals per second with high accuracy.