Image annotation by semantic sparse recoding of visual content

Abstract

This paper presents a new semantic sparse recoding method to generate more descriptive and robust representation of visual content for image annotation. Although the visual bag-of-words (BOW) representation has been reported to achieve promising results in image annotation, its visual codebook is completely learnt from low-level visual features using quantization techniques and thus the so-called semantic gap remains unbridgeable. To handle such challenging issue, we utilize both the annotations of training images and the predicted annotations of test images to improve the original visual BOW representation. This is further formulated as a sparse coding problem so that the noise issue induced by the inaccurate quantization of visual features can also be handled to some extent. By developing an efficient sparse coding algorithm, we successfully generate a new visual BOW representation for image annotation. Since such sparse coding has actually incorporated the high-level semantic information into the original visual codebook, we thus consider it as semantic sparse recoding of the visual content. Although the predicted annotations of test images are also used as inputs by the traditional image annotation refinement, we focus on the visual BOW representation refinement for image annotation in this paper. The experimental results on two benchmark datasets show the superior performance of our semantic sparse recoding method in image annotation.