SemRank: Semantic rank learning for multimedia retrieval

Abstract

Multimedia retrieval suffers from the lack of common feature representation between a text based query and the visual content of a video repository. One approach to bridging this representation gap is known as query-by-concept, where a query and video are mapped into a common semantic feature space. One of the challenges with using semantic concepts for multimedia retrieval, is that the available vocabulary size is generally not sufficient for representing the content of the query and video. In addition, the lack of training data and visual feature representation often leads to low precision models. In this work, we explore the use of a query-by-concept approach for the multimedia Known Item Search (KIS) problem. We propose a semantic rank learning model, called SemRank, to overcome the challenges of the vocabulary size and lack of training data. First, we construct a semantic fusion model to combine the output from many noisy classifiers. Next, we train a gradient boosted regression tree model, using a semantic feature space derived from the query, video, and query-video similarity. Our approach is evaluated over a large internet video repository, and the results show that query-by-concept can be an effective model for multimedia KIS.