Re-ranking Content Based Social Image Search Results by Multi Modal Relevance Feedback

Social image hosting websites, such as Flickr, have a rapid growth recently. Content based image retrieval on such websites is an useful potential service but is still unavailable because its performance is unsatisfactory. We propose a multi modal relevance feedback (MMRF) scheme and a supervised re-ranking approach based on it to improve the performance for practical application. Our multi modal scheme utilizes both image and social tag relevance feedback instances. The approach propagates visual and textual information as well as multi modal relevance feedback information on the graph with a mutual reinforcement process. We conduct experiments based on real world data from Flickr to evaluate the performance of our approach. We also conduct an experiment to show that our multi modal relevance feedback scheme significantly improves performance compared with traditional single modal relevance feedback (SMRF) scheme.