Toward an Automatic Evaluation of Retrieval Performance with Large Scale Image Collections

The public availability of large-scale multimedia collections, such as YFCC, facilitates the evaluation of image retrieval approaches in real-life conditions. However, due to their size, the creation of exhaustive ground truth would require huge annotation effort, even for limited sets of queries. This paper investigates whether it is possible to estimate retrieval performance in absence of manually created ground truth data. Our hypothesis is that it is possible to leverage existing weak user annotations (tags) to automatically build ground truth data. To test our hypothesis, we implemented a large-scale retrieval pipeline based on two state-of-the-art image descriptors and two compressed versions of each. The top 50 results obtained with each configuration are manually annotated in order to estimate their performance. Alternately, we produce an automatic performance estimation that is based on pre-existing user tags. The automatic performance estimations exhibit strong positive correlation with the manual ones and the systems rankings obtained in the two evaluation settings are found to be similar. This indicates that, although incomplete and sometimes imprecision, weak user annotations can be effectively exploited to assess retrieval performance. As a by-product, we release state-of-the-art image features, as well as a reusable evaluation package that will encourage the use of YFCC in the community.