lifeXplore at the Lifelog Search Challenge 2023

Abstract

Searching substantial data archives of lifeloggers is a challenging task. The Lifelog Search Challenge (LSC) is an annually held competition with the aim of encouraging international teams to develop interactive content retrieval systems capable of searching large lifelog databases. LSC takes place as a live event co-located with the ACM International Conference on Multimedia Retrieval (ICMR), where teams compete against each other by solving retrieval tasks issued by the lifelogger. This paper presents our newest version of lifeXplore, a lifelog retrieval system that has been participating in LSC since 2018. For this year, we significantly redesign the entire system (backend, middleware, and frontend) and integrate free text-search using embeddings from vision transformers trained with large sets of text-image pairs. We present a novel architecture for multi-source search, where results from image embeddings are used together with results from traditional content analysis (for objects, concepts, and recognized text). We also perform intensive analysis of vision transformer models in order to know which one fits best to the requirements of the LSC.