Proceedings of the 4th Annual on Lifelog Search Challenge最新文献

{"title":"Myscéal 2.0: A Revised Experimental Interactive Lifelog Retrieval System for LSC'21","authors":"Ly-Duyen Tran, Manh-Duy Nguyen, N. Binh, Hyowon Lee, C. Gurrin","doi":"10.1145/3463948.3469064","DOIUrl":"https://doi.org/10.1145/3463948.3469064","url":null,"abstract":"Building an interactive retrieval system for lifelogging contains many challenges due to massive multi-modal personal data besides the requirement of accuracy and rapid response for such a tool. The Lifelog Search Challenge (LSC) is the international lifelog retrieval competition that inspires researchers to develop their systems to cope with the challenges and evaluates the effectiveness of their solutions. In this paper, we upgrade our previous Myscéal and present Myscéal 2.0 system for the LSC'21 with the improved features inspired by the novice users experiments. The experiments show that a novice user achieved more than half of the expert score on average. To mitigate the gap of them, some potential enhancements were identified and integrated to the enhanced version.","PeriodicalId":150532,"journal":{"name":"Proceedings of the 4th Annual on Lifelog Search Challenge","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2021-08-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132232907","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Exploring Intuitive Lifelog Retrieval and Interaction Modes in Virtual Reality with vitrivr-VR","authors":"Florian Spiess, Ralph Gasser, Silvan Heller, Luca Rossetto, Loris Sauter, Milan van Zanten, H. Schuldt","doi":"10.1145/3463948.3469061","DOIUrl":"https://doi.org/10.1145/3463948.3469061","url":null,"abstract":"The multimodal nature of lifelog data collections poses unique challenges for multimedia management and retrieval systems. The Lifelog Search Challenge (LSC) offers an annual evaluation platform for such interactive retrieval systems. They compete against one another in finding items of interest within a set time frame. In this paper, we present the multimedia retrieval system vitrivr-VR, the latest addition to the vitrivr stack, which participated in the LSC in recent years. vitrivr-VR leverages the 3D space in virtual reality (VR) to offer novel retrieval and user interaction models, which we describe with a special focus on design decisions taken for the participation in the LSC.","PeriodicalId":150532,"journal":{"name":"Proceedings of the 4th Annual on Lifelog Search Challenge","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2021-08-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121517122","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Enhanced SOMHunter for Known-item Search in Lifelog Data","authors":"Jakub Lokoč, František Mejzlík, Patrik Veselý, Tomás Soucek","doi":"10.1145/3463948.3469074","DOIUrl":"https://doi.org/10.1145/3463948.3469074","url":null,"abstract":"SOMHunter represents a modern light-weight framework for known-item search in datasets of visual data like images or videos. The framework combines an effective W2VV++ text-to-image search approach, a traditional Bayesian like model for maintenance of relevance scores influenced by positive examples, and several types of exploration and exploitation displays. With this initial setting in 2020, already the first prototype of the system turned out to be highly competitive in comparison with other state-of-the-art systems at Video Browser Showdown and Lifelog Search Challenge competitions. In this paper, we present a new version of the system further extending the list of visual data search capabilities. The new version combines localized text queries with collage queries tested at VBS 2021 in two separate systems by our team. Furthermore, the new version of SOMHunter will integrate also the new CLIP text search model recently released by OpenAI. We believe that all the extensions will improve chances to effectively initialize the search that can continue with already supported browsing capabilities.","PeriodicalId":150532,"journal":{"name":"Proceedings of the 4th Annual on Lifelog Search Challenge","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2021-08-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116765322","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Interactive Multimodal Lifelog Retrieval with vitrivr at LSC 2021","authors":"Silvan Heller, Ralph Gasser, Mahnaz Parian-Scherb, Sanja Popovic, Luca Rossetto, Loris Sauter, Florian Spiess, H. Schuldt","doi":"10.1145/3463948.3469062","DOIUrl":"https://doi.org/10.1145/3463948.3469062","url":null,"abstract":"The Lifelog Search Challenge (LSC) is an annual benchmarking competition for interactive multimedia retrieval systems, where participating systems compete in finding events based on textual descriptions containing hints about structured, semi-structured, and/or unstructured data. In this paper, we present the multimedia retrieval system vitrivr, a long-time participant to LSC, with a focus on new functionality. Specifically, we introduce the image stabilisation module which is added prior to the feature extraction to reduce the image degradation caused by lifelogger movements, and discuss how geodata is used during query formulation, query execution, and result presentation.","PeriodicalId":150532,"journal":{"name":"Proceedings of the 4th Annual on Lifelog Search Challenge","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2021-08-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131752502","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"LifeSeeker 3.0: An Interactive Lifelog Search Engine for LSC'21","authors":"Thao-Nhu Nguyen, Tu-Khiem Le, Van-Tu Ninh, M. Tran, Nguyen Thanh Binh, G. Healy, A. Caputo, C. Gurrin","doi":"10.1145/3463948.3469065","DOIUrl":"https://doi.org/10.1145/3463948.3469065","url":null,"abstract":"In this paper, we present the interactive lifelog retrieval engine developed for the LSC'21 comparative benchmarking challenge. The LifeSeeker 3.0 interactive lifelog retrieval engine is an enhanced version of our previous system participating in LSC'20 - LifeSeeker 2.0. The system is developed by both Dublin City University and the Ho Chi Minh City University of Science. The implementation of LifeSeeker 3.0 focuses on searching and filtering by text query using a weighted Bag-of-Words model with visual concept augmentation and three weighted vocabularies. The visual similarity search is improved using a bag of local convolutional features; while improving the previous version's performance, enhancing query processing time, result displaying, and browsing support.","PeriodicalId":150532,"journal":{"name":"Proceedings of the 4th Annual on Lifelog Search Challenge","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2021-08-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115468388","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Lifelogging as a Memory Prosthetic","authors":"A. Smeaton","doi":"10.1145/3463948.3469271","DOIUrl":"https://doi.org/10.1145/3463948.3469271","url":null,"abstract":"Since computers were first used to address the challenge of managing information rather than performing calculations computing arithmetic values, or even before that since the time that MEMEX was designed by Vannevar Bush in the 1940s, we have been building systems that help people like us to find information accurately and quickly. These systems have grown to be technological marvels, discovering and indexing information almost as soon as it appears available online and making it available to billions of people for searching and delivery within fractions of a second, and across a range of devices. Yet it is well known that half the time people are actually searching for things that they once knew but have since forgotten, or can't remember where they found that information first time around, and need to re-find it. As our science of information seeking and information discovery has progressed, we rarely ask why people forgot those things in the first place. If we were allowed to jump back in time say 50 years, and to re-start the development of information retrieval as a technology then perhaps we would be build systems that help us to remember and to learn, rather than trying to plug the gap and find information for us when we forget. In separate but parallel and sometimes overlapping developments, the analysis and indexing of visual information -- images and video -- has also made spectacular progress mostly within the last decade. Using automated processes we can detect and track objects, we can describe visual content as tags or even as text captions, we can now generate realistic high quality visual content using machine learning and we can compute high-level abstract features of visual content like salience, aesthetics, and even memorability. One of the areas where information management/retrieval with its 50 years of technological progress meets computer vision with its recent decade of spectacular development is in lifelogging. At this intersection we can apply computer vision techniques to analyse and index visual lifelogs generated from wearable cameras, for example, in order to support lifelog search and browsing tasks. But we should ask ourselves whether this really is the right way for us to use our lifelogs. Memory is one of the core features that make us what we are yet it is fragile and only partly understood. We have no real control over what we remember and what we forget and when we really do need to remember something that could be important then we make ham-fisted efforts to consciously over-ride our natural tendency to forget. We do this, for example, rehearsing and replaying information, building on the Ebbinghaus principle of repeated conscious reviewing to overcome transience which is the general deterioration of memory over time. In this presentation I will probe deeper into memory, recall, recognition, memorability and memory triggers and how our lifelogs could really act as memory prosthetics, visual triggers for our ","PeriodicalId":150532,"journal":{"name":"Proceedings of the 4th Annual on Lifelog Search Challenge","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2021-08-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131384187","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Exploring Graph-querying approaches in LifeGraph","authors":"Luca Rossetto, Matthias Baumgartner, Ralph Gasser, Lucien Heitz, Ruijie Wang, A. Bernstein","doi":"10.1145/3463948.3469068","DOIUrl":"https://doi.org/10.1145/3463948.3469068","url":null,"abstract":"The multi-modal and interrelated nature of lifelog data makes it well suited for graph-based representations. In this paper, we present the second iteration of LifeGraph, a Knowledge Graph for Lifelog Data, initially introduced during the 3rd Lifelog Search Challenge in 2020. This second iteration incorporates several lessons learned from the previous version. While the actual graph has undergone only small changes, the mechanisms by which it is traversed during querying as well as the underlying storage system which performs the traversal have been changed. The means for query formulation have also been slightly extended in capability and made more efficient and intuitive. All these changes have the aim of improving result quality and reducing query time.","PeriodicalId":150532,"journal":{"name":"Proceedings of the 4th Annual on Lifelog Search Challenge","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2021-08-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131195021","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Flexible Interactive Retrieval SysTem 2.0 for Visual Lifelog Exploration at LSC 2021","authors":"Hoang-Phuc Trang-Trung, Thanh-Cong Le, Mai-Khiem Tran, Van-Tu Ninh, Tu-Khiem Le, C. Gurrin, Minh-Triet Tran","doi":"10.1145/3463948.3469072","DOIUrl":"https://doi.org/10.1145/3463948.3469072","url":null,"abstract":"With a huge collection of photos and video clips, it is essential to provide an efficient and easy-to-use system for users to retrieve moments of interest with a wide variation of query types. This motivates us to develop and upgrade our flexible interactive retrieval system for visual lifelog exploration. In this paper, we briefly introduce version 2 of our system with the following main features. Our system supports multiple modalities for interaction and query processing, including visual query by meta-data, text query and visual information matching based on a joint embedding model, scene clustering based on visual and location information, flexible temporal event navigation, and query expansion with visual examples. With the flexibility in system architecture, we expect our system can easily integrate new modules to enhance its functionalities.","PeriodicalId":150532,"journal":{"name":"Proceedings of the 4th Annual on Lifelog Search Challenge","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2021-08-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116968791","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"ViRMA: Virtual Reality Multimedia Analytics at LSC 2021","authors":"Aaron Duane, Björn þór Jónsson","doi":"10.1145/3463948.3469067","DOIUrl":"https://doi.org/10.1145/3463948.3469067","url":null,"abstract":"In this paper we describe the first iteration of the ViRMA prototype system, a novel approach to multimedia analysis in virtual reality and inspired by the M3 data model. We intend to evaluate our approach via the Lifelog Search Challenge (LSC) to serve as a benchmark against other multimedia analytics systems.","PeriodicalId":150532,"journal":{"name":"Proceedings of the 4th Annual on Lifelog Search Challenge","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2021-08-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130976411","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"LifeConcept: An Interactive Approach for Multimodal Lifelog Retrieval through Concept Recommendation","authors":"Wei-Hong Ang, An-Zi Yen, Tai-Te Chu, Hen-Hsen Huang, Hsin-Hsi Chen","doi":"10.1145/3463948.3469070","DOIUrl":"https://doi.org/10.1145/3463948.3469070","url":null,"abstract":"The major challenge in visual lifelog retrieval is the semantic gap between textual queries and visual concepts. This paper presents our work on the Lifelog Search Challenge 2021 (LSC'21), an annual comparative benchmarking activity for comparing approaches to interactive retrieval from multimodal lifelogs. We propose LifeConcept, an interactive lifelog search system that is aimed at accelerating the retrieval process and retrieving more precise results. In this work, we introduce several new features such as the number of people, location cluster, and object with color. Moreover, we obtain visual concepts from the images with computer vision models and propose a concept recommendation method to reduce the semantic gap. In this way, users can efficiently set up the related conditions for their requirements and search the desired images with appropriate query terms based on the suggestion.","PeriodicalId":150532,"journal":{"name":"Proceedings of the 4th Annual on Lifelog Search Challenge","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2021-08-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131197290","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}