多模态生活日志检索中外部文本知识集成的交互式方法

Chia-Chun Chang, Min-Huan Fu, Hen-Hsen Huang, Hsin-Hsi Chen
{"title":"多模态生活日志检索中外部文本知识集成的交互式方法","authors":"Chia-Chun Chang, Min-Huan Fu, Hen-Hsen Huang, Hsin-Hsi Chen","doi":"10.1145/3326460.3329163","DOIUrl":null,"url":null,"abstract":"The semantic gap between textual queries and visual concepts is one of the key challenges in lifelog retrieval. This work presents an interactive system aimed at improving the retrieval accuracy by query term suggestion. Besides, this system also assists users to refine the retrieval results by image similarity clustering. For recommending a list of candidate words, we extract visual concepts from images by using computer vision models, and then incorporate both official and additional concepts into our system using pre-trained word embedding, in which textual knowledge is inherent. We also purpose an intelligent mechanism for rapidly removing multiple irrelevant search results. For reaching out this purpose, we build kd-trees [1] offline for reducing the computational overhead and cluster similar images by nearest neighbor search in the embedding space. Whenever users exclude some irrelevant images, their nearest neighbors in the image embedding space are also removed. In this way, users can efficiently screen out the relevant results and purge the irrelevant ones, scanning over more retrieval results in a shorter period of time.","PeriodicalId":266823,"journal":{"name":"Proceedings of the ACM Workshop on Lifelog Search Challenge","volume":"13 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2019-06-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"14","resultStr":"{\"title\":\"An Interactive Approach to Integrating External Textual Knowledge for Multimodal Lifelog Retrieval\",\"authors\":\"Chia-Chun Chang, Min-Huan Fu, Hen-Hsen Huang, Hsin-Hsi Chen\",\"doi\":\"10.1145/3326460.3329163\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"The semantic gap between textual queries and visual concepts is one of the key challenges in lifelog retrieval. This work presents an interactive system aimed at improving the retrieval accuracy by query term suggestion. Besides, this system also assists users to refine the retrieval results by image similarity clustering. For recommending a list of candidate words, we extract visual concepts from images by using computer vision models, and then incorporate both official and additional concepts into our system using pre-trained word embedding, in which textual knowledge is inherent. We also purpose an intelligent mechanism for rapidly removing multiple irrelevant search results. For reaching out this purpose, we build kd-trees [1] offline for reducing the computational overhead and cluster similar images by nearest neighbor search in the embedding space. Whenever users exclude some irrelevant images, their nearest neighbors in the image embedding space are also removed. In this way, users can efficiently screen out the relevant results and purge the irrelevant ones, scanning over more retrieval results in a shorter period of time.\",\"PeriodicalId\":266823,\"journal\":{\"name\":\"Proceedings of the ACM Workshop on Lifelog Search Challenge\",\"volume\":\"13 1\",\"pages\":\"0\"},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2019-06-05\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"14\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Proceedings of the ACM Workshop on Lifelog Search Challenge\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.1145/3326460.3329163\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Proceedings of the ACM Workshop on Lifelog Search Challenge","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1145/3326460.3329163","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 14

摘要

文本查询和视觉概念之间的语义差距是生活日志检索的关键挑战之一。本文提出了一种基于检索词建议的交互式检索系统。此外,该系统还通过图像相似度聚类来帮助用户细化检索结果。为了推荐候选词列表,我们使用计算机视觉模型从图像中提取视觉概念,然后使用预先训练的词嵌入将官方和附加概念合并到我们的系统中,其中文本知识是固有的。我们还设计了一种智能机制来快速删除多个不相关的搜索结果。为了达到这一目的,我们离线构建kd-trees[1]以减少计算开销,并在嵌入空间中通过最近邻搜索对相似图像进行聚类。当用户排除一些不相关的图像时,它们在图像嵌入空间中的近邻也会被去除。这样,用户可以有效地筛选出相关的结果,清除不相关的结果,在更短的时间内浏览更多的检索结果。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
An Interactive Approach to Integrating External Textual Knowledge for Multimodal Lifelog Retrieval
The semantic gap between textual queries and visual concepts is one of the key challenges in lifelog retrieval. This work presents an interactive system aimed at improving the retrieval accuracy by query term suggestion. Besides, this system also assists users to refine the retrieval results by image similarity clustering. For recommending a list of candidate words, we extract visual concepts from images by using computer vision models, and then incorporate both official and additional concepts into our system using pre-trained word embedding, in which textual knowledge is inherent. We also purpose an intelligent mechanism for rapidly removing multiple irrelevant search results. For reaching out this purpose, we build kd-trees [1] offline for reducing the computational overhead and cluster similar images by nearest neighbor search in the embedding space. Whenever users exclude some irrelevant images, their nearest neighbors in the image embedding space are also removed. In this way, users can efficiently screen out the relevant results and purge the irrelevant ones, scanning over more retrieval results in a shorter period of time.
求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信