DETECLAP: Enhancing Audio-Visual Representation Learning with Object Information

Shota Nakada, Taichi Nishimura, Hokuto Munakata, Masayoshi Kondo, Tatsuya Komatsu
{"title":"DETECLAP: Enhancing Audio-Visual Representation Learning with Object Information","authors":"Shota Nakada, Taichi Nishimura, Hokuto Munakata, Masayoshi Kondo, Tatsuya Komatsu","doi":"arxiv-2409.11729","DOIUrl":null,"url":null,"abstract":"Current audio-visual representation learning can capture rough object\ncategories (e.g., ``animals'' and ``instruments''), but it lacks the ability to\nrecognize fine-grained details, such as specific categories like ``dogs'' and\n``flutes'' within animals and instruments. To address this issue, we introduce\nDETECLAP, a method to enhance audio-visual representation learning with object\ninformation. Our key idea is to introduce an audio-visual label prediction loss\nto the existing Contrastive Audio-Visual Masked AutoEncoder to enhance its\nobject awareness. To avoid costly manual annotations, we prepare object labels\nfrom both audio and visual inputs using state-of-the-art language-audio models\nand object detectors. We evaluate the method of audio-visual retrieval and\nclassification using the VGGSound and AudioSet20K datasets. Our method achieves\nimprovements in recall@10 of +1.5% and +1.2% for audio-to-visual and\nvisual-to-audio retrieval, respectively, and an improvement in accuracy of\n+0.6% for audio-visual classification.","PeriodicalId":501284,"journal":{"name":"arXiv - EE - Audio and Speech Processing","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2024-09-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - EE - Audio and Speech Processing","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2409.11729","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

Abstract

Current audio-visual representation learning can capture rough object categories (e.g., ``animals'' and ``instruments''), but it lacks the ability to recognize fine-grained details, such as specific categories like ``dogs'' and ``flutes'' within animals and instruments. To address this issue, we introduce DETECLAP, a method to enhance audio-visual representation learning with object information. Our key idea is to introduce an audio-visual label prediction loss to the existing Contrastive Audio-Visual Masked AutoEncoder to enhance its object awareness. To avoid costly manual annotations, we prepare object labels from both audio and visual inputs using state-of-the-art language-audio models and object detectors. We evaluate the method of audio-visual retrieval and classification using the VGGSound and AudioSet20K datasets. Our method achieves improvements in recall@10 of +1.5% and +1.2% for audio-to-visual and visual-to-audio retrieval, respectively, and an improvement in accuracy of +0.6% for audio-visual classification.
DETECLAP:利用对象信息加强视听表征学习
目前的视听表征学习可以捕捉粗略的对象类别(例如 "动物 "和 "乐器"),但缺乏识别细粒度细节的能力,例如动物和乐器中的 "狗 "和 "长笛 "等具体类别。为了解决这个问题,我们引入了DETECLAP,这是一种利用对象信息增强视听表征学习的方法。我们的主要想法是在现有的对比视听屏蔽自动编码器中引入视听标签预测,以增强其对象感知能力。为了避免昂贵的人工标注,我们使用最先进的语言-音频模型和对象检测器从音频和视觉输入中准备对象标签。我们使用 VGGSound 和 AudioSet20K 数据集对音视频检索和分类方法进行了评估。我们的方法在音频到视频和视频到音频检索方面的召回率@10分别提高了+1.5%和+1.2%,在音频到视频分类方面的准确率提高了+0.6%。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信