ArXiv.org最新文献

筛选

英文中文

Cross-Modal Learning for Music-to-Music-Video Description Generation 音乐到音乐视频描述生成的跨模态学习

ArXiv.org Pub Date : 2025-03-14 DOI: 10.48550/arxiv.2503.11190

Zhuoyuan Mao, Mengjie Zhao, Qiyu Wu, Zhi Zhong, Wei‐Hsiang Liao, Hiromi Wakaki, Yuki Mitsufuji

{"title":"Cross-Modal Learning for Music-to-Music-Video Description Generation","authors":"Zhuoyuan Mao, Mengjie Zhao, Qiyu Wu, Zhi Zhong, Wei‐Hsiang Liao, Hiromi Wakaki, Yuki Mitsufuji","doi":"10.48550/arxiv.2503.11190","DOIUrl":"https://doi.org/10.48550/arxiv.2503.11190","url":null,"abstract":"Music-to-music-video generation is a challenging task due to the intrinsic differences between the music and video modalities. The advent of powerful text-to-video diffusion models has opened a promising pathway for music-video (MV) generation by first addressing the music-to-MV description task and subsequently leveraging these models for video generation. In this study, we focus on the MV description generation task and propose a comprehensive pipeline encompassing training data construction and multimodal model fine-tuning. We fine-tune existing pre-trained multimodal models on our newly constructed music-to-MV description dataset based on the Music4All dataset, which integrates both musical and visual information. Our experimental results demonstrate that music representations can be effectively mapped to textual domains, enabling the generation of meaningful MV description directly from music inputs. We also identify key components in the dataset construction pipeline that critically impact the quality of MV description and highlight specific musical attributes that warrant greater focus for improved MV description generation.","PeriodicalId":523646,"journal":{"name":"ArXiv.org","volume":"1 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2025-03-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://arxiv.org/pdf/2503.11190","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147919497","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0