对音乐理解预训练语言模型的评估

Yannis Vasilakis, Rachel Bittner, Johan Pauwels
{"title":"对音乐理解预训练语言模型的评估","authors":"Yannis Vasilakis, Rachel Bittner, Johan Pauwels","doi":"arxiv-2409.11449","DOIUrl":null,"url":null,"abstract":"Music-text multimodal systems have enabled new approaches to Music\nInformation Research (MIR) applications such as audio-to-text and text-to-audio\nretrieval, text-based song generation, and music captioning. Despite the\nreported success, little effort has been put into evaluating the musical\nknowledge of Large Language Models (LLM). In this paper, we demonstrate that\nLLMs suffer from 1) prompt sensitivity, 2) inability to model negation (e.g.\n'rock song without guitar'), and 3) sensitivity towards the presence of\nspecific words. We quantified these properties as a triplet-based accuracy,\nevaluating the ability to model the relative similarity of labels in a\nhierarchical ontology. We leveraged the Audioset ontology to generate triplets\nconsisting of an anchor, a positive (relevant) label, and a negative (less\nrelevant) label for the genre and instruments sub-tree. We evaluated the\ntriplet-based musical knowledge for six general-purpose Transformer-based\nmodels. The triplets obtained through this methodology required filtering, as\nsome were difficult to judge and therefore relatively uninformative for\nevaluation purposes. Despite the relatively high accuracy reported,\ninconsistencies are evident in all six models, suggesting that off-the-shelf\nLLMs need adaptation to music before use.","PeriodicalId":501284,"journal":{"name":"arXiv - EE - Audio and Speech Processing","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2024-09-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Evaluation of pretrained language models on music understanding\",\"authors\":\"Yannis Vasilakis, Rachel Bittner, Johan Pauwels\",\"doi\":\"arxiv-2409.11449\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"Music-text multimodal systems have enabled new approaches to Music\\nInformation Research (MIR) applications such as audio-to-text and text-to-audio\\nretrieval, text-based song generation, and music captioning. Despite the\\nreported success, little effort has been put into evaluating the musical\\nknowledge of Large Language Models (LLM). In this paper, we demonstrate that\\nLLMs suffer from 1) prompt sensitivity, 2) inability to model negation (e.g.\\n'rock song without guitar'), and 3) sensitivity towards the presence of\\nspecific words. We quantified these properties as a triplet-based accuracy,\\nevaluating the ability to model the relative similarity of labels in a\\nhierarchical ontology. We leveraged the Audioset ontology to generate triplets\\nconsisting of an anchor, a positive (relevant) label, and a negative (less\\nrelevant) label for the genre and instruments sub-tree. We evaluated the\\ntriplet-based musical knowledge for six general-purpose Transformer-based\\nmodels. The triplets obtained through this methodology required filtering, as\\nsome were difficult to judge and therefore relatively uninformative for\\nevaluation purposes. Despite the relatively high accuracy reported,\\ninconsistencies are evident in all six models, suggesting that off-the-shelf\\nLLMs need adaptation to music before use.\",\"PeriodicalId\":501284,\"journal\":{\"name\":\"arXiv - EE - Audio and Speech Processing\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2024-09-17\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"arXiv - EE - Audio and Speech Processing\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/arxiv-2409.11449\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - EE - Audio and Speech Processing","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2409.11449","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

摘要

音乐-文本多模态系统为音乐信息研究(MIR)应用提供了新方法,如音频-文本和文本-音频检索、基于文本的歌曲生成和音乐字幕。尽管取得了巨大的成功,但在评估大型语言模型(LLM)的音乐知识方面却鲜有建树。在本文中,我们证明了大型语言模型存在以下问题:1)对提示敏感;2)无法对否定进行建模(例如 "没有吉他的摇滚歌曲");3)对特定词语的存在敏感。我们将这些特性量化为基于三元组的准确度,评估对层次本体中标签的相对相似性进行建模的能力。我们利用 Audioset 本体论为流派和乐器子树生成由一个锚点、一个正面(相关)标签和一个负面(不太相关)标签组成的三元组。我们对六个基于通用转换器的模型进行了基于三元组的音乐知识评估。通过这种方法获得的三连音需要过滤,其中一些很难判断,因此对评估目的而言信息量相对较小。尽管报告的准确率相对较高,但所有六个模型都存在明显的不一致性,这表明现成的LLM 在使用前需要适应音乐。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
Evaluation of pretrained language models on music understanding
Music-text multimodal systems have enabled new approaches to Music Information Research (MIR) applications such as audio-to-text and text-to-audio retrieval, text-based song generation, and music captioning. Despite the reported success, little effort has been put into evaluating the musical knowledge of Large Language Models (LLM). In this paper, we demonstrate that LLMs suffer from 1) prompt sensitivity, 2) inability to model negation (e.g. 'rock song without guitar'), and 3) sensitivity towards the presence of specific words. We quantified these properties as a triplet-based accuracy, evaluating the ability to model the relative similarity of labels in a hierarchical ontology. We leveraged the Audioset ontology to generate triplets consisting of an anchor, a positive (relevant) label, and a negative (less relevant) label for the genre and instruments sub-tree. We evaluated the triplet-based musical knowledge for six general-purpose Transformer-based models. The triplets obtained through this methodology required filtering, as some were difficult to judge and therefore relatively uninformative for evaluation purposes. Despite the relatively high accuracy reported, inconsistencies are evident in all six models, suggesting that off-the-shelf LLMs need adaptation to music before use.
求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信