Audio-Language Datasets of Scenes and Events: A Survey

IF 3.4 3区计算机科学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

IEEE Access Pub Date : 2025-01-27 DOI:10.1109/ACCESS.2025.3534621

Gijs Wijngaard;Elia Formisano;Michele Esposito;Michel Dumontier

{"title":"Audio-Language Datasets of Scenes and Events: A Survey","authors":"Gijs Wijngaard;Elia Formisano;Michele Esposito;Michel Dumontier","doi":"10.1109/ACCESS.2025.3534621","DOIUrl":null,"url":null,"abstract":"Audio-language models (ALMs) generate linguistic descriptions of sound-producing events and scenes. Advances in dataset creation and computational power have led to significant progress in this domain. This paper surveys 69 datasets used to train ALMs, covering research up to September 2024 (<uri>https://github.com/GLJS/audio-datasets</uri>). The survey provides a comprehensive analysis of dataset origins, audio and linguistic characteristics, and use cases. Key sources include YouTube-based datasets such as AudioSet, with over two million samples, and community platforms such as Freesound, with over one million samples. The survey evaluates acoustic and linguistic variability across datasets through principal component analysis of audio and text embeddings. The survey also analyzes data leakage through CLAP embeddings, and examines sound category distributions to identify imbalances. Finally, the survey identifies key challenges in developing large, diverse datasets to enhance ALM performance, including dataset overlap, biases, accessibility barriers, and the predominance of English-language content, while highlighting specific areas requiring attention: multilingual dataset development, specialized domain coverage and improved dataset accessibility.","PeriodicalId":13079,"journal":{"name":"IEEE Access","volume":"13 ","pages":"20328-20360"},"PeriodicalIF":3.4000,"publicationDate":"2025-01-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10854210","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"IEEE Access","FirstCategoryId":"94","ListUrlMain":"https://ieeexplore.ieee.org/document/10854210/","RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"COMPUTER SCIENCE, INFORMATION SYSTEMS","Score":null,"Total":0}

引用次数: 0

Abstract

Audio-language models (ALMs) generate linguistic descriptions of sound-producing events and scenes. Advances in dataset creation and computational power have led to significant progress in this domain. This paper surveys 69 datasets used to train ALMs, covering research up to September 2024 (https://github.com/GLJS/audio-datasets). The survey provides a comprehensive analysis of dataset origins, audio and linguistic characteristics, and use cases. Key sources include YouTube-based datasets such as AudioSet, with over two million samples, and community platforms such as Freesound, with over one million samples. The survey evaluates acoustic and linguistic variability across datasets through principal component analysis of audio and text embeddings. The survey also analyzes data leakage through CLAP embeddings, and examines sound category distributions to identify imbalances. Finally, the survey identifies key challenges in developing large, diverse datasets to enhance ALM performance, including dataset overlap, biases, accessibility barriers, and the predominance of English-language content, while highlighting specific areas requiring attention: multilingual dataset development, specialized domain coverage and improved dataset accessibility.

查看原文本刊更多论文

求助全文

约1分钟内获得全文求助全文

来源期刊

IEEE Access COMPUTER SCIENCE, INFORMATION SYSTEMSENGIN-ENGINEERING, ELECTRICAL & ELECTRONIC

CiteScore

9.80

自引率

7.70%

发文量

6673

审稿时长

6 weeks

期刊介绍： IEEE Access® is a multidisciplinary, open access (OA), applications-oriented, all-electronic archival journal that continuously presents the results of original research or development across all of IEEE''s fields of interest. IEEE Access will publish articles that are of high interest to readers, original, technically correct, and clearly presented. Supported by author publication charges (APC), its hallmarks are a rapid peer review and publication process with open access to all readers. Unlike IEEE''s traditional Transactions or Journals, reviews are "binary", in that reviewers will either Accept or Reject an article in the form it is submitted in order to achieve rapid turnaround. Especially encouraged are submissions on: Multidisciplinary topics, or applications-oriented articles and negative results that do not fit within the scope of IEEE''s traditional journals. Practical articles discussing new experiments or measurement techniques, interesting solutions to engineering. Development of new or improved fabrication or manufacturing techniques. Reviews or survey articles of new or evolving fields oriented to assist others in understanding the new area.