基于复杂环境的中文连续手语数据集

Qidan Zhu, Jing Li, Fei Yuan, Jiaojiao Fan, Quan Gan
{"title":"基于复杂环境的中文连续手语数据集","authors":"Qidan Zhu, Jing Li, Fei Yuan, Jiaojiao Fan, Quan Gan","doi":"arxiv-2409.11960","DOIUrl":null,"url":null,"abstract":"The current bottleneck in continuous sign language recognition (CSLR)\nresearch lies in the fact that most publicly available datasets are limited to\nlaboratory environments or television program recordings, resulting in a single\nbackground environment with uniform lighting, which significantly deviates from\nthe diversity and complexity found in real-life scenarios. To address this\nchallenge, we have constructed a new, large-scale dataset for Chinese\ncontinuous sign language (CSL) based on complex environments, termed the\ncomplex environment - chinese sign language dataset (CE-CSL). This dataset\nencompasses 5,988 continuous CSL video clips collected from daily life scenes,\nfeaturing more than 70 different complex backgrounds to ensure\nrepresentativeness and generalization capability. To tackle the impact of\ncomplex backgrounds on CSLR performance, we propose a time-frequency network\n(TFNet) model for continuous sign language recognition. This model extracts\nframe-level features and then utilizes both temporal and spectral information\nto separately derive sequence features before fusion, aiming to achieve\nefficient and accurate CSLR. Experimental results demonstrate that our approach\nachieves significant performance improvements on the CE-CSL, validating its\neffectiveness under complex background conditions. Additionally, our proposed\nmethod has also yielded highly competitive results when applied to three\npublicly available CSL datasets.","PeriodicalId":501130,"journal":{"name":"arXiv - CS - Computer Vision and Pattern Recognition","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2024-09-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"A Chinese Continuous Sign Language Dataset Based on Complex Environments\",\"authors\":\"Qidan Zhu, Jing Li, Fei Yuan, Jiaojiao Fan, Quan Gan\",\"doi\":\"arxiv-2409.11960\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"The current bottleneck in continuous sign language recognition (CSLR)\\nresearch lies in the fact that most publicly available datasets are limited to\\nlaboratory environments or television program recordings, resulting in a single\\nbackground environment with uniform lighting, which significantly deviates from\\nthe diversity and complexity found in real-life scenarios. To address this\\nchallenge, we have constructed a new, large-scale dataset for Chinese\\ncontinuous sign language (CSL) based on complex environments, termed the\\ncomplex environment - chinese sign language dataset (CE-CSL). This dataset\\nencompasses 5,988 continuous CSL video clips collected from daily life scenes,\\nfeaturing more than 70 different complex backgrounds to ensure\\nrepresentativeness and generalization capability. To tackle the impact of\\ncomplex backgrounds on CSLR performance, we propose a time-frequency network\\n(TFNet) model for continuous sign language recognition. This model extracts\\nframe-level features and then utilizes both temporal and spectral information\\nto separately derive sequence features before fusion, aiming to achieve\\nefficient and accurate CSLR. Experimental results demonstrate that our approach\\nachieves significant performance improvements on the CE-CSL, validating its\\neffectiveness under complex background conditions. Additionally, our proposed\\nmethod has also yielded highly competitive results when applied to three\\npublicly available CSL datasets.\",\"PeriodicalId\":501130,\"journal\":{\"name\":\"arXiv - CS - Computer Vision and Pattern Recognition\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2024-09-18\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"arXiv - CS - Computer Vision and Pattern Recognition\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/arxiv-2409.11960\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - CS - Computer Vision and Pattern Recognition","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2409.11960","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

摘要

目前连续手语识别(CSLR)研究的瓶颈在于,大多数公开可用的数据集仅限于实验室环境或电视节目录制,导致背景环境单一、光照均匀,与现实生活中的多样性和复杂性大相径庭。为了应对这一挑战,我们构建了一个基于复杂环境的全新大规模中文连续手语(CSL)数据集,称为 "复杂环境-中文手语数据集"(CE-CSL)。该数据集包含 5,988 个从日常生活场景中采集的连续手语视频片段,具有 70 多种不同的复杂背景,以确保代表性和泛化能力。针对复杂背景对 CSLR 性能的影响,我们提出了一种用于连续手语识别的时频网络(TFNet)模型。该模型提取帧级特征,然后利用时间信息和频谱信息分别提取序列特征,再进行融合,从而实现高效、准确的 CSLR。实验结果表明,我们的方法显著提高了 CE-CSL 的性能,验证了它在复杂背景条件下的有效性。此外,我们提出的方法在应用于三个公开的 CSL 数据集时也取得了极具竞争力的结果。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
A Chinese Continuous Sign Language Dataset Based on Complex Environments
The current bottleneck in continuous sign language recognition (CSLR) research lies in the fact that most publicly available datasets are limited to laboratory environments or television program recordings, resulting in a single background environment with uniform lighting, which significantly deviates from the diversity and complexity found in real-life scenarios. To address this challenge, we have constructed a new, large-scale dataset for Chinese continuous sign language (CSL) based on complex environments, termed the complex environment - chinese sign language dataset (CE-CSL). This dataset encompasses 5,988 continuous CSL video clips collected from daily life scenes, featuring more than 70 different complex backgrounds to ensure representativeness and generalization capability. To tackle the impact of complex backgrounds on CSLR performance, we propose a time-frequency network (TFNet) model for continuous sign language recognition. This model extracts frame-level features and then utilizes both temporal and spectral information to separately derive sequence features before fusion, aiming to achieve efficient and accurate CSLR. Experimental results demonstrate that our approach achieves significant performance improvements on the CE-CSL, validating its effectiveness under complex background conditions. Additionally, our proposed method has also yielded highly competitive results when applied to three publicly available CSL datasets.
求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信