An approach based on deep learning for Indian sign language translation

Kinjal Mistree, D. Thakor, Brijesh Bhatt
{"title":"An approach based on deep learning for Indian sign language translation","authors":"Kinjal Mistree, D. Thakor, Brijesh Bhatt","doi":"10.1108/ijicc-08-2022-0227","DOIUrl":null,"url":null,"abstract":"PurposeAccording to the Indian Sign Language Research and Training Centre (ISLRTC), India has approximately 300 certified human interpreters to help people with hearing loss. This paper aims to address the issue of Indian Sign Language (ISL) sentence recognition and translation into semantically equivalent English text in a signer-independent mode.Design/methodology/approachThis study presents an approach that translates ISL sentences into English text using the MobileNetV2 model and Neural Machine Translation (NMT). The authors have created an ISL corpus from the Brown corpus using ISL grammar rules to perform machine translation. The authors’ approach converts ISL videos of the newly created dataset into ISL gloss sequences using the MobileNetV2 model and the recognized ISL gloss sequence is then fed to a machine translation module that generates an English sentence for each ISL sentence.FindingsAs per the experimental results, pretrained MobileNetV2 model was proven the best-suited model for the recognition of ISL sentences and NMT provided better results than Statistical Machine Translation (SMT) to convert ISL text into English text. The automatic and human evaluation of the proposed approach yielded accuracies of 83.3 and 86.1%, respectively.Research limitations/implicationsIt can be seen that the neural machine translation systems produced translations with repetitions of other translated words, strange translations when the total number of words per sentence is increased and one or more unexpected terms that had no relation to the source text on occasion. The most common type of error is the mistranslation of places, numbers and dates. Although this has little effect on the overall structure of the translated sentence, it indicates that the embedding learned for these few words could be improved.Originality/valueSign language recognition and translation is a crucial step toward improving communication between the deaf and the rest of society. Because of the shortage of human interpreters, an alternative approach is desired to help people achieve smooth communication with the Deaf. To motivate research in this field, the authors generated an ISL corpus of 13,720 sentences and a video dataset of 47,880 ISL videos. As there is no public dataset available for ISl videos incorporating signs released by ISLRTC, the authors created a new video dataset and ISL corpus.","PeriodicalId":352072,"journal":{"name":"Int. J. Intell. Comput. Cybern.","volume":"350 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2022-12-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Int. J. Intell. Comput. Cybern.","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1108/ijicc-08-2022-0227","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

Abstract

PurposeAccording to the Indian Sign Language Research and Training Centre (ISLRTC), India has approximately 300 certified human interpreters to help people with hearing loss. This paper aims to address the issue of Indian Sign Language (ISL) sentence recognition and translation into semantically equivalent English text in a signer-independent mode.Design/methodology/approachThis study presents an approach that translates ISL sentences into English text using the MobileNetV2 model and Neural Machine Translation (NMT). The authors have created an ISL corpus from the Brown corpus using ISL grammar rules to perform machine translation. The authors’ approach converts ISL videos of the newly created dataset into ISL gloss sequences using the MobileNetV2 model and the recognized ISL gloss sequence is then fed to a machine translation module that generates an English sentence for each ISL sentence.FindingsAs per the experimental results, pretrained MobileNetV2 model was proven the best-suited model for the recognition of ISL sentences and NMT provided better results than Statistical Machine Translation (SMT) to convert ISL text into English text. The automatic and human evaluation of the proposed approach yielded accuracies of 83.3 and 86.1%, respectively.Research limitations/implicationsIt can be seen that the neural machine translation systems produced translations with repetitions of other translated words, strange translations when the total number of words per sentence is increased and one or more unexpected terms that had no relation to the source text on occasion. The most common type of error is the mistranslation of places, numbers and dates. Although this has little effect on the overall structure of the translated sentence, it indicates that the embedding learned for these few words could be improved.Originality/valueSign language recognition and translation is a crucial step toward improving communication between the deaf and the rest of society. Because of the shortage of human interpreters, an alternative approach is desired to help people achieve smooth communication with the Deaf. To motivate research in this field, the authors generated an ISL corpus of 13,720 sentences and a video dataset of 47,880 ISL videos. As there is no public dataset available for ISl videos incorporating signs released by ISLRTC, the authors created a new video dataset and ISL corpus.
基于深度学习的印度手语翻译方法
根据印度手语研究和培训中心(ISLRTC)的数据,印度有大约300名经过认证的口译员来帮助听力损失的人。本文旨在研究在手语独立模式下,印度手语(ISL)句子识别和翻译成语义等效的英语文本问题。设计/方法/方法本研究提出了一种使用MobileNetV2模型和神经机器翻译(NMT)将ISL句子翻译成英语文本的方法。作者从布朗语料库中创建了一个ISL语料库,使用ISL语法规则进行机器翻译。作者的方法使用MobileNetV2模型将新创建数据集的ISL视频转换为ISL光泽序列,然后将识别的ISL光泽序列馈送到机器翻译模块,该模块为每个ISL句子生成一个英语句子。实验结果表明,预训练的MobileNetV2模型是最适合于ISL句子识别的模型,NMT在将ISL文本转换为英语文本方面的效果优于统计机器翻译(SMT)。该方法的自动和人工评估的准确率分别为83.3%和86.1%。研究局限/启示可以看到,神经机器翻译系统产生的译文中有其他翻译单词的重复,每句单词总数增加时的奇怪译文,以及偶尔出现一个或多个与源文本无关的意外术语。最常见的错误是地点、数字和日期的误译。虽然这对翻译句子的整体结构影响不大,但这表明对这几个词的嵌入学习是可以提高的。设计语言识别和翻译是改善聋人与社会其他人之间沟通的关键一步。由于人工口译人员的短缺,需要一种替代方法来帮助人们与聋人顺利沟通。为了激发这一领域的研究,作者生成了一个包含13720个句子的ISL语料库和一个包含47880个ISL视频的视频数据集。由于没有包含ISLRTC发布的符号的ISl视频的公共数据集,作者创建了一个新的视频数据集和ISl语料库。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信