Exploring automatic text-to-sign translation in a healthcare setting

IF 2.1 4区计算机科学 Q3 COMPUTER SCIENCE, CYBERNETICS

Universal Access in the Information Society Pub Date : 2023-09-29 DOI:10.1007/s10209-023-01042-6

Lyke Esselink, Floris Roelofsen, Jakub Dotlačil, Shani Mende-Gillings, Maartje de Meulder, Nienke Sijm, Anika Smeijers

{"title":"Exploring automatic text-to-sign translation in a healthcare setting","authors":"Lyke Esselink, Floris Roelofsen, Jakub Dotlačil, Shani Mende-Gillings, Maartje de Meulder, Nienke Sijm, Anika Smeijers","doi":"10.1007/s10209-023-01042-6","DOIUrl":null,"url":null,"abstract":"Abstract Communication between healthcare professionals and deaf patients has been particularly challenging during the COVID-19 pandemic. We have explored the possibility to automatically translate phrases that are frequently used in the diagnosis and treatment of hospital patients, in particular phrases related to COVID-19, from Dutch or English to Dutch Sign Language (NGT). The prototype system we developed displays translations either by means of pre-recorded videos featuring a deaf human signer (for a limited number of sentences) or by means of animations featuring a computer-generated signing avatar (for a larger, though still restricted number of sentences). We evaluated the comprehensibility of the signing avatar, as compared to the human signer. We found that, while individual signs are recognized correctly when signed by the avatar almost as frequently as when signed by a human, sentence comprehension rates and clarity scores for the avatar are substantially lower than for the human signer. We identify a number of concrete limitations of the JASigning avatar engine that underlies our system. Namely, the engine currently does not offer sufficient control over mouth shapes, the relative speed and intensity of signs in a sentence (prosody), and transitions between signs. These limitations need to be overcome in future work for the engine to become usable in practice.","PeriodicalId":49115,"journal":{"name":"Universal Access in the Information Society","volume":null,"pages":null},"PeriodicalIF":2.1000,"publicationDate":"2023-09-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Universal Access in the Information Society","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1007/s10209-023-01042-6","RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"COMPUTER SCIENCE, CYBERNETICS","Score":null,"Total":0}

引用次数: 0

Abstract

Abstract Communication between healthcare professionals and deaf patients has been particularly challenging during the COVID-19 pandemic. We have explored the possibility to automatically translate phrases that are frequently used in the diagnosis and treatment of hospital patients, in particular phrases related to COVID-19, from Dutch or English to Dutch Sign Language (NGT). The prototype system we developed displays translations either by means of pre-recorded videos featuring a deaf human signer (for a limited number of sentences) or by means of animations featuring a computer-generated signing avatar (for a larger, though still restricted number of sentences). We evaluated the comprehensibility of the signing avatar, as compared to the human signer. We found that, while individual signs are recognized correctly when signed by the avatar almost as frequently as when signed by a human, sentence comprehension rates and clarity scores for the avatar are substantially lower than for the human signer. We identify a number of concrete limitations of the JASigning avatar engine that underlies our system. Namely, the engine currently does not offer sufficient control over mouth shapes, the relative speed and intensity of signs in a sentence (prosody), and transitions between signs. These limitations need to be overcome in future work for the engine to become usable in practice.

查看原文本刊更多论文

探索医疗保健设置中的自动文本到签名转换

在COVID-19大流行期间，医护人员与失聪患者之间的沟通尤其具有挑战性。我们探索了将医院患者诊断和治疗中经常使用的短语，特别是与COVID-19相关的短语，从荷兰语或英语自动翻译为荷兰语手语(NGT)的可能性。我们开发的原型系统可以通过预先录制的视频来显示翻译，视频中有一个聋哑人的签名(对于有限数量的句子)，或者通过计算机生成的签名化身的动画来显示翻译(对于更大的，尽管仍然有限数量的句子)。与人类签名者相比，我们评估了签名头像的可理解性。我们发现，虽然由虚拟角色签名的个体符号被正确识别的频率几乎与人类签名时一样高，但虚拟角色的句子理解率和清晰度得分明显低于人类签名者。我们确定了jasign头像引擎的一些具体限制，这些限制是我们系统的基础。也就是说，该引擎目前还不能提供足够的控制口型、句子中符号的相对速度和强度(韵律)以及符号之间的转换。这些限制需要在未来的工作中克服，使发动机在实践中可用。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Universal Access in the Information Society COMPUTER SCIENCE, CYBERNETICS-

CiteScore

6.10

自引率

16.70%

发文量

审稿时长

>12 weeks

期刊介绍： Universal Access in the Information Society (UAIS) is an international, interdisciplinary refereed journal that solicits original research contributions addressing the accessibility, usability, and, ultimately, acceptability of Information Society Technologies by anyone, anywhere, at anytime, and through any media and device. Universal access refers to the conscious and systematic effort to proactively apply principles, methods and tools of universal design order to develop Information Society Technologies that are accessible and usable by all citizens, including the very young and the elderly and people with different types of disabilities, thus avoiding the need for a posteriori adaptations or specialized design. The journal''s unique focus is on theoretical, methodological, and empirical research, of both technological and non-technological nature, that addresses equitable access and active participation of potentially all citizens in the information society.