Sign-to-speech translation using machine-learning-assisted stretchable sensor arrays

IF 33.7 1区工程技术 Q1 ENGINEERING, ELECTRICAL & ELECTRONIC

Nature Electronics Pub Date : 2020-06-29 DOI:10.1038/s41928-020-0428-6

Zhihao Zhou, Kyle Chen, Xiaoshi Li, Songlin Zhang, Yufen Wu, Yihao Zhou, Keyu Meng, Chenchen Sun, Qiang He, Wenjing Fan, Endong Fan, Zhiwei Lin, Xulong Tan, Weili Deng, Jin Yang, Jun Chen

{"title":"Sign-to-speech translation using machine-learning-assisted stretchable sensor arrays","authors":"Zhihao Zhou, Kyle Chen, Xiaoshi Li, Songlin Zhang, Yufen Wu, Yihao Zhou, Keyu Meng, Chenchen Sun, Qiang He, Wenjing Fan, Endong Fan, Zhiwei Lin, Xulong Tan, Weili Deng, Jin Yang, Jun Chen","doi":"10.1038/s41928-020-0428-6","DOIUrl":null,"url":null,"abstract":"Signed languages are not as pervasive a conversational medium as spoken languages due to the history of institutional suppression of the former and the linguistic hegemony of the latter. This has led to a communication barrier between signers and non-signers that could be mitigated by technology-mediated approaches. Here, we show that a wearable sign-to-speech translation system, assisted by machine learning, can accurately translate the hand gestures of American Sign Language into speech. The wearable sign-to-speech translation system is composed of yarn-based stretchable sensor arrays and a wireless printed circuit board, and offers a high sensitivity and fast response time, allowing real-time translation of signs into spoken words to be performed. By analysing 660 acquired sign language hand gesture recognition patterns, we demonstrate a recognition rate of up to 98.63% and a recognition time of less than 1 s. Wearable yarn-based stretchable sensor arrays, combined with machine learning, can be used to translate American Sign Language into speech in real time.","PeriodicalId":19064,"journal":{"name":"Nature Electronics","volume":null,"pages":null},"PeriodicalIF":33.7000,"publicationDate":"2020-06-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1038/s41928-020-0428-6","citationCount":"394","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Nature Electronics","FirstCategoryId":"5","ListUrlMain":"https://www.nature.com/articles/s41928-020-0428-6","RegionNum":1,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"ENGINEERING, ELECTRICAL & ELECTRONIC","Score":null,"Total":0}

引用次数: 394

Abstract

Signed languages are not as pervasive a conversational medium as spoken languages due to the history of institutional suppression of the former and the linguistic hegemony of the latter. This has led to a communication barrier between signers and non-signers that could be mitigated by technology-mediated approaches. Here, we show that a wearable sign-to-speech translation system, assisted by machine learning, can accurately translate the hand gestures of American Sign Language into speech. The wearable sign-to-speech translation system is composed of yarn-based stretchable sensor arrays and a wireless printed circuit board, and offers a high sensitivity and fast response time, allowing real-time translation of signs into spoken words to be performed. By analysing 660 acquired sign language hand gesture recognition patterns, we demonstrate a recognition rate of up to 98.63% and a recognition time of less than 1 s. Wearable yarn-based stretchable sensor arrays, combined with machine learning, can be used to translate American Sign Language into speech in real time.

Abstract Image

查看原文本刊更多论文

利用机器学习辅助可拉伸传感器阵列实现手势到语音的翻译

由于前者在制度上受到压制，而后者在语言上占据霸权地位，手语并不像口语那样是一种普遍的会话媒介。这就造成了手语使用者与非手语使用者之间的交流障碍，而这种障碍可以通过技术手段来缓解。在这里，我们展示了一种可穿戴的手语转语音翻译系统，在机器学习的辅助下，可以准确地将美国手语的手势翻译成语音。可穿戴手语翻译系统由基于纱线的可拉伸传感器阵列和无线印刷电路板组成，灵敏度高、响应速度快，可将手势实时翻译成口语。通过分析获得的 660 个手语手势识别模式，我们展示了高达 98.63% 的识别率和小于 1 秒的识别时间。基于纱线的可伸缩可穿戴传感器阵列与机器学习相结合，可用于将美国手语实时翻译成语音。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Nature Electronics Engineering-Electrical and Electronic Engineering

CiteScore

47.50

自引率

2.30%

发文量

159

期刊介绍： Nature Electronics is a comprehensive journal that publishes both fundamental and applied research in the field of electronics. It encompasses a wide range of topics, including the study of new phenomena and devices, the design and construction of electronic circuits, and the practical applications of electronics. In addition, the journal explores the commercial and industrial aspects of electronics research. The primary focus of Nature Electronics is on the development of technology and its potential impact on society. The journal incorporates the contributions of scientists, engineers, and industry professionals, offering a platform for their research findings. Moreover, Nature Electronics provides insightful commentary, thorough reviews, and analysis of the key issues that shape the field, as well as the technologies that are reshaping society. Like all journals within the prestigious Nature brand, Nature Electronics upholds the highest standards of quality. It maintains a dedicated team of professional editors and follows a fair and rigorous peer-review process. The journal also ensures impeccable copy-editing and production, enabling swift publication. Additionally, Nature Electronics prides itself on its editorial independence, ensuring unbiased and impartial reporting. In summary, Nature Electronics is a leading journal that publishes cutting-edge research in electronics. With its multidisciplinary approach and commitment to excellence, the journal serves as a valuable resource for scientists, engineers, and industry professionals seeking to stay at the forefront of advancements in the field.