Visualizing speech styles in captions for deaf and hard-of-hearing viewers

IF 5.3 2区计算机科学 Q1 COMPUTER SCIENCE, CYBERNETICS

International Journal of Human-Computer Studies Pub Date : 2024-10-16 DOI:10.1016/j.ijhcs.2024.103386

SooYeon Ahn , JooYeong Kim , Choonsung Shin , Jin-Hyuk Hong

{"title":"Visualizing speech styles in captions for deaf and hard-of-hearing viewers","authors":"SooYeon Ahn , JooYeong Kim , Choonsung Shin , Jin-Hyuk Hong","doi":"10.1016/j.ijhcs.2024.103386","DOIUrl":null,"url":null,"abstract":"<div><div>Speech styles such as extension, emphasis, and pause play an important role in capturing the audience's attention and conveying a message accurately. Unfortunately, it is challenging for Deaf and Hard-of-Hearing (DHH) people to enjoy these benefits when watching lectures with common captions. In this paper, we propose a new caption system that automatically analyzes speech styles from audio and visualizes them using visualization elements such as punctuation, paint-on, color, and boldness. We conducted a comparative study with 26 DHH viewers and found that the proposed caption system enabled them to recognize the speaker's speech style in lectures. As a result, the DHH viewers were able to watch lecture videos more vividly and were more engaged with the lectures. In particular, punctuation can be a practical solution to visualize speech styles and ensure legibility. Participants expressed a desire to use our caption system in their daily lives, providing valuable insights for future sound-visualized caption research.</div></div>","PeriodicalId":54955,"journal":{"name":"International Journal of Human-Computer Studies","volume":null,"pages":null},"PeriodicalIF":5.3000,"publicationDate":"2024-10-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"International Journal of Human-Computer Studies","FirstCategoryId":"94","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S1071581924001691","RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMPUTER SCIENCE, CYBERNETICS","Score":null,"Total":0}

引用次数: 0

Abstract

Speech styles such as extension, emphasis, and pause play an important role in capturing the audience's attention and conveying a message accurately. Unfortunately, it is challenging for Deaf and Hard-of-Hearing (DHH) people to enjoy these benefits when watching lectures with common captions. In this paper, we propose a new caption system that automatically analyzes speech styles from audio and visualizes them using visualization elements such as punctuation, paint-on, color, and boldness. We conducted a comparative study with 26 DHH viewers and found that the proposed caption system enabled them to recognize the speaker's speech style in lectures. As a result, the DHH viewers were able to watch lecture videos more vividly and were more engaged with the lectures. In particular, punctuation can be a practical solution to visualize speech styles and ensure legibility. Participants expressed a desire to use our caption system in their daily lives, providing valuable insights for future sound-visualized caption research.

查看原文本刊更多论文

为聋人和重听观众可视化字幕中的语言风格

延伸、强调和停顿等演讲方式在吸引听众注意力和准确传达信息方面发挥着重要作用。遗憾的是，聋人和重听人（DHH）在观看使用普通字幕的讲座时，很难享受到这些好处。在本文中，我们提出了一种新的字幕系统，它能自动分析音频中的语音风格，并使用标点符号、上色、颜色和粗体等可视化元素将其可视化。我们对 26 名 DHH 观众进行了对比研究，发现所提出的字幕系统使他们能够识别演讲者在讲座中的演讲风格。因此，DHH 观众能够更生动地观看讲座视频，并更投入地听讲。特别是，标点符号可以成为可视化语言风格和确保可读性的实用解决方案。参与者表示希望在日常生活中使用我们的字幕系统，这为未来的声音可视化字幕研究提供了宝贵的启示。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

International Journal of Human-Computer Studies 工程技术-计算机：控制论

CiteScore

11.50

自引率

5.60%

发文量

108

审稿时长

3 months

期刊介绍： The International Journal of Human-Computer Studies publishes original research over the whole spectrum of work relevant to the theory and practice of innovative interactive systems. The journal is inherently interdisciplinary, covering research in computing, artificial intelligence, psychology, linguistics, communication, design, engineering, and social organization, which is relevant to the design, analysis, evaluation and application of innovative interactive systems. Papers at the boundaries of these disciplines are especially welcome, as it is our view that interdisciplinary approaches are needed for producing theoretical insights in this complex area and for effective deployment of innovative technologies in concrete user communities. Research areas relevant to the journal include, but are not limited to: • Innovative interaction techniques • Multimodal interaction • Speech interaction • Graphic interaction • Natural language interaction • Interaction in mobile and embedded systems • Interface design and evaluation methodologies • Design and evaluation of innovative interactive systems • User interface prototyping and management systems • Ubiquitous computing • Wearable computers • Pervasive computing • Affective computing • Empirical studies of user behaviour • Empirical studies of programming and software engineering • Computer supported cooperative work • Computer mediated communication • Virtual reality • Mixed and augmented Reality • Intelligent user interfaces • Presence ...