Enhancing Video Experiences for DHH Individuals through Sound-Inspired Motion Caption-based Spatiotemporal Tacton.

IF 2.4 3区计算机科学 Q2 COMPUTER SCIENCE, CYBERNETICS

IEEE Transactions on Haptics Pub Date : 2025-04-01 DOI:10.1109/TOH.2025.3556667

Sooyeon Ahn, Gyungmin Jin, Gunhyuk Park, Jin-Hyuk Hong

引用次数: 0

Abstract

When deaf and hard of hearing (DHH) individuals watch videos, captions are essential for them to understand the linguistic content. Current captions, however, are not suitable for conveying non-verbal sound information, such as background music, sound effects, or speech nuances. In this paper, we designed a multimodal system, Motion Caption Haptic System (MCHS), that enables DHH individuals to encounter sounds in videos through animated caption and spatiotemporal vibration patterns, supporting a more vivid and immersive experience. We elaborately designed motion captions and spatiotemporal haptic patterns for representative sound effects and spoken emotions to work well together through surveys from 27 DHH and 64 hearing participants. An evaluation with 19 DHH individuals demonstrated the capabilities and potential of the MCHS to improve their video viewing experience, along with a discussion of important issues that need to be addressed when designing multimodal captioning systems for the DHH viewers.

查看原文本刊更多论文

通过基于声音启发的动作字幕的时空战术增强DHH个人的视频体验。

当聋人和重听人观看视频时，字幕对他们理解语言内容至关重要。然而，当前的字幕不适合传达非语言的声音信息，如背景音乐、声音效果或语音的细微差别。在本文中，我们设计了一个多模态系统，运动字幕触觉系统（MCHS），使DHH个体能够通过动画字幕和时空振动模式在视频中遇到声音，支持更生动和身临其境的体验。通过对27名DHH和64名听力参与者的调查，我们精心设计了具有代表性的声音效果和言语情绪的动作字幕和时空触觉模式，使其能够很好地协同工作。对19名DHH个人的评估显示了MCHS改善其视频观看体验的能力和潜力，同时讨论了为DHH观众设计多模式字幕系统时需要解决的重要问题。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

IEEE Transactions on Haptics COMPUTER SCIENCE, CYBERNETICS-

CiteScore

5.90

自引率

13.80%

发文量

109

审稿时长

>12 weeks

期刊介绍： IEEE Transactions on Haptics (ToH) is a scholarly archival journal that addresses the science, technology, and applications associated with information acquisition and object manipulation through touch. Haptic interactions relevant to this journal include all aspects of manual exploration and manipulation of objects by humans, machines and interactions between the two, performed in real, virtual, teleoperated or networked environments. Research areas of relevance to this publication include, but are not limited to, the following topics: Human haptic and multi-sensory perception and action, Aspects of motor control that explicitly pertain to human haptics, Haptic interactions via passive or active tools and machines, Devices that sense, enable, or create haptic interactions locally or at a distance, Haptic rendering and its association with graphic and auditory rendering in virtual reality, Algorithms, controls, and dynamics of haptic devices, users, and interactions between the two, Human-machine performance and safety with haptic feedback, Haptics in the context of human-computer interactions, Systems and networks using haptic devices and interactions, including multi-modal feedback, Application of the above, for example in areas such as education, rehabilitation, medicine, computer-aided design, skills training, computer games, driver controls, simulation, and visualization.