Enhancing Video Experiences for DHH Individuals through Sound-Inspired Motion Caption-based Spatiotemporal Tacton.

IF 2.4 3区计算机科学 Q2 COMPUTER SCIENCE, CYBERNETICS

IEEE Transactions on Haptics Pub Date : 2025-04-01 DOI:10.1109/TOH.2025.3556667

Sooyeon Ahn, Gyungmin Jin, Gunhyuk Park, Jin-Hyuk Hong

引用次数: 0

Abstract

When deaf and hard of hearing (DHH) individuals watch videos, captions are essential for them to understand the linguistic content. Current captions, however, are not suitable for conveying non-verbal sound information, such as background music, sound effects, or speech nuances. In this paper, we designed a multimodal system, Motion Caption Haptic System (MCHS), that enables DHH individuals to encounter sounds in videos through animated caption and spatiotemporal vibration patterns, supporting a more vivid and immersive experience. We elaborately designed motion captions and spatiotemporal haptic patterns for representative sound effects and spoken emotions to work well together through surveys from 27 DHH and 64 hearing participants. An evaluation with 19 DHH individuals demonstrated the capabilities and potential of the MCHS to improve their video viewing experience, along with a discussion of important issues that need to be addressed when designing multimodal captioning systems for the DHH viewers.

查看原文本刊更多论文

求助全文

约1分钟内获得全文求助全文

来源期刊

IEEE Transactions on Haptics COMPUTER SCIENCE, CYBERNETICS-

CiteScore

5.90

自引率

13.80%

发文量

109

审稿时长

>12 weeks

期刊介绍： IEEE Transactions on Haptics (ToH) is a scholarly archival journal that addresses the science, technology, and applications associated with information acquisition and object manipulation through touch. Haptic interactions relevant to this journal include all aspects of manual exploration and manipulation of objects by humans, machines and interactions between the two, performed in real, virtual, teleoperated or networked environments. Research areas of relevance to this publication include, but are not limited to, the following topics: Human haptic and multi-sensory perception and action, Aspects of motor control that explicitly pertain to human haptics, Haptic interactions via passive or active tools and machines, Devices that sense, enable, or create haptic interactions locally or at a distance, Haptic rendering and its association with graphic and auditory rendering in virtual reality, Algorithms, controls, and dynamics of haptic devices, users, and interactions between the two, Human-machine performance and safety with haptic feedback, Haptics in the context of human-computer interactions, Systems and networks using haptic devices and interactions, including multi-modal feedback, Application of the above, for example in areas such as education, rehabilitation, medicine, computer-aided design, skills training, computer games, driver controls, simulation, and visualization.