An Explorative Study on Crossmodal Congruence Between Visual and Tactile Icons Based on Emotional Responses

Tactile icons, brief tactile stimuli conveying abstract information, have found their use in various applications, and their use with visual elements is increasing on touchscreen user interfaces. However, effective design guidelines of tactile icons for crossmodal use have not been established. This paper addresses this problem by investigating the congruence between visual and tactile icons based on the hypothesis that emotional agreement between the icons improves congruence. The validity of this hypothesis was examined in three experiments. In Exp. I, we selected common visual icons and estimated their emotional responses using the circumplex model of affect. Tactile icons to be used as a pair were designed in Exp. II by varying their amplitude, frequency, and envelope (rhythm). Their emotional responses were also evaluated. In Exp. III, the congruence of 192 crossmodal icons made by combining the visual icons (8) and the tactile icons (24) was evaluated, and these congruence scores were compared with the valence and arousal scores of the two unimodal icons obtained in Exp. I and II. Experimental results suggested that the congruence of a crossmodal icon highly depends on the agreement in the emotional responses between its visual and tactile icons. This finding provides feasibility to the development of general design guidelines and heuristics for crossmodal icons that rely on the relationship between the emotional responses from the individual modalities. Our approach is expected to advance the current practice that associates the physical parameters between the different senses with better intuitiveness and simplicity.