Using immersive virtual reality to recreate the synaesthetic experience.

Abstract

Synaesthesia is a condition where people experience unusual sensory or cognitive sensations in response to apparently unrelated stimuli. This paper presents two experiments which aimed to examine whether Virtual Reality (VR) technology can be used to recreate the synaesthetic experience. There is a lack of research in this area, with most studies focussing primarily on synaesthetic colors. Experiment 1 aimed to build on previous research by using not only a traditional color-picker but also VR to capture a more nuanced picture of synaesthetic perception. A multiple case study design was used to examine the experiences of six participants in detail. Data gathering took place via Zoom. During the initial data-gathering session, participants used a color-picker to provide grapheme-color associations. After this session, some of the participants' synaesthetic experiences were recreated using a VR-by-proxy approach. Results indicated that VR is capable of capturing elements of synaesthetic perception that other methods have been unable to, such as texture, small degrees of movement, and 3D structure. Experiment 2 expanded upon these findings by moving beyond the VR-by-proxy approach and asking three participants to recreate their own audiovisual synaesthetic associations in the VR environment. Inductive Thematic Analysis was used to analyze the results of this experiment. The potential of expanding this technique to other forms of perceptual diversity is discussed.