Using Pupillometry in Virtual Reality as a Tool for Speech-in-Noise Research.

Abstract

Objectives: Virtual reality (VR) could be used in speech perception research to reduce the gap between the laboratory and real life. However, the suitability of using VR head-mounted displays (HMDs) warrants investigation, especially when pupillometric measurements are required. The present study aimed to assess if pupil measurements taken within an HMD would be sensitive to changes in listening effort related to a speech perception task. Task load of a VR speech-in-noise task was manipulated while pupil size was recorded within an HMD. The present study also assessed if VR could be used to simulate the copresence of other persons during listening, which is often an important aspect of real-life listening. To this end, participants completed the speech-in-noise task both in the copresence of virtual persons (agents) and while the virtual persons were replaced with visual distractors.

Design: Thirty-three normal-hearing participants were provided with a VR-HMD and completed a speech-in-noise task in a virtual environment while their pupil size was measured. Participants were simultaneously presented with two sentences-one to each ear-which were masked by stationary noise that was 3 dB louder (-3 dB signal to noise ratio) than the sentences. Task load was manipulated by having participants attend to and repeat either one sentence or both sentences. Participants did the task both while accompanied by two virtual agents who provided positive (head nodding) and negative (head shaking) feedback on some trials, or in the presence of two visual distractors that did not provide feedback (control condition). We assessed the effect of task load and copresence on performance, measures of pupil size (baseline pupil size and peak pupil dilation), and several subjective ratings. Participants also completed two questionnaires related to their experience of the virtual environment.

Results: Task load significantly affected baseline pupil size, peak pupil dilation, and subjective ratings of effort, task difficulty, and performance. However, the manipulation of virtual copresence did not affect any of the outcome measures. The effect of task load on performance could not be assessed, as single-sentence conditions often resulted in a ceiling score (100% correct). An exploratory analysis provided some indication that trials following positive feedback from the agents (as compared to no feedback) showed increased baseline pupil sizes. Scores on the questionnaires indicated that participants were not highly immersed in the virtual environment, possibly explaining why they were largely unaffected by the virtual copresence manipulation.

Conclusions: The finding that baseline pupil size and peak pupil dilation were sensitive to the manipulation of task load suggests that HMD pupillometry is sensitive to changes in arousal and effort. This supports the idea that VR-HMDs can be successfully combined with speech perception research using pupillometry. The lack of an effect of the virtual copresence manipulation on the physiological and subjective measures suggests that more advanced simulations may be required in a VR setting to study the effects of copresence. Weak evidence was found that positive feedback to participants was associated with increased baseline pupil size on subsequent trials; future studies should further examine the impact of feedback on listening.