Spatial audio signal processing for augmented telepresence applications

Abstract

During the COVID-19 pandemic, the shift to remote communication, particularly through video calls, led to both opportunities and challenges. While initially a welcome alternative to in-person meetings, virtual gatherings became increasingly overwhelming, culminating in the term “zoom fatigue.” However, reduced travel highlighted the potential environmental benefits of online meetings. My PhD research focuses on improving the naturalness of remote communication to enhance the appeal of virtual meetings. Specifically, I develop signal processing techniques that preserve spatial and acoustic cues important for natural speech perception, such as the cocktail party effect. By modeling microphone array signals, particularly those integrated into smart glasses or augmented reality headsets, I estimate and apply spatial room transfer functions to create natural binaural audio experiences. My work also addresses challenges posed by head movement, using continuous-space domain estimation to update room transfer functions during head rotations. First results show the effectiveness of the method under controlled conditions. Future work will investigate the approach in more realistic scenarios.