Electroencephalography-driven three-dimensional object decoding with multi-view perception diffusion

With the rapid development of artificial intelligence and neuroscience, existing research has shown that two-dimensional (2D) visual stimuli observed by subjects can be reconstructed from Electroencephalography (EEG) signals. However, few studies have attempted to decode three-dimensional (3D) visual objects from EEG signals evoked by 2D visual stimuli. Neuroscience research has confirmed that the brain can perceive 3D forms and cues from 2D visual stimuli, and this perceptual activity is reflected in EEG signals. To address this gap, this paper explores high-fidelity 3D object decoding from EEG signals by leveraging a multi-view perception diffusion model combined with Neural Radiance Field (NeRF) representations. The proposed EEG-to-3D method employs a two-stage learning process. The first stage captures latent EEG codes that imply 3D perceptual representations, by presenting a multi-task optimization strategy that combines EEG signal reconstruction with semantic classification tasks. In the second stage, a multi-view perception diffusion model, conditioned on latent EEG codes, is fine-tuned to constrain and optimize the parameters of the NeRF model, thereby generating multi-views of 3D objects with both semantic and viewpoint consistency. Experimental results demonstrate the effectiveness and superiority of the proposed method for 3D object decoding from EEG signals. The source codes are publicly available at: https://github.com/xiangxinhello/EEG_to_3D.