Memory recall: Retrieval-Augmented mind reconstruction for brain decoding

Abstract

Reconstructing visual stimuli from functional magnetic resonance imaging (fMRI) is a complex challenge in neuroscience. Most existing approaches rely on mapping neural signals to pretrained models to generate latent variables, which are then used to reconstruct images via a diffusion model. However, this multi-step process can result in the loss of crucial semantic details, limiting reconstruction accuracy. In this paper, we introduce a novel brain decoding framework, called Memory Recall (MR), inspired by bionic brain mechanisms. MR mimics the human visual perception process, where the brain retrieves stored visual experiences to compensate for incomplete visual cues. Initially, low- and high-level visual cues are extracted using spatial mapping techniques based on VAE and CLIP, replicating the brain’s ability to interpret degraded stimuli. A visual experience database is then created to retrieve complementary information that enriches these high-level representations, simulating the brain’s memory retrieval process. Finally, an Attentive Visual Signal Fusion Network (AVSFN) with a novel attention scoring mechanism integrates the retrieved information, enhancing the generative model’s performance and emulating the brain’s refinement of visual perception. Experimental results show that MR outperforms state-of-the-art models across multiple evaluation metrics and subjective assessments. Moreover, our results provide new evidence supporting a well-known psychological conclusion that the basic information capacity of short-term memory is four items, further demonstrating the informativeness and interpretability of our model.