Balancing semantic and structural decoding for fMRI-to-image reconstruction

Abstract

Reconstructing visual images from fMRI signals is an enticing task that opens new horizons in understanding the intricate workings of human cognition. Most existing methods benefit from the diffusion model to decode high-level semantic information from fMRI signals, achieving promising semantic reconstruction. However, such a solution ignores low-level structure information, e.g., object location and color, leading to an uncompleted visual reconstruction. In this work, we present a novel fMRI-to-image approach to reconstruct high-quality images by balancing semantic and structural decoding in the diffusion model. Specifically, we first utilize the CLIP model and an MLP module to extract sufficient semantic information and structural details, respectively. Then we design a Semantic and Structural Awareness Balanced module (SSAB) to predict the weight of structural information for the current denoising step, thus generating high-quality images by gradually integrating semantic and structural information during image reconstruction. Experimental results demonstrate that the proposed SSAB model is effective with only a few extra parameters, it achieves state-of-the-art performance in comprehensively evaluating both semantic and structural metrics. All code is available on https://github.com/Venchy-he/SSAB.