Imagenation - A DCGAN based method for Image Reconstruction from fMRI

Abstract

We propose a method to reconstruct natural grayscale images and handwritten characters from Functional Magnetic Resonance Imaging (fMRI) data and achieve a high degree of similarity to the original stimuli images. The approach utilizes a pre-trained Deep Convolutional Generative Adversarial Network (DCGAN) to reconstruct images and provide visual confirmations regarding the resemblance between the reconstructed and original images. A linear regressor is used to elicit information from the fMRI data and estimate a latent space representation for the formerly trained generative model. A composite loss function combining the Perceptual and Multi-Scale Structural Similarity Index (MS-SSIM) losses is used to train the regressor. The advantages of both functions are evident with the Perceptual loss capturing semantic information and the MS-SSIM loss carrying information about objects in a scene. With this loss function, we were able to reconstruct human objects in the stimuli to a degree of accuracy. The reconstructions obtained were then validated using the Scale Invariant Feature Transform (SIFT) method to elucidate the number of features matched between the original and recreated images. The SSIM scores for the reconstructed images are observed to be higher than state-of-the-art methods. Parallels are drawn between the distortions produced in images submerged underwater and those in the reconstructed images using the Contrast Limited Adaptive Histogram Equalization (CLAHE), an image enhancement technique. A sharp increase in the number of SIFT features matched, is observed with the application of CLAHE on the reconstructed images.