Transfer learning from 2D natural images to 4D fMRI brain images via geometric mapping

Abstract

Functional magnetic resonance imaging (fMRI) allows real-time observation of brain activity through blood oxygen level-dependent (BOLD) signals and is extensively used in studies related to sex classification, age estimation, behavioral measurements prediction, and mental disorder diagnosis. However, the application of deep learning techniques to brain fMRI analysis is hindered by the small sample size of fMRI datasets. Transfer learning offers a solution to this problem, but most existing approaches are designed for large-scale 2D natural images. The heterogeneity between 4D fMRI data and 2D natural images makes direct model transfer infeasible. This study proposes a novel geometric mapping-based fMRI transfer learning method that enables transfer learning from 2D natural images to 4D fMRI brain images, bridging the transfer learning gap between fMRI data and natural images. The proposed Multi-scale Multi-domain Feature Aggregation (MMFA) module extracts effective aggregated features and reduces the dimensionality of fMRI data to 3D space. By treating the cerebral cortex as a folded Riemannian manifold in 3D space and mapping it into 2D space using surface geometric mapping, we make the transfer learning from 2D natural images to 4D brain images possible. Moreover, the topological relationships of the cerebral cortex are maintained with our method, and calculations are performed along the Riemannian manifold of the brain, effectively addressing signal interference problems. The experimental results based on the Human Connectome Project (HCP) dataset demonstrate the effectiveness of the proposed method. Our method achieved state-of-the-art performance in sex classification, age estimation, and behavioral measurement prediction tasks. Moreover, we propose a cascaded transfer learning approach for depression diagnosis, and proved its effectiveness on 23 depression datasets. In summary, the proposed fMRI transfer learning method, which accounts for the structural characteristics of the brain, is promising for applying transfer learning from natural images to brain fMRI images, significantly enhancing the performance in various fMRI analysis tasks.