Interpretable modality-specific and interactive graph convolutional network on brain functional and structural connectomes

Abstract

Both brain functional connectivity (FC) and structural connectivity (SC) provide distinct neural mechanisms for cognition and neurological disease. In addition, interactions between SC and FC within distributed association regions are related to alterations in cognition or neurological diseases, considering the inherent linkage between neural function and structure. However, there is a scarcity of existing learning-based methods that leverage both modality-specific characteristics and high-order interactions between the two modalities for regression or classification. Hence, this study proposes an interpretable modality-specific and interactive graph convolutional network (MS-Inter-GCN) that incorporates modality-specific information, reflecting the unique neural mechanism for each modality, and structure–function interactions, capturing the underlying foundation provided by white-matter fiber tracts for high-level brain function. In MS-Inter-GCN, we generate modality-specific task-relevant embeddings separately from both FC and SC using a graph convolutional encoder–decoder module. Subsequently, we learn the interactive weights between corresponding regions of FC and SC, reflecting the coupling strength, by employing an interactive module on the embeddings of both modalities. A novel graph structure is constructed, which uses modality-specific task-relevant embeddings and inserts the interactive weights as edges connecting corresponding regions of two modalities, and then is used for the regression or classification task. Finally, a post-hoc explainable technology - GNNExplainer- is used to identify salient regions and connections of each modality as well as salient interactions between FC and SC associated with tasks. We apply the proposed framework to fluid cognition prediction, Parkinson’s disease (PD), Alzheimer’s disease (AD), and schizophrenia (SZ) classification. Experimental results demonstrate that our method outperforms the other ten state-of-the-art methods on multi-modal brain features on all tasks. The GNNExplainer identifies salient structural and functional regions and connections for fluid cognition, PD, AD, and SZ. It confirms that strong structure–function coupling within the executive and control networks, combined with weak coupling within the motor network, is associated with fluid cognition. Moreover, structure–function decoupling in specific brain regions serves as a marker for different diseases: decoupling of the prefrontal, superior parietal, and superior occipital cortices is a marker of PD; decoupling of the middle frontal and lateral parietal cortices, temporal pole, and subcortical regions is indicative of AD; and decoupling of the prefrontal, parietal, and temporal cortices, as well as the cerebellum, contributes to SZ.