Do Transformers and CNNs Learn Different Concepts of Brain Age?

“Predicted brain age” refers to a biomarker of structural brain health derived from machine learning analysis of T1-weighted brain magnetic resonance (MR) images. A range of machine learning methods have been used to predict brain age, with convolutional neural networks (CNNs) currently yielding state-of-the-art accuracies. Recent advances in deep learning have introduced transformers, which are conceptually distinct from CNNs, and appear to set new benchmarks in various domains of computer vision. Given that transformers are not yet established in brain age prediction, we present three key contributions to this field: First, we examine whether transformers outperform CNNs in predicting brain age. Second, we identify that different deep learning model architectures potentially capture different (sub-)sets of brain aging effects, reflecting divergent “concepts of brain age”. Third, we analyze whether such differences manifest in practice. To investigate these questions, we adapted a Simple Vision Transformer (sViT) and a shifted window transformer (SwinT) to predict brain age, and compared both models with a ResNet50 on 46,381 T1-weighted structural MR images from the UK Biobank. We found that SwinT and ResNet performed on par, though SwinT is likely to surpass ResNet in prediction accuracy with additional training data. Furthermore, to assess whether sViT, SwinT, and ResNet capture different concepts of brain age, we systematically analyzed variations in their predictions and clinical utility for indicating deviations in neurological and psychiatric disorders. Reassuringly, we observed no substantial differences in the structure of brain age predictions across the model architectures. Our findings suggest that the choice of deep learning model architecture does not appear to have a confounding effect on brain age studies.