MetaGT: A lightweight graph transformer via meta learning for large-scale graphs

Abstract

Graph Transformers (GTs) excel at modeling long-distance dependencies and global information in graphs, but their scalability on large-scale graphs is limited by the quadratic growth of memory and computation costs introduced by self-attention. To address these challenges, existing methods such as linear attention mechanisms and graph partitioning have been proposed. However, these methods have inherent limitations. Linear attention mechanisms still incur significant computational overhead, while subgraph-based training often leads to reduced accuracy due to the lack of global context and the structural differences among subgraphs. In this paper, we propose MetaGT, a lightweight Graph Transformer model for large-scale graphs. MetaGT first partitions the global graph into structurally compact subgraphs to reduce memory requirements. To further ensure both efficiency and accuracy, we introduce a “pre- and post-training” paradigm into subgraph-based GT models. In pre-training, meta-learning is employed on a subset of subgraphs to obtain generalizable parameters, which are then enhanced by minimal fine-tuning during post-training. MetaGT is plug-and-play and can be seamlessly integrated into mainstream GT models. Extensive experiments demonstrate that MetaGT consistently achieves promising efficiency, memory consumption, and accuracy across various backbone GT models on large-scale graph datasets. Our code is available at https://anonymous.4open.science/r/MetaGT-D2B0.