A unified gradient regularization method for heterogeneous graph neural networks

Abstract

Heterogeneous Graph Neural Networks (HGNNs) are advanced deep learning methods widely applied for learning representations of heterogeneous graphs. However, they face challenges such as over-smoothing and non-robustness. Existing methods can mitigate these issues by applying gradient regularization to one of the three information dimensions: node, edge, or propagation message. However, these methods have problems such as unstable training, difficulty in parameter convergence, and inadequate utilization of heterogeneous information. We propose a novel gradient regularization method called Grug, which iteratively applies regularization to the gradients derived from both node type and message matrix during the message-passing process. A detailed theoretical analysis demonstrates its advantages in Stability and Diversity. Notably, Grug potentially exceeds the theoretical upper bounds set by DropMessage. In addition, Grug offers a unified gradient regularization framework that integrates the existing dropping and adversarial training methods, and provides theoretical guidance for their further optimization in different data and tasks. We validate Grug through extensive experiments on six public datasets, showing significant improvements in performance and effectiveness.