Rethinking multi-level information fusion in temporal graphs: Pre-training then distilling for better embedding

Abstract

Temporal graphs occupy an important place in graph data, which store node interactions in sequences, thus enabling a more microscopic view of each node’s dynamics. However, many temporal graph methods primarily concentrate on shallow-level temporal or neighborhood information, while acquiring deep-level community or global graph information necessitates increased computational costs, thereby significantly impacting model efficiency. Inspired by this, we rethink how this information is acquired: if it is difficult to acquire it during model training, why not obtain it before training? Consequently, we propose ReMIT, a novel method for temporal graph learning, which incorporates the concepts of feature pre-training and knowledge distillation to Rethink the embedding of Multi-level Information fusion in Temporal graphs. ReMIT facilitates the “remitting” of prior knowledge to model, wherein hard-to-access information is captured and distilled to the train module by introducing a pre-train module. Experimental results on multiple real-world datasets validate the validity and feasibility of our proposed framework. Our method improves performance by up to 10.2% while reducing almost 30% training time.