Pretraining graph transformer for molecular representation with fusion of multimodal information

Abstract

Molecular representation learning (MRL) is essential in certain applications including drug discovery and life science. Despite advancements in multiview and multimodal learning in MRL, existing models have explored only a limited range of perspectives, and the fusion of different views and modalities in MRL remains underexplored. Besides, obtaining the geometric conformer of molecules is not feasible in many tasks due to the high computational cost. Designing a general-purpose pertaining model for MRL is worthwhile yet challenging. This paper proposes a novel graph Transformer pretraining framework with fusion of node and graph views, along with the 2D topology and 3D geometry modalities of molecules, called MolGT. This MolGT model integrates node-level and graph-level pretext tasks on 2D topology and 3D geometry, leveraging a customized modality-shared graph Transformer that has versatility regarding parameter efficiency and knowledge sharing across modalities. Moreover, MolGT can produce implicit 3D geometry by leveraging contrastive learning between 2D topological and 3D geometric modalities. We provide extensive experiments and in-depth analyses, verifying that MolGT can (1) indeed leverage multiview and multimodal information to represent molecules accurately, and (2) infer nearly identical results using 2D molecules without requiring the expensive computation of generating conformers. Code is available on GitHub11https://github.com/robbenplus/MolGT..