Mol-L2: Transferring text knowledge with frozen language models for molecular representation learning

How to integrate abundant chemical text descriptions to produce expressive molecular representations is a compelling challenge. In this study, we propose a deep network architecture called Mol-L2, which aims to leverage powerful language models to transfer chemical text knowledge and enhance molecular representation learning. The main novelty of this work is the use of a two-stage training pipeline to align text and chemical spaces, where stage 1 pre-trains the language model using a specially constructed multi-objective loss, and stage 2 fine-tunes on molecular captioning. Subsequently, the output of the language model encoder is converted into a fixed-length text-enhanced embedding via a lightweight mapping network. Furthermore, a dedicated encoder containing information propagation of specific functional groups is designed to generate molecular initial representation and integrated with the text-enhanced embeddings using a weighted fusion module. Finally, the enhanced molecular representation is utilized for various downstream tasks through an additional output layer. The performance of the proposed Mol-L2 is tested on several standard benchmarks for molecular machine learning, including molecular properties prediction, drug-target interaction (DTI), and drug-drug interaction (DDI). Through comprehensive experiments, we demonstrate the merits and state-of-the-art performance of the Mol-L2 framework. Take blood–brain barrier penetration prediction, for instance, where Mol-L2 achieves the smallest prediction error, while the best comparison method is 91.8%, an improvement of 3.1%.