Maotao Liu , Qun Liu , Xu Gong , Yunsong Luo , Guoyin Wang
{"title":"Mol-L2: Transferring text knowledge with frozen language models for molecular representation learning","authors":"Maotao Liu , Qun Liu , Xu Gong , Yunsong Luo , Guoyin Wang","doi":"10.1016/j.neucom.2025.130837","DOIUrl":null,"url":null,"abstract":"<div><div>How to integrate abundant chemical text descriptions to produce expressive molecular representations is a compelling challenge. In this study, we propose a deep network architecture called Mol-L2, which aims to leverage powerful language models to transfer chemical text knowledge and enhance molecular representation learning. The main novelty of this work is the use of a two-stage training pipeline to align text and chemical spaces, where stage 1 pre-trains the language model using a specially constructed multi-objective loss, and stage 2 fine-tunes on molecular captioning. Subsequently, the output of the language model encoder is converted into a fixed-length text-enhanced embedding via a lightweight mapping network. Furthermore, a dedicated encoder containing information propagation of specific functional groups is designed to generate molecular initial representation and integrated with the text-enhanced embeddings using a weighted fusion module. Finally, the enhanced molecular representation is utilized for various downstream tasks through an additional output layer. The performance of the proposed Mol-L2 is tested on several standard benchmarks for molecular machine learning, including molecular properties prediction, drug-target interaction (DTI), and drug-drug interaction (DDI). Through comprehensive experiments, we demonstrate the merits and state-of-the-art performance of the Mol-L2 framework. Take blood–brain barrier penetration prediction, for instance, where Mol-L2 achieves the smallest prediction error, while the best comparison method is 91.8%, an improvement of 3.1%.</div></div>","PeriodicalId":19268,"journal":{"name":"Neurocomputing","volume":"651 ","pages":"Article 130837"},"PeriodicalIF":6.5000,"publicationDate":"2025-07-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Neurocomputing","FirstCategoryId":"94","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0925231225015097","RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE","Score":null,"Total":0}
引用次数: 0
Abstract
How to integrate abundant chemical text descriptions to produce expressive molecular representations is a compelling challenge. In this study, we propose a deep network architecture called Mol-L2, which aims to leverage powerful language models to transfer chemical text knowledge and enhance molecular representation learning. The main novelty of this work is the use of a two-stage training pipeline to align text and chemical spaces, where stage 1 pre-trains the language model using a specially constructed multi-objective loss, and stage 2 fine-tunes on molecular captioning. Subsequently, the output of the language model encoder is converted into a fixed-length text-enhanced embedding via a lightweight mapping network. Furthermore, a dedicated encoder containing information propagation of specific functional groups is designed to generate molecular initial representation and integrated with the text-enhanced embeddings using a weighted fusion module. Finally, the enhanced molecular representation is utilized for various downstream tasks through an additional output layer. The performance of the proposed Mol-L2 is tested on several standard benchmarks for molecular machine learning, including molecular properties prediction, drug-target interaction (DTI), and drug-drug interaction (DDI). Through comprehensive experiments, we demonstrate the merits and state-of-the-art performance of the Mol-L2 framework. Take blood–brain barrier penetration prediction, for instance, where Mol-L2 achieves the smallest prediction error, while the best comparison method is 91.8%, an improvement of 3.1%.
期刊介绍:
Neurocomputing publishes articles describing recent fundamental contributions in the field of neurocomputing. Neurocomputing theory, practice and applications are the essential topics being covered.