Mol-L2: Transferring text knowledge with frozen language models for molecular representation learning

IF 6.5 2区 计算机科学 Q1 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE
Maotao Liu , Qun Liu , Xu Gong , Yunsong Luo , Guoyin Wang
{"title":"Mol-L2: Transferring text knowledge with frozen language models for molecular representation learning","authors":"Maotao Liu ,&nbsp;Qun Liu ,&nbsp;Xu Gong ,&nbsp;Yunsong Luo ,&nbsp;Guoyin Wang","doi":"10.1016/j.neucom.2025.130837","DOIUrl":null,"url":null,"abstract":"<div><div>How to integrate abundant chemical text descriptions to produce expressive molecular representations is a compelling challenge. In this study, we propose a deep network architecture called Mol-L2, which aims to leverage powerful language models to transfer chemical text knowledge and enhance molecular representation learning. The main novelty of this work is the use of a two-stage training pipeline to align text and chemical spaces, where stage 1 pre-trains the language model using a specially constructed multi-objective loss, and stage 2 fine-tunes on molecular captioning. Subsequently, the output of the language model encoder is converted into a fixed-length text-enhanced embedding via a lightweight mapping network. Furthermore, a dedicated encoder containing information propagation of specific functional groups is designed to generate molecular initial representation and integrated with the text-enhanced embeddings using a weighted fusion module. Finally, the enhanced molecular representation is utilized for various downstream tasks through an additional output layer. The performance of the proposed Mol-L2 is tested on several standard benchmarks for molecular machine learning, including molecular properties prediction, drug-target interaction (DTI), and drug-drug interaction (DDI). Through comprehensive experiments, we demonstrate the merits and state-of-the-art performance of the Mol-L2 framework. Take blood–brain barrier penetration prediction, for instance, where Mol-L2 achieves the smallest prediction error, while the best comparison method is 91.8%, an improvement of 3.1%.</div></div>","PeriodicalId":19268,"journal":{"name":"Neurocomputing","volume":"651 ","pages":"Article 130837"},"PeriodicalIF":6.5000,"publicationDate":"2025-07-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Neurocomputing","FirstCategoryId":"94","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0925231225015097","RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE","Score":null,"Total":0}
引用次数: 0

Abstract

How to integrate abundant chemical text descriptions to produce expressive molecular representations is a compelling challenge. In this study, we propose a deep network architecture called Mol-L2, which aims to leverage powerful language models to transfer chemical text knowledge and enhance molecular representation learning. The main novelty of this work is the use of a two-stage training pipeline to align text and chemical spaces, where stage 1 pre-trains the language model using a specially constructed multi-objective loss, and stage 2 fine-tunes on molecular captioning. Subsequently, the output of the language model encoder is converted into a fixed-length text-enhanced embedding via a lightweight mapping network. Furthermore, a dedicated encoder containing information propagation of specific functional groups is designed to generate molecular initial representation and integrated with the text-enhanced embeddings using a weighted fusion module. Finally, the enhanced molecular representation is utilized for various downstream tasks through an additional output layer. The performance of the proposed Mol-L2 is tested on several standard benchmarks for molecular machine learning, including molecular properties prediction, drug-target interaction (DTI), and drug-drug interaction (DDI). Through comprehensive experiments, we demonstrate the merits and state-of-the-art performance of the Mol-L2 framework. Take blood–brain barrier penetration prediction, for instance, where Mol-L2 achieves the smallest prediction error, while the best comparison method is 91.8%, an improvement of 3.1%.
Mol-L2:用冻结语言模型转移文本知识用于分子表示学习
如何整合丰富的化学文本描述以产生富有表现力的分子表征是一个引人注目的挑战。在本研究中,我们提出了一种称为Mol-L2的深度网络架构,旨在利用强大的语言模型来传递化学文本知识并增强分子表征学习。这项工作的主要新颖之处在于使用了一个两阶段的训练管道来对齐文本和化学空间,其中第一阶段使用特殊构建的多目标损失对语言模型进行预训练,第二阶段对分子字幕进行微调。随后,语言模型编码器的输出通过轻量级映射网络转换为固定长度的文本增强嵌入。此外,设计了包含特定官能团信息传播的专用编码器来生成分子初始表示,并使用加权融合模块与文本增强嵌入集成。最后,通过额外的输出层,增强的分子表示用于各种下游任务。提出的Mol-L2的性能在分子机器学习的几个标准基准上进行了测试,包括分子性质预测、药物-靶标相互作用(DTI)和药物-药物相互作用(DDI)。通过综合实验,我们展示了Mol-L2框架的优点和最先进的性能。以血脑屏障穿透预测为例,Mol-L2的预测误差最小,而最佳对比方法的预测误差为91.8%,提高了3.1%。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
Neurocomputing
Neurocomputing 工程技术-计算机:人工智能
CiteScore
13.10
自引率
10.00%
发文量
1382
审稿时长
70 days
期刊介绍: Neurocomputing publishes articles describing recent fundamental contributions in the field of neurocomputing. Neurocomputing theory, practice and applications are the essential topics being covered.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信