Boost Protein Language Model with Injected Structure Information Through Parameter Efficient Fine-tuning

Abstract

Large-scale Protein Language Models (PLMs), such as the Evolutionary Scale Modeling (ESM) family, have significantly advanced our understanding of protein structures and functions. These models have shown immense potential in biomedical applications, including drug discovery, protein design, and understanding disease mechanisms at the molecular level. However, PLMs are typically pre-trained on residue sequences alone, with limited incorporation of structural information, presenting opportunities for further enhancement. In this paper, we propose Structure Information Injecting Tuning (SI-Tuning), a parameter-efficient fine-tuning method, to integrate structural information into PLMs. SI-Tuning maintains the original model parameters in a frozen state while optimizing task-specific vectors for input embedding and attention maps. Structural features, including dihedral angles and distance maps, are used to derive this vector, injecting the structural information that improves model performance in downstream tasks. Extensive experiments on 650M ESM-2 demonstrate the effectiveness of our SI-Tuning across multiple downstream tasks. Specifically, our SI-Tuning achieves an accuracy of 93.95% on DeepLoc binary classification, and 76.05% on Metal Ion Binding, outperforming SaProt, a large-scale pre-trained PLM with structural modeling. SI-Tuning effectively enhances the performance of PLMs by incorporating structural information in a parameter-efficient manner. Our method not only advances downstream task performance, but also offers significant computational efficiency, making it a valuable strategy for applying large-scale PLM to broad biomedical downstream applications. Code is available at https://github.com/Nocturne0256/SI-tuning.