Rethinking transformers with convolution and graph embeddings for few-shot molecular property discovery

Abstract

The prediction of molecular properties is a critical step in drug discovery campaigns. Computational methods such as graph neural networks (GNNs) and Transformers have effectively leveraged the small-range and long-range dependencies in molecules to preserve the local and global patterns for multiple molecular property prediction tasks. However, the dependence of these models on large amounts of experimental data poses a challenge, particularly on smaller biological datasets prevalent across the drug discovery pipeline. This paper introduces FS-GCvTR, a few-shot graph-based convolutional Transformer architecture designed to predict chemical properties with a small amount of labeled compounds. The convolutional Transformer is presented as a crucial component, effectively integrating both local and global dependencies of molecular graph embeddings by propagating a set of convolutional tokens across Transformer attention layers for molecular property prediction. Furthermore, a few-shot meta-learning approach is introduced to iteratively adapt model parameters across multiple few-shot tasks while generalizing to new chemical properties with limited available data. Experiments including few-shot evaluations on multi-property datasets show that the FS-GCvTR model outperformed other few-shot graph-based baselines in specific molecular property prediction tasks.