Leveraging Transformer Models to Capture Multi-Scale Dynamics in Biomolecules by Nano-GPT

Abstract

Long-term biomolecular dynamics is critical for understanding key evolutionary transformations in molecular systems. However, capturing these processes requires extended simulation timescales that often exceed the practical limits of conventional models. To address this, shorter simulations, initialized with diverse perturbations, are commonly used to sample the phase space and explore a wide range of behaviors. Recent advances have leveraged language models to infer long-term behavior from short trajectories, but methods such as long short-term memory (LSTM) networks are constrained to low-dimensional reaction coordinates, limiting their applicability to complex systems. In this work, we present nano-GPT, a novel deep learning model inspired by the GPT architecture specifically designed to capture long-term dynamics in molecular systems with fine-grained conformational states and complex transitions. The model employs a two-pass training mechanism that incrementally replaces molecular dynamics (MD) tokens with model-generated predictions, effectively mitigating the accumulation errors inherent in the training window. We validate nano-GPT on three distinct systems: a four-state model potential, the alanine dipeptide, a well-studied simple molecule, and the Fip35 WW domain, a complex biomolecular system. Our results show that nano-GPT effectively captures long-time scale dynamics by learning high-order dependencies through an attention mechanism, offering a novel perspective for interpreting biomolecular processes.