Predicting synergistic drug combinations via hierarchical molecular representation and cell line latent space fusion

Cancer treatment often benefits from the synergistic effects of drug combinations. Predicting these synergies is critical for developing effective combination therapies. Existing deep learning models typically represent drugs using a single graph structure and use cell line gene expression profiles directly, potentially leading to loss of detailed molecular features and introduction of noise. This study aims to improve the prediction of drug synergy by developing a novel deep learning model that captures both local and global features of drug molecules at multiple levels and reduces noise in cell line data. The proposed model hierarchically represents drug molecules at the node, motif, and graph levels to capture comprehensive feature information. The Mamba module and graph attention-based convolution are employed to effectively extract deep feature information from drug pairs. An encoder–decoder structure projects cell lines into a latent space, minimizing noise and enhancing the integration with drug pair data through star operations and an attention mechanism. The model was trained and validated on benchmark datasets containing drug response data from various cancer cell lines. The evaluation of the model against benchmark datasets demonstrated superior performance compared to existing methods. These results indicate that the model can more accurately predict synergistic anticancer drug combinations, providing reliable support for the design of combination therapies. The enhanced predictive accuracy can facilitate the discovery of effective drug combinations, potentially accelerating the development of personalized cancer treatments.