Multimodal transformers predict cancer therapy response from tumor mechanics.

Abstract

Precise prediction of cancer therapy response remains challenging because conventional biomarkers capture molecular features but overlook the physical state of tumors. We developed a multimodal deep-learning framework integrating ultrasound shear wave elastography (SWE) images with quantitative stiffness measurements (elastic modulus, kPa) to predict treatment outcomes in preclinical murine tumors. Each image-stiffness pair is tokenized within a transformer that learns interactions between local elastographic texture and global rigidity. A lightweight convolutional encoder extracts image features, the modulus is embedded as a numeric token, and self-attention fuses both modalities for classification. Trained on 1578 baseline SWE images from five syngeneic tumor models, the model classified tumors as responders, stable, or non-responders. Across five random seeds, it achieved 92.4% ± 1.3% accuracy, macro-F1 0.92, and ROC-AUC 0.99 on a held-out test set, with well-calibrated probabilities. Matched-split ablations-image-only, stiffness-token-removed, stiffness-shuffled, late-fusion, and stiffness-only-showed that performance reflected genuine cross-modal learning rather than scalar stiffness alone; shuffling image-stiffness pairings significantly reduced accuracy (all corrected p < 0.01). Leave-one-tumor-model-out analysis demonstrated generalization to unseen tumor types, with 95.5% ± 1.5% accuracy (range 93.9-97.9%) across five held-out models. Lower baseline stiffness correlated with better response, supporting the hypothesis that mechanically normalized tumors respond more effectively. These preclinical proof-of-concept findings establish tumor mechanics as candidate predictive biomarkers and transformer-based multimodal learning as a scalable approach to biomechanically informed response prediction, while requiring validation in human cohorts.