Towards unified molecule-enhanced pathology image representation learning via integrating spatial transcriptomics

Recent advancements in multimodal pre-training have advanced computational pathology, but current visual-language approaches lack molecular perspective and face performance bottlenecks in clinical settings. Here, we introduce a Unified Molecule-enhanced Pathology Image REpresentation Learning framework ($\text{UMPIRE}$) that enhances the robustness and generalization capabilities of pathology image analysis across diverse tissue types and sequencing platforms. $\text{UMPIRE}$ leverages complementary information from gene expression profiles to guide multimodal pre-training, addressing the challenge of distribution shifts between research and clinical environments. To overcome the scarcity of paired data, we collected more than 4 million entries of spatial transcriptomics gene expression to train the gene encoder. $\text{UMPIRE}$ aligns modalities across 697K pathology image-gene expression pairs, creating a foundation model that demonstrates superior generalization across multiple sequencing platforms and downstream tasks without additional fine-tuning. Comprehensive evaluation shows $\text{UMPIRE}$’s effectiveness in gene expression prediction, spot classification, and mutation state prediction in whole slide images, with significant improvements over state-of-the-art methods. Our findings demonstrate how molecular data integration enhances visual pattern recognition in computational pathology, providing a resilient approach for bench-to-bedside translation. The code and pre-trained weights are available at https://github.com/Hanminghao/Umpire.