Histo-Genomic Knowledge Association for Cancer Prognosis From Histopathology Whole Slide Images

Abstract

Histo-genomic multi-modal methods have emerged as a powerful paradigm, demonstrating significant potential for cancer prognosis. However, genome sequencing, unlike histopathology imaging, is still not widely accessible in underdeveloped regions, limiting the application of these multi-modal approaches in clinical settings. To address this, we propose a novel Genome-informed Hyper-Attention Network, termed G-HANet, which is capable of effectively learning the histo-genomic associations during training to elevate uni-modal whole slide image (WSI)-based inference for the first time. Compared with the potential knowledge distillation strategy for this setting (i.e., distilling a multi-modal network to a uni-modal network), our end-to-end model is superior in training efficiency and learning cross-modal interactions. Specifically, the network comprises cross-modal associating branch (CAB) and hyper-attention survival branch (HSB). Through the genomic data reconstruction from WSIs, CAB effectively distills the associations between functional genotypes and morphological phenotypes and offers insights into the gene expression profiles in the feature space. Subsequently, HSB leverages the distilled histo-genomic associations as well as the generated morphology-based weights to achieve the hyper-attention modeling of the patients from both histopathology and genomic perspectives to improve cancer prognosis. Extensive experiments are conducted on five TCGA benchmarking datasets and the results demonstrate that G-HANet significantly outperforms the state-of-the-art WSI-based methods and achieves competitive performance with genome-based and multi-modal methods. G-HANet is expected to be explored as a useful tool by the research community to address the current bottleneck of insufficient histo-genomic data pairing in the context of cancer prognosis and precision oncology. The code is available at https://github.com/ZacharyWang-007/G-HANet.