Multimodal deep learning model for predicting microsatellite instability in colorectal cancer by contrast-enhanced computed tomography and histopathology

Objectives

To develop and validate a multimodal deep learning (DL) model that integrates preoperative contrast-enhanced computed tomography (CECT) and postoperative whole-slide images (WSIs) to predict microsatellite instability (MSI) status in colorectal cancer (CRC).

Materials and methods

This retrospective, multicenter study enrolled 305 CRC patients with paired CECT and WSIs. Patients from Center I and II were allocated to the training (n = 169) and internal validation (n = 85) sets, while those from Center III formed the external test set (n = 51). Pathology-based DL (PathDL) and venous-phase CECT (VPDL) models were constructed using EfficientNet-b0 and ResNet 101 architectures, respectively. A fusion model (F-VP-PathDL, Fusion of venous phase CT and pathology with deep learning) was developed using an adaptive residual network to integrate features from both modalities. Model performance was evaluated using area under the receiver operating characteristic curve (AUC), accuracy, sensitivity, specificity, and F1 score.

Results

The F-VP-PathDL model achieved strong performance on the internal validation set, with an AUC of 0.883 (95 % CI: 0.732–0.967). On the external test set, the model achieved an AUC of 0.905 (95 % CI: 0.831–0.945), outperforming single-modality and alternative fusion models (PathDL: 0.794; VPDL: 0.858; APDL: 0.802; F-AVPDL: 0.813). The model also demonstrated robust accuracy (84.2 %, 95 % CI: 69.1 %–92.8 %), sensitivity (80.3 %, 95 % CI: 28.4 %–98.7 %), specificity (83.7 %, 95% CI: 68.8 %–93.9 %) and F1 score (0.837, 95 % CI: 0.326–0.999) on the external test set.

Conclusions

The F-VP-PathDL model demonstrates robust generalizability across centers and offers a clinically scalable tool for MSI prediction in CRC, supporting patient stratification and informing immunotherapy decisions.