ICMIL: An Interactive Contrastive Multiple Instance Learning Framework Using Foundation Model Features for Histopathology Image Classification

Abstract

Multiple instance learning (MIL) is a crucial paradigm addressing weakly supervised classification in histopathological images. However, existing MIL methods struggle to model tile/patch interactions, which can capture important contextual information. Moreover, MIL is limited by suboptimal bag embeddings, as traditional methods focus primarily on extracting distinct embeddings for individual instances rather than for the entire bag. These limitations restrict MIL’s ability to leverage contextual information and generate discriminative aggregated representations. To address these limitations, we propose Interactive Contrastive Multiple Instance Learning (ICMIL), a novel MIL framework that integrates graph learning (GL) and contrastive learning (CL) to enhance MIL’s contextual awareness and discriminative power. ICMIL introduces two key innovations: 1) Transformer-based Graph Attention (TransGAT) models comprehensive patch interactions by constructing fully connected graphs and generating long-range edge attention for information aggregation. This approach addresses the problem of limited patch interactions in MIL and boosts MIL’s contextual awareness. 2) Reinforced Contrastive MIL (ReCMIL) refines the bag embedding space by selecting the most advantageous contrastive bag pairs using a policy network. ReCMIL addresses the problem of suboptimal bag embeddings, enabling MIL to generate discriminative aggregated embeddings. Experimental results demonstrate the superiority of our proposed ICMIL method over state-of-the-art approaches on four publicly available datasets, which span three anatomical sites and encompass five classification tasks (including both binary and multiclass classification). Furthermore, we extend ICMIL with features from a pretrained foundation model and achieve the best performance. Specifically, ICMIL achieves validation accuracies of 92.39% on CRC-DX, 86.86% on CRC-KR, 100.00% on BRACS (binary), 66.67% on BRACS (multiclass), and 96.04% on TCGA-Lung. These findings highlight the strong potential of integrating ICMIL with pretrained foundation models for histopathology image analysis. The code will be available at: https://github.com/JingjiaoLou/ICMIL