PHIM-MIL: Multiple instance learning with prototype similarity-guided feature fusion and hard instance mining for whole slide image classification

The large size of whole slide images (WSIs) in pathology makes it difficult to obtain fine-grained annotations. Therefore, multi-instance learning (MIL) methods are typically utilized to classify histopathology WSIs. However, current models overly focus on local features of instances, neglecting connection between local features and global features. Additionally, they tend to recognize simple instances while struggling to distinguish hard instances. To address the above issues, we design a two-stage MIL model training approach (PHIM-MIL). In the first stage, a downstream aggregation model is pre-trained to equip it with the ability to recognize simple instances. In the second stage, we integrate global information and make the model focus on mining hard instances. First, the similarity between instances and prototypes is leveraged for weighted aggregation and hence obtaining semi-global features, which helps model understand the relationship between each instance and the global features. Then, instance features and semi-global features are fused to enhance instance feature information, bringing similar instances closer while alienating dissimilar ones. Finally, the hard instance mining strategy is employed to process the fused features, improving the pre-trained aggregation model’s capability to recognize and handle hard instances. Extensive experimental results on the GastricCancer and Camelyon16 datasets demonstrate that PHIM-MIL outperforms other latest state-of-the-art methods in terms of performance and computing cost. Meanwhile, PHIM-MIL continues to deliver consistent performance improvements when the feature extraction network is replaced.