PM2: A new prompting multi-modal model paradigm for few-shot medical image classification

Abstract

Background and Objective:

Few-shot learning has emerged as a key technological solution to address challenges such as limited data and the difficulty of acquiring annotations in medical image classification. However, relying solely on a single image modality is insufficient to capture conceptual categories. Therefore, medical image classification requires a comprehensive approach to capture conceptual category information that aids in the interpretation of image content.

Methods:

This study proposes a novel medical image classification paradigm based on a multi-modal foundation model, called PM². In addition to the image modality, PM² introduces supplementary text input (prompt) to further describe images or conceptual categories and facilitate cross-modal few-shot learning. We empirically studied five different prompting schemes under this new paradigm. Furthermore, linear probing in multi-modal models only takes class token as input, ignoring the rich statistical data contained in high-level visual tokens. Therefore, we alternately perform linear classification on the feature distributions of visual tokens and class token. To effectively extract statistical information, we use global covariance pool with efficient matrix power normalization to aggregate the visual tokens. We then combine two classification heads: one for handling image class token and prompt representations encoded by the text encoder, and the other for classifying the feature distributions of visual tokens.

Results:

Experimental results on three datasets: breast cancer, brain tumor, and diabetic retinopathy demonstrate that PM² effectively improves the performance of medical image classification. Compared to existing multi-modal models, PM² achieves state-of-the-art performance.

Conclusions:

Integrating text prompts as supplementary samples effectively enhances the model’s performance. Additionally, by leveraging second-order features of visual tokens to enrich the category feature space and combining them with class token, the model’s representational capacity is significantly strengthened.