Clinical knowledge enhanced medical image classification

Due to the scarcity of data in medical field, deep learning-based medical image classification faces challenges in both accuracy and reliability. Foundation models (FMs) provide a promising enhancement strategy by extracting the text medical knowledge embeddings from FMs and use it to guide the specific classification model. However, the clinical knowledge is generally structurized, and the use of pure text as knowledge representation may not be significant enough for enhancing downstream model. Moreover, the lesion areas are generally subtle, combining FMs to downstream model in a coarse-grained manner still faces challenge in precisely attending the lesions. To tackle these challenges, we propose a novel medical image classification model that effectively embeds clinical knowledge through combining graphs and FMs. First, we represent the clinical rules as graphs, where the node describes the critical characteristics of disease. During training, we use FMs to extract the embeddings of node text description, and use graph transformer to extract global representation of graphs. By employing vision transformer to encode input images, we propose a global-local alignment module to transfer clinical knowledge where the embeddings of image branch and graph branch are aligned from image-to-graph level and patch-to-vertex level, respectively. Moreover, we propose a dynamic image patch selection method to reduce the attention of the model to irrelevant and noisy regions. Experimental results on bladder tumor classification dataset verifies that even with limited training data, the proposed method can not only achieve SOTA performance, but also accurately attend the lesion areas, thus improving the trustworthiness.