Research on Multi-classification of TCM Tongue Symptoms Based on Self-supervised Learning

The advancement of computer technology has been progressing rapidly and the need for social life development, “Internet + health care” has become a necessary trend in human life, and the intelligent research of TCM (Traditional Chinese Medicine) tongue diagnosis is also one of the development directions. In recent years, neural networks have made significant progress in the field of medical image classification. However, neural networks often need to be built based on an extensive amount of data instances to achieve more precise results. In the medical field, collecting large and labeled datasets is costly. Considering this situation, we introduce a self-supervised Transformer framework, named SViT-T5, and build a masked image modeling task based on the Vision Transformer (ViT) architecture. The method treats restoring pixels as a proxy task for self-supervised learning, and image classification as a downstream task. Pre-train the model by using a great deal of unlabeled data and fine-tuning with the TongueNet dataset we created, five categories of tongue images are studied: cold syndrome, heat syndrome, deficiency syndrome, excess syndrome, and normal tongue image. And compare the classification results with the basic models without pre-training (ConvNeXt, ResNeXt, ViT, Swin Transformer) and pre-training on the ImageNet dataset. According to the results, the classification accuracy of our proposed SViT-T5 framework based on self-supervised learning reaches 85.938%. In contrast to other methods, our approach exhibits superior classification accuracy and generalization ability.