Deep learning by Vision Transformer to classify bacterial and fungal keratitis using different types of anterior segment images

Abstract

Purpose

To develop three novel Vision Transformer (ViT) frameworks for the specific diagnosis of bacterial and fungal keratitis using different types of anterior segment images and compare their performances.

Design

Retrospective study.

Methods

A ViT was used to classify bacterial and fungal keratitis. We integrated one or more ViTs by adding a vector or by using self-attention to combine different types of anterior segment images (broad-beam, slit-beam, and blue-light). We compared the area under the receiver operating characteristic curve (AUROC) and area under the precision-recall curve (AUPRC) of the models. Cross-validation was performed thrice, and there was no overlap between the validation sets. The training/validation set was divided in an 8:2 ratio based on the number of individuals.

Results

A total of 283 broad-beam, 610 slit-beam, and 342 blue-light images were obtained from 79 patients. 62 (78 %) patients were assigned for training and 17 (22 %) for validation. The AUROC of ViT with broad-beam images was 0.72. The top AUROC score (0.93) was attained by combining the outputs from two ViT models utilizing self-attention, incorporating both broad-beam and slit-beam images. Similarly, the highest AUPRC score (0.93) was reached by fusing the outputs from three ViTs with self-attention, involving broad-beam, slit-beam, and blue-light images.

Conclusions

Despite the limited dataset, we validated ViT with self-attention to learn different types of images to improve recognition accuracy in diagnosing bacterial and fungal keratitis. ViT with self-attention has a meaningful effect on enhancing the diagnostic performance of bacterial and fungal keratitis by combining two or more types of anterior segment images.