HyCoViT: Hybrid Convolution Vision Transformer With Dynamic Dropout for Enhanced Medical Chest X-Ray Classification

Abstract

Medical chest X-ray (CXR) classification necessitates balancing detailed local feature extraction with capturing broader, long-range dependencies, especially when working with limited and heterogeneous datasets. In this paper, we propose HyCoViT, a hybrid model that integrates a custom Convolutional Neural Network (CNN) block with Vision Transformers (ViTs). This approach combines the locality of CNN-based latent space representations with the global attention mechanisms of ViTs. To address overfitting in data-scarce scenarios, we introduce a Dynamic Dropout (DD) algorithm that adaptively adjusts the dropout rate during training. Additionally, we enhance model generalization using a combination of traditional data augmentation and MixUp techniques. We evaluate HyCoViT on a multi-class classification task involving COVID-19, pneumonia, lung opacity, and normal CXR images. While COVID-19 serves as a case study, the model’s design is generalizable to various medical imaging applications. Experimental results show that HyCoViT achieves state-of-the-art (SOTA) performance, with 98.81% accuracy for three-class surpassing the existing CNN-based model by average +4.90%., and SOTA transformer-based average by 2.05%. In four-class classification, HyCoViT achieves the highest accuracy at 96.56%, which is 8.32% higher than the average accuracy of SOTA CNN-based models and 4.96% higher than the average accuracy of other SOTA transformer-based models. These results surpass many existing CNN-based and transformer-based models, demonstrating the robust generalization capabilities of our method. Furthermore, we provide interpretable, attention-based visualizations that highlight crucial lung regions to support context-aware decisions and ultimately improve patient outcomes.