CNN-ViT: A multi-feature learning based approach for driver drowsiness detection

Abstract

Driver drowsiness remains a critical contributor to road accidents, frequently resulting in severe injuries and fatalities. To address this issue, the present study proposes an advanced drowsiness detection system that combines the competencies of Convolutional Neural Networks (CNNs) — namely DenseNet121, VGG16, VGG19, and ResNet50 — with a Vision Transformer (ViT). This hybrid framework is designed to harness the complementary strengths of CNNs and transformers: CNNs excel at capturing fine-grained local features, while ViT effectively models global dependencies within images. The input images are processed simultaneously through both branches, and their extracted features are merged and used to classify the driver’s state into one of four categories: Closed, Open, no_yawn, or yawn. The proposed system was evaluated on two separate datasets, named Dataset-1 and Dataset-2. Results demonstrated that the ResNet50_ViT hybrid attained a high accuracy of 99.76% on Dataset-1, while the VGG19_ViT model attained 98.21% on Dataset-2. Performance was assessed using metrics such as accuracy, precision, F1-score, and recall. The strong results, supported by optimized hyperparameters, highlight the reliability and effectiveness of the hybrid model for real-time driver drowsiness detection.