A dual deep learning framework using MaskShiftNet and transformer-augmented CNNs for oral cancer diagnosis from histopathology images

Oral squamous cell carcinoma (OSCC) is the most prevalent form of oral cancer and poses a significant health threat to the community due to its high death rate. The early detection of OSCC serves as a crucial element for both successful treatment and better patient survival outcomes. A biopsy represents the traditional method for OSCC detection which requires extensive manual processing and expert evaluation. This paper introduces two innovative deep learning architectures, MaskShiftNet and a combined Convolutional neural network with vision Transformer Network (CNN-TransNet), for the efficient segmentation and classification of OSCC from histopathology images. MaskShiftNet amalgamates color, texture, and shape attributes to precisely delineate malignant areas, enhancing localization while minimizing false positives and negatives. CNN-TransNet is a hybrid model that integrates CNN with transformer-based attention mechanisms for efficient gathering of local as well as global contextual data for the robust identification of early-stage OSCC. Comprehensive experimental assessments indicate that the suggested framework outperforms current methodologies, achieving a classification accuracy of 98.94 %, with precision, sensitivity, and specificity at 98.9 %, 98.96 %, and 97.18 %, respectively. Ablation experiments further emphasize the essential functions of segmentation and hybrid feature extraction in improving OSCC classification. These findings validate the capability of CNN-TransNet as a dependable and effective instrument for automated oral cancer detection.