The preliminary detection of colon cancer serves as critically important in optimizing patient outcomes and chances of survival. A surface plasmon resonance-based refractive index biosensor provides an incredible approach to the non-invasive, superior sensitive detection of colon cancerous tumors. In this proposed work, the design and analysis of the SPR-based square within an incomplete circle refractive index biosensor (SWICRIB) for determination of the colorectal cancer has been reported. This innovative design integrates the advantages of the SPR method with a unique geometry configuration which enhances sensitivity and specificity. Our results demonstrate that SWICRIB’s superior performance in detecting low-concentration analytes makes it a promising tool for early diagnosis and monitoring of colon cancer biomarkers. This study explores the various applications of the SPR-based biosensor in identifying the refractive index contrast between normal colonic mucosa (cecum and sigmoid) and some different kinds of colorectal tumors such as tubulovillous adenomas and adenocarcinomas. The working range of the reported SWICRIB is 1870 nm to 1950 nm. By utilization of the SPR method, we measure the changes in the refractive index at the metal–dielectric interface demonstrating the potential for precise and early detection of colon cancerous cells. For the superior results with a sensitivity of 700 RIU/nm for detecting adenocarcinoma (Ad) cancer cell, the quality factor is 818.52 nm/RIU for tubulovillous adenoma (TA) cancer cell, and a figure of merit is 298.81 adenocarcinomas (Ad) cancer cell. The superior value 0.001631 has been achieved as a detection limit value. The outcomes suggest that reported SWICRIB can differentiate between healthy and cancerous tissues with high accuracy which paves the way for their integration into routine diagnostic techniques. In the reported sensor work, the machine learning (ML) functional on the width variation (4600–5200 nm) and highest R2 evaluation explores 0.96.