Full fine-tuning strategy for endoscopic foundation models with expanded learnable offset parameters.

Abstract

In the medical field, endoscopic video analysis is crucial for disease diagnosis and minimally invasive surgery. The Endoscopic Foundation Models (Endo-FM) utilize large-scale self-supervised pre-training on endoscopic video data and leverage video transformer models to capture long-range spatiotemporal dependencies. However, detecting complex lesions such as gastrointestinal metaplasia (GIM) in endoscopic videos remains challenging due to unclear boundaries and indistinct features, and Endo-FM has not demonstrated good performance. To this end, we propose a fully fine-tuning strategy with an Extended Learnable Offset Parameter (ELOP), which improves model performance by introducing learnable offset parameters in the input space. Specifically, we propose a novel loss function that combines cross-entropy loss and focal loss through a weighted sum, enabling the model to better focus on hard-to-classify samples during training. We validated ELOP on a private GIM dataset from a local grade-A tertiary hospital and a public polyp detection dataset. Experimental results show that ELOP significantly improves the detection accuracy, achieving accuracy improvements of 6.25 % and 3.75%respectively compared to the original Endo-FM. In summary, ELOP provides an excellent solution for detecting complex lesions in endoscopic videos, achieving more precise diagnoses.