Real-Time, Multi-Task Mobile Application for Automatic Bleeding and Non-Bleeding Frame Analysis in Video Capsule Endoscopy Using an Ensemble of Faster R-CNN and LinkNet

Abstract

Real-time, multi-task mobile application for automatic bleeding and non-bleeding frame analysis in video capsule endoscopy (VCE) frames is critical for early diagnosis but is currently underexplored. This study presents a mobile application using Flutter that can automatically classify VCE frames as bleeding and non-bleeding, and further identify and segment bleeding areas in real time. The application utilizes an ensemble deep learning model that integrates Faster Region-based Convolutional Neural Network (R-CNN) for frame-level classification and LinkNet for pixel-level segmentation. Faster R-CNN first detects and classifies VCE frames as bleeding or non-bleeding, and subsequently, LinkNet segments the bleeding regions within the frames identified as bleeding. Both models were trained and validated using the publicly available WCEBleedGen dataset. To evaluate the effectiveness of the proposed ensemble, a comparative analysis was conducted with existing studies and state-of-the-art (SOTA) models in the field. For detection, the performance of Faster R-CNN was compared with two You Only Look Once (YOLO) variants: YOLOv5 and YOLOv12. For segmentation, LinkNet was compared with SegNet and UNet. Evaluation metrics included mean Average Precision at 0.5 ([email protected]), Dice coefficient, and Eigen class activation maps. The mobile application achieved an average inference time of 2.88 s per frame and 23.33 s for a batch of 10 frames. Overall, the ensemble model attained a [email protected] of 0.92 and a Dice coefficient of 0.96, outperforming existing studies. For SOTA models, Faster R-CNN outperformed YOLO variants by achieving a 25% higher [email protected], and LinkNet achieved a 26% higher Dice coefficient than SegNet and 5% higher than UNet on the validation dataset and achieved more focused Eigen maps for different bleeding areas. This study represents the first attempt to develop a real-time, multi-task mobile application for VCE bleeding analysis. The application is open-source and freely available at https://github.com/misahub2023/VCE-BleedGen-Application, supporting accessibility, reproducibility, and future research in this field.