Adapting 2D Vision Transformer Backbones for 3D Thoracic Multi-Organ Segmentation

Abstract

Accurate multi-organ segmentation in thoracic CT scans is essential for radiotherapy planning and clinical diagnosis. However, this task remains challenging due to large anatomical variability, small organ sizes, inter-slice discontinuities, and the computational demands of volumetric segmentation. We propose PVT3D-ThoraxNet, a hybrid 2D–3D framework that integrates a Pyramid Vision Transformer (PVTv2) encoder with a convolutional 3D decoder via a novel 3D Context Encoder, enabling effective fusion of multi-slice features. To further enhance structural consistency, we introduce a 3D Trainable Guided Filter (TGF) in the decoder for boundary refinement. On the Lung CT Segmentation Challenge (LCTSC) dataset across five thoracic organs (esophagus, heart, left lung, right lung, spinal cord), PVT3D-ThoraxNet achieves a mean Dice Similarity Coefficient of 0.903 and a mean HD95 of 3.59 mm. On a private thoracic CT dataset, it generalizes well with a mean Dice of 0.875 and a mean HD95 of 4.81 mm, without dataset-specific fine-tuning. Compared with recent multi-stage and transformer-based approaches, our framework provides a lightweight, robust, and accurate solution for thoracic multi-organ segmentation.