Encoder-shared visual state space network for anterior segment reconstruction

Abstract

The 3D (three-dimensional) reconstruction of the anterior segment obtained from AS-OCT scanning devices is essential for diagnosing and monitoring cornea and iris, as well as for localizing and quantifying keratitis. However, this process faces two significant challenges: (1) The consecutive images acquired through rotational scanning are difficult to align and register; (2) The existing medical image segmentation technology cannot effectively segment the cornea, which are critical preprocessing steps for effective 3D visualization of the anterior segment. To tackle these dual challenges, an encoder-shared visual state space network for the 3D reconstruction of the anterior segment is proposed. This network integrates image alignment and segmentation into a unified framework. It employs the same encoder to handle spatial images for both alignment and segmentation tasks. A visual state space projection method is utilized to compute the homography matrix of adjacent images, thereby facilitating their alignment. Furthermore, we introduce a channel-wise visual state space fusion technique in conjunction with a decoder block that captures complex contextual interdependencies, enhances shape-preserving feature representation, and improves segmentation accuracy. Based on the resulting corneal segmentation outcomes, we accurately reconstruct 3D volume data from the aligned images. Experimental results on the AIDK-Align and CORNEA datasets demonstrate that our proposed method exhibits remarkable performance in terms of anterior segment alignment, corneal segmentation and 3D reconstruction. Furthermore, we compared encoder-shared visual state space network with state-of-the-art medical image segmentation methods and image alignment algorithms, highlighting its advantages in both alignment and segmentation precision. Our code will be made available at https://github.com/qianguiping/Es-VSS.