Multimodality Image Segmentation Embedded with Metasurface Encoder

Feature dimensionality plays an important role for modern medical diagnosis and image processing. In this work, the study introduces an optoelectronic neural network for multimodal image segmentation, which dramatically improves computing speed and decreases imaging acquisition cost in brain tumor diagnostics. Multi-layer metasurfaces are utilized as an image preprocessor that reduces image dimensionality at the physical layer. The low-dimensional image is then processed to a U-Net semantic segmentation network, to handle the complex and heterogeneous nature of brain image data. By using diffractive neural network, the metasurface encoder is optimized and physically constructed with high-efficiency transmission metasurfaces. The entire optoelectronic network attains a structural similarity index measure (SSIM) of 96%, demonstrating its potential to revolutionize on-site medical image processing with its high precision in segmenting brain imaging data.