Reconstructing 3D Biomedical Architectural Order at Multiple Spatial Scales with Multimodal Stack Input

Microscopy, crucial for exploring biological structures, often uses polarizing microscopes to observe tissue anisotropy and reconstruct label-free images. However, these images typically show low contrast, and while fluorescence imaging offers higher contrast, it is phototoxic and can disrupt natural assembly dynamics. This study focuses on reconstructing fluorescence images from label-free ones using a complex nonlinear transformation, specifically aiming to identify organelles within diverse optical properties of tissues. A multimodal deep learning model, 3DTransMDL, was developed, employing the Stokes vector to analyze the sample’s retardance, phase, and orientation. This model incorporates isotropy and anisotropy to differentiate organelles, enhancing the input with structures' varied optical properties. Additionally, techniques like background distortion normalization and covariate shift methods were applied to reduce noise and overfitting, improving model generalization. The approach was tested on mouse kidney and human brain tissues, successfully identifying specific organelles and demonstrating superior performance in reconstructing 3D images, significantly reducing artifacts compared to 2D predictions. Evaluation metrics such as SSIM, PCC, and R2 score confirm the model's efficacy, with improvements observed in multi-modality input setups. This advancement suggests potential applications in molecular dynamics, aiming for further enhancements in future studies.