IEEE transactions on medical imaging最新文献

筛选
英文 中文
CTUSurv: A Cell-Aware Transformer-Based Network With Uncertainty for Survival Prediction Using Whole Slide Images ct篡位:一种基于细胞感知变压器的不确定性网络,用于使用整个幻灯片图像进行生存预测
IEEE transactions on medical imaging Pub Date : 2025-01-08 DOI: 10.1109/TMI.2025.3526848
Zhihao Tang;Lin Yang;Zongyi Chen;Li Liu;Chaozhuo Li;Ruanqi Chen;Xi Zhang;Qingfeng Zheng
{"title":"CTUSurv: A Cell-Aware Transformer-Based Network With Uncertainty for Survival Prediction Using Whole Slide Images","authors":"Zhihao Tang;Lin Yang;Zongyi Chen;Li Liu;Chaozhuo Li;Ruanqi Chen;Xi Zhang;Qingfeng Zheng","doi":"10.1109/TMI.2025.3526848","DOIUrl":"10.1109/TMI.2025.3526848","url":null,"abstract":"Image-based survival prediction through deep learning techniques represents a burgeoning frontier aimed at augmenting the diagnostic capabilities of pathologists. However, directly applying existing deep learning models to survival prediction may not be a panacea due to the inherent complexity and sophistication of whole slide images (WSIs). The intricate nature of high-resolution WSIs, characterized by sophisticated patterns and inherent noise, presents significant challenges in terms of effectiveness and trustworthiness. In this paper, we propose CTUSurv, a novel survival prediction model designed to simultaneously capture cell-to-cell and cell-to-microenvironment interactions, complemented by a region-based uncertainty estimation framework to assess the reliability of survival predictions. Our approach incorporates an innovative region sampling strategy to extract task-relevant, informative regions from high-resolution WSIs. To address the challenges posed by sophisticated biological patterns, a cell-aware encoding module is integrated to model the interactions among biological entities. Furthermore, CTUSurv includes a novel aleatoric uncertainty estimation module to provide fine-grained uncertainty scores at the region level. Extensive evaluations across four datasets demonstrate the superiority of our proposed approach in terms of both predictive accuracy and reliability.","PeriodicalId":94033,"journal":{"name":"IEEE transactions on medical imaging","volume":"44 4","pages":"1750-1764"},"PeriodicalIF":0.0,"publicationDate":"2025-01-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142936704","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Amyloid-β Deposition Prediction With Large Language Model Driven and Task-Oriented Learning of Brain Functional Networks 脑功能网络的大语言模型驱动和任务导向学习预测淀粉样蛋白-β沉积
IEEE transactions on medical imaging Pub Date : 2025-01-03 DOI: 10.1109/TMI.2024.3525022
Yuxiao Liu;Mianxin Liu;Yuanwang Zhang;Yihui Guan;Qihao Guo;Fang Xie;Dinggang Shen
{"title":"Amyloid-β Deposition Prediction With Large Language Model Driven and Task-Oriented Learning of Brain Functional Networks","authors":"Yuxiao Liu;Mianxin Liu;Yuanwang Zhang;Yihui Guan;Qihao Guo;Fang Xie;Dinggang Shen","doi":"10.1109/TMI.2024.3525022","DOIUrl":"10.1109/TMI.2024.3525022","url":null,"abstract":"Amyloid-<inline-formula> <tex-math>$beta $ </tex-math></inline-formula> positron emission tomography can reflect the Amyloid-<inline-formula> <tex-math>$beta $ </tex-math></inline-formula> protein deposition in the brain and thus serves as one of the golden standards for Alzheimer’s disease (AD) diagnosis. However, its practical cost and high radioactivity hinder its application in large-scale early AD screening. Recent neuroscience studies suggest a strong association between changes in functional connectivity network (FCN) derived from functional MRI (fMRI), and deposition patterns of Amyloid-<inline-formula> <tex-math>$beta $ </tex-math></inline-formula> protein in the brain. This enables an FCN-based approach to assess the Amyloid-<inline-formula> <tex-math>$beta $ </tex-math></inline-formula> protein deposition with less expense and radioactivity. However, an effective FCN-based Amyloid-<inline-formula> <tex-math>$beta $ </tex-math></inline-formula> assessment remains lacking for practice. In this paper, we introduce a novel deep learning framework tailored for this task. Our framework comprises three innovative components: 1) a pre-trained Large Language Model Nodal Embedding Encoder, designed to extract task-related features from fMRI signals; 2) a task-oriented Hierarchical-order FCN Learning module, used to enhance the representation of complex correlations among different brain regions for improved prediction of Amyloid-<inline-formula> <tex-math>$beta $ </tex-math></inline-formula> deposition; and 3) task-feature consistency losses for promoting similarity between predicted and real Amyloid-<inline-formula> <tex-math>$beta $ </tex-math></inline-formula> values and ensuring effectiveness of predicted Amyloid-<inline-formula> <tex-math>$beta $ </tex-math></inline-formula> in downstream classification task. Experimental results show superiority of our method over several state-of-the-art FCN-based methods. Additionally, we identify crucial functional sub-networks for predicting Amyloid-<inline-formula> <tex-math>$beta $ </tex-math></inline-formula> depositions. The proposed method is anticipated to contribute valuable insights into the understanding of mechanisms of AD and its prevention.","PeriodicalId":94033,"journal":{"name":"IEEE transactions on medical imaging","volume":"44 4","pages":"1809-1820"},"PeriodicalIF":0.0,"publicationDate":"2025-01-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142924506","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Exploring Contrastive Pre-Training for Domain Connections in Medical Image Segmentation 医学图像分割领域连接对比预训练的探索
IEEE transactions on medical imaging Pub Date : 2025-01-03 DOI: 10.1109/TMI.2024.3525095
Zequn Zhang;Yun Jiang;Yunnan Wang;Baao Xie;Wenyao Zhang;Yuhang Li;Zhen Chen;Xin Jin;Wenjun Zeng
{"title":"Exploring Contrastive Pre-Training for Domain Connections in Medical Image Segmentation","authors":"Zequn Zhang;Yun Jiang;Yunnan Wang;Baao Xie;Wenyao Zhang;Yuhang Li;Zhen Chen;Xin Jin;Wenjun Zeng","doi":"10.1109/TMI.2024.3525095","DOIUrl":"10.1109/TMI.2024.3525095","url":null,"abstract":"Unsupervised domain adaptation (UDA) in medical image segmentation aims to improve the generalization of deep models by alleviating domain gaps caused by inconsistency across equipment, imaging protocols, and patient conditions. However, existing UDA works remain insufficiently explored and present great limitations: 1) Exhibit cumbersome designs that prioritize aligning statistical metrics and distributions, which limits the model’s flexibility and generalization while also overlooking the potential knowledge embedded in unlabeled data; 2) More applicable in a certain domain, lack the generalization capability to handle diverse shifts encountered in clinical scenarios. To overcome these limitations, we introduce MedCon, a unified framework that leverages general unsupervised contrastive pre-training to establish domain connections, effectively handling diverse domain shifts without tailored adjustments. Specifically, it initially explores a general contrastive pre-training to establish domain connections by leveraging the rich prior knowledge from unlabeled images. Thereafter, the pre-trained backbone is fine-tuned using source-based images to ultimately identify per-pixel semantic categories. To capture both intra- and inter-domain connections of anatomical structures, we construct positive-negative pairs from a hybrid aspect of both local and global scales. In this regard, a shared-weight encoder-decoder is employed to generate pixel-level representations, which are then mapped into hyper-spherical space using a non-learnable projection head to facilitate positive pair matching. Comprehensive experiments on diverse medical image datasets confirm that MedCon outperforms previous methods by effectively managing a wide range of domain shifts and showcasing superior generalization capabilities.","PeriodicalId":94033,"journal":{"name":"IEEE transactions on medical imaging","volume":"44 4","pages":"1686-1698"},"PeriodicalIF":0.0,"publicationDate":"2025-01-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142924430","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Enhanced DTCMR With Cascaded Alignment and Adaptive Diffusion 基于级联对准和自适应扩散的增强DTCMR
IEEE transactions on medical imaging Pub Date : 2024-12-30 DOI: 10.1109/TMI.2024.3523431
Fanwen Wang;Yihao Luo;Camila Munoz;Ke Wen;Yaqing Luo;Jiahao Huang;Yinzhe Wu;Zohya Khalique;Maria Molto;Ramyah Rajakulasingam;Ranil de Silva;Dudley J. Pennell;Pedro F. Ferreira;Andrew D. Scott;Sonia Nielles-Vallespin;Guang Yang
{"title":"Enhanced DTCMR With Cascaded Alignment and Adaptive Diffusion","authors":"Fanwen Wang;Yihao Luo;Camila Munoz;Ke Wen;Yaqing Luo;Jiahao Huang;Yinzhe Wu;Zohya Khalique;Maria Molto;Ramyah Rajakulasingam;Ranil de Silva;Dudley J. Pennell;Pedro F. Ferreira;Andrew D. Scott;Sonia Nielles-Vallespin;Guang Yang","doi":"10.1109/TMI.2024.3523431","DOIUrl":"10.1109/TMI.2024.3523431","url":null,"abstract":"Diffusion tensor cardiovascular magnetic resonance (DTCMR) is the only non-invasive method for visualizing myocardial microstructure, but it is challenged by inconsistent breath-holds and imperfect cardiac triggering, causing in-plane shifts and through-plane warping with an inadequate tensor fitting. While rigid registration corrects in-plane shifts, deformable registration risks distorting the diffusion distribution, and selecting a reference frame among low SNR frames is challenging. Existing pairwise deep learning and iterative methods are unsuitable for DTCMR due to their inability to handle the drastic in-plane motion and disentangle the diffusion contrast distortion with through-plane motions on low SNR frames, which compromises the accuracy of clinical biomarker tensor estimation. Our study introduces a novel deep learning framework incorporating tensor information for groupwise deformable registration, effectively correcting intra-subject inter-frame motion. This framework features a cascaded registration branch for addressing in-plane and through-plane motions and a parallel branch for generating pseudo-frames with diffusion contrasts and template updates to guide registration with a refined loss function and denoising. We evaluated our method on four DTCMR-specific metrics using data from over 900 cases from 2012 to 2023. Our method outperformed three traditional and two deep learning-based methods, achieving reduced fitting errors, the lowest percentage of negative eigenvalues at 0.446%, the highest R2 of HA line profiles at 0.911, no negative Jacobian Determinant, and the shortest reference time of 0.06 seconds per case. In conclusion, our deep learning framework significantly improves DTCMR imaging by effectively correcting inter-frame motion and surpassing existing methods across multiple metrics, demonstrating substantial clinical potential.","PeriodicalId":94033,"journal":{"name":"IEEE transactions on medical imaging","volume":"44 4","pages":"1866-1877"},"PeriodicalIF":0.0,"publicationDate":"2024-12-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142905100","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Single-Sided Magnetic Particle Imaging Device With Offset Field Based Spatial Encoding 基于偏移场空间编码的单面磁粉成像设备
IEEE transactions on medical imaging Pub Date : 2024-12-30 DOI: 10.1109/TMI.2024.3522979
Qibin Wang;Zhonghao Zhang;Lei Li;Franziska Schrank;Yu Zeng;Pengyue Guo;Harald Radermacher;Volkmar Schulz;Shouping Zhu
{"title":"Single-Sided Magnetic Particle Imaging Device With Offset Field Based Spatial Encoding","authors":"Qibin Wang;Zhonghao Zhang;Lei Li;Franziska Schrank;Yu Zeng;Pengyue Guo;Harald Radermacher;Volkmar Schulz;Shouping Zhu","doi":"10.1109/TMI.2024.3522979","DOIUrl":"10.1109/TMI.2024.3522979","url":null,"abstract":"Single-sided Magnetic Particle Imaging (MPI) devices enable easy imaging of areas outside the MPI device, allowing objects of any size to be imaged and improving clinical applicability. However, current single-sided MPI devices face challenges in generating high-gradient selection fields and experience a decrease in gradient strength with increasing detection depth, which limits the detection depth and resolution. We introduce a novel spatial encoding method. This method combines high-frequency alternating excitation fields with variable offset fields, leveraging the inherent characteristic of single-sided MPI devices where the magnetic field strength attenuates with distance. Consequently, the harmonic signals of particle responses at different spatial positions vary. By manipulating multiple offset fields, we correlate the nonlinear harmonic responses of magnetic particles with spatial position data. In this work, we employed an image reconstruction using a system matrix approach, which takes into account the spatial distribution of the magnetic field during the movement of the device within the field of view. Our proposed encoding approach eliminates the need for the classical selection field and directly links the spatial resolution to the strength and spatial distribution of the magnetic field, thus reducing the dependency of resolution on selection field gradients strength. We have demonstrated the feasibility of the proposed method through simulations and phantom measurements.","PeriodicalId":94033,"journal":{"name":"IEEE transactions on medical imaging","volume":"44 4","pages":"1878-1889"},"PeriodicalIF":0.0,"publicationDate":"2024-12-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142905101","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
MRI Motion Correction Through Disentangled CycleGAN Based on Multi-Mask K-Space Subsampling 基于多掩模k空间子采样的解纠缠CycleGAN MRI运动校正
IEEE transactions on medical imaging Pub Date : 2024-12-30 DOI: 10.1109/TMI.2024.3523949
Gang Chen;Han Xie;Xinglong Rao;Xinjie Liu;Martins Otikovs;Lucio Frydman;Peng Sun;Zhi Zhang;Feng Pan;Lian Yang;Xin Zhou;Maili Liu;Qingjia Bao;Chaoyang Liu
{"title":"MRI Motion Correction Through Disentangled CycleGAN Based on Multi-Mask K-Space Subsampling","authors":"Gang Chen;Han Xie;Xinglong Rao;Xinjie Liu;Martins Otikovs;Lucio Frydman;Peng Sun;Zhi Zhang;Feng Pan;Lian Yang;Xin Zhou;Maili Liu;Qingjia Bao;Chaoyang Liu","doi":"10.1109/TMI.2024.3523949","DOIUrl":"10.1109/TMI.2024.3523949","url":null,"abstract":"This work proposes a new retrospective motion correction method, termed DCGAN-MS, which employs disentangled CycleGAN based on multi-mask k-space subsampling (DCGAN-MS) to address the image domain translation challenge. The multi-mask k-space subsampling operator is utilized to decrease the complexity of motion artifacts by randomly discarding motion-affected k-space lines. The network then disentangles the subsampled, motion-corrupted images into content and artifact features using specialized encoders, and generates motion-corrected images by decoding the content features. By utilizing multi-mask k-space subsampling, motion artifact features become more sparse compared to the original image domain, enhancing the efficiency of the DCGAN-MS network. This method effectively corrects motion artifacts in clinical gadoxetic acid-enhanced human liver MRI, human brain MRI from fastMRI, and preclinical rodent brain MRI. Quantitative improvements are demonstrated with SSIM values increasing from 0.75 to 0.86 for human liver MRI with simulated motion artifacts, and from 0.72 to 0.82 for rodent brain MRI with simulated motion artifacts. Correspondingly, PSNR values increased from 26.09 to 31.09 and from 25.10 to 31.77. The method’s performance was further validated on clinical and preclinical motion-corrupted MRI using the Kernel Inception Distance (KID) and Fréchet Inception Distance (FID) metrics. Additionally, ablation experiments were conducted to confirm the effectiveness of the multi-mask k-space subsampling approach.","PeriodicalId":94033,"journal":{"name":"IEEE transactions on medical imaging","volume":"44 4","pages":"1907-1921"},"PeriodicalIF":0.0,"publicationDate":"2024-12-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142905099","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Cross-Modal Conditioned Reconstruction for Language-Guided Medical Image Segmentation 语言引导下医学图像分割的跨模态条件重构
IEEE transactions on medical imaging Pub Date : 2024-12-26 DOI: 10.1109/TMI.2024.3523333
Xiaoshuang Huang;Hongxiang Li;Meng Cao;Long Chen;Chenyu You;Dong An
{"title":"Cross-Modal Conditioned Reconstruction for Language-Guided Medical Image Segmentation","authors":"Xiaoshuang Huang;Hongxiang Li;Meng Cao;Long Chen;Chenyu You;Dong An","doi":"10.1109/TMI.2024.3523333","DOIUrl":"10.1109/TMI.2024.3523333","url":null,"abstract":"Recent developments underscore the potential of textual information in enhancing learning models for a deeper understanding of medical visual semantics. However, language-guided medical image segmentation still faces a challenging issue. Previous works employ implicit architectures to embed textual information. This leads to segmentation results that are inconsistent with the semantics represented by the language, sometimes even diverging significantly. To this end, we propose a novel cross-modal conditioned Reconstruction for Language-guided Medical Image Segmentation (RecLMIS) to explicitly capture cross-modal interactions, which assumes that well-aligned medical visual features and medical notes can effectively reconstruct each other. We introduce conditioned interaction to adaptively predict patches and words of interest. Subsequently, they are utilized as conditioning factors for mutual reconstruction to align with regions described in the medical notes. Extensive experiments demonstrate the superiority of our RecLMIS, surpassing LViT by 3.74% mIoU on the MosMedData+ dataset and 1.89% mIoU on the QATA-CoV19 dataset. More importantly, we achieve a relative reduction of 20.2% in parameter count and a 55.5% decrease in computational load. The code will be available at <uri>https://github.com/ShawnHuang497/RecLMIS</uri>.","PeriodicalId":94033,"journal":{"name":"IEEE transactions on medical imaging","volume":"44 4","pages":"1821-1835"},"PeriodicalIF":0.0,"publicationDate":"2024-12-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142887567","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
VBVT-Net: VOI-Based VVBP-Tensor Network for High-Attenuation Artifact Suppression in Digital Breast Tomosynthesis Imaging VBVT-Net:基于voi的vvbp张量网络在数字乳房断层合成成像中的高衰减伪影抑制
IEEE transactions on medical imaging Pub Date : 2024-12-26 DOI: 10.1109/TMI.2024.3522242
Manman Zhu;Zidan Wang;Chen Wang;Cuidie Zeng;Dong Zeng;Jianhua Ma;Yongbo Wang
{"title":"VBVT-Net: VOI-Based VVBP-Tensor Network for High-Attenuation Artifact Suppression in Digital Breast Tomosynthesis Imaging","authors":"Manman Zhu;Zidan Wang;Chen Wang;Cuidie Zeng;Dong Zeng;Jianhua Ma;Yongbo Wang","doi":"10.1109/TMI.2024.3522242","DOIUrl":"10.1109/TMI.2024.3522242","url":null,"abstract":"High-attenuation (HA) artifacts may lead to obscured subtle lesions and lesion over-estimation in digital breast tomosynthesis (DBT) imaging. High-attenuation artifact suppression (HAAS) is vital for widespread DBT applications in clinic. The conventional HAAS methods usually rely on the segmentation accuracy of HA objects and manual weighting schemes, without considering the geometry information in DBT reconstruction. And the global weighted strategy designed for HA artifacts may decrease the resolution in low-contrast soft-tissue regions. Moreover, the view-by-view backprojection tensor (VVBP-Tensor) domain has recently developed as a new intermediary domain that contains the lossless information in projection domain and the structural details in image domain. Therefore, we propose a VOI-Based VVBP-Tensor Network (VBVT-Net) for HAAS task in DBT imaging, which learns a local implicit weighted strategy based on the analytical FDK reconstruction mechanism. Specifically, the VBVT-Net method incorporates a volume of interest (VOI) recognition sub-network and a HAAS sub-network. The VOI recognition sub-network automatically extracts all 4D VVBP-Tensor patches containing HA artifacts. The HAAS sub-network reduces HA artifacts in these 4D VVBP-Tensor patches by leveraging the ray-trace backprojection features and extra neighborhood information. All results on four datasets demonstrate that the proposed VBVT-Net method could accurately detect HA regions, effectively reduce HA artifacts and simultaneously preserve structures in soft-tissue background regions. The proposed VBVT-Net method has a good interpretability as a general variant of the weighted FDK algorithm, which is potential to be applied in the next generation DBT prototype system in the future.","PeriodicalId":94033,"journal":{"name":"IEEE transactions on medical imaging","volume":"44 4","pages":"1953-1968"},"PeriodicalIF":0.0,"publicationDate":"2024-12-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142887565","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Tackling Modality-Heterogeneous Client Drift Holistically for Heterogeneous Multimodal Federated Learning 从整体上解决异构多模态联邦学习的模态-异构客户端漂移
IEEE transactions on medical imaging Pub Date : 2024-12-26 DOI: 10.1109/TMI.2024.3523378
Haoyue Song;Jiacheng Wang;Jianjun Zhou;Liansheng Wang
{"title":"Tackling Modality-Heterogeneous Client Drift Holistically for Heterogeneous Multimodal Federated Learning","authors":"Haoyue Song;Jiacheng Wang;Jianjun Zhou;Liansheng Wang","doi":"10.1109/TMI.2024.3523378","DOIUrl":"10.1109/TMI.2024.3523378","url":null,"abstract":"Multimodal Federated Learning (MFL) has emerged as a collaborative paradigm for training models across decentralized devices, harnessing various data modalities to facilitate effective learning while respecting data ownership. In this realm, notably, a pivotal shift from homogeneous to heterogeneous MFL has taken place. While the former assumes uniformity in input modalities across clients, the latter accommodates modality-incongruous setups, which is often the case in practical situations. For example, while some advanced medical institutions have the luxury of utilizing both MRI and CT for disease diagnosis, remote hospitals often find themselves constrained to employ CT exclusively due to its cost-effectiveness. Although heterogeneous MFL can apply to a broader scenario, it introduces a new challenge: modality-heterogeneous client drift, arising from diverse modality-coupled local optimization. To address this, we introduce FedMM, a simple yet effective approach. During local optimization, FedMM employs modality dropout, randomly masking available modalities, and promoting weight alignment while preserving model expressivity on its original modality combination. To enhance the modality dropout process, FedMM incorporates a task-specific inter- and intra-modal regularizer, which acts as an additional constraint, forcing that weight distribution remains more consistent across diverse input modalities and therefore eases the optimization process with modality dropout enabled. By combining them, our approach holistically addresses client drift. It fosters convergence among client models while considering each client’s unique input modalities, enhancing heterogeneous MFL performance. Comprehensive evaluations in three medical image segmentation datasets demonstrate FedMM’s superiority over state-of-the-art heterogeneous MFL methods.","PeriodicalId":94033,"journal":{"name":"IEEE transactions on medical imaging","volume":"44 4","pages":"1931-1941"},"PeriodicalIF":0.0,"publicationDate":"2024-12-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142887410","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Three-Dimensional Whole-Body Small Animal Photoacoustic Tomography Using a Multi-View Fabry-Perot Scanner 三维全身小动物光声断层扫描使用多视图法布里-珀罗扫描仪
IEEE transactions on medical imaging Pub Date : 2024-12-25 DOI: 10.1109/TMI.2024.3522220
Olumide Ogunlade;Robert Ellwood;Edward Zhang;Benjamin T. Cox;Paul Beard
{"title":"Three-Dimensional Whole-Body Small Animal Photoacoustic Tomography Using a Multi-View Fabry-Perot Scanner","authors":"Olumide Ogunlade;Robert Ellwood;Edward Zhang;Benjamin T. Cox;Paul Beard","doi":"10.1109/TMI.2024.3522220","DOIUrl":"10.1109/TMI.2024.3522220","url":null,"abstract":"Photoacoustic tomography (PAT) has the potential to become a widely used imaging tool in preclinical studies of small animals. This is because it can provide non-invasive, label free images of whole-body mouse anatomy, in a manner which is challenging for more established imaging modalities. However, existing PAT scanners are limited because they either do not implement a full 3-D tomographic reconstruction using all the recorded photoacoustic (PA) data and/or do not record the available 3-D PA time-series data around the mouse with sufficiently high spatial resolution (<inline-formula> <tex-math>$sim 100mu $ </tex-math></inline-formula>m), which compromises image quality in terms of resolution, imaging depth and the introduction of artefacts. In this study, we address these limitations by demonstrating an all-optical, multi-view Fabry-Perot based scanner for whole body small animal imaging. The scanner densely samples the acoustic field with a large number of detection points (>100,000), evenly distributed around the mouse. The locations of the detection points were registered onto a common coordinate system, before a tomographic reconstruction using all the recorded PA time series was implemented. This enabled the acquisition of high resolution, whole-body PAT images of ex-vivo mice, with anatomical features visible across the entire cross section.","PeriodicalId":94033,"journal":{"name":"IEEE transactions on medical imaging","volume":"44 4","pages":"1922-1930"},"PeriodicalIF":0.0,"publicationDate":"2024-12-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142887643","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
相关产品
×
本文献相关产品
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信