早期和晚期融合对不完全配准多模态磁共振成像胰腺分割的影响。

IF 1.7 Q3 RADIOLOGY, NUCLEAR MEDICINE & MEDICAL IMAGING
Journal of Medical Imaging Pub Date : 2025-03-01 Epub Date: 2025-04-26 DOI:10.1117/1.JMI.12.2.024008
Lucas W Remedios, Han Liu, Samuel W Remedios, Lianrui Zuo, Adam M Saunders, Shunxing Bao, Yuankai Huo, Alvin C Powers, John Virostko, Bennett A Landman
{"title":"早期和晚期融合对不完全配准多模态磁共振成像胰腺分割的影响。","authors":"Lucas W Remedios, Han Liu, Samuel W Remedios, Lianrui Zuo, Adam M Saunders, Shunxing Bao, Yuankai Huo, Alvin C Powers, John Virostko, Bennett A Landman","doi":"10.1117/1.JMI.12.2.024008","DOIUrl":null,"url":null,"abstract":"<p><strong>Purpose: </strong>Combining different types of medical imaging data, through multimodal fusion, promises better segmentation of anatomical structures, such as the pancreas. Strategic implementation of multimodal fusion could improve our ability to study diseases such as diabetes. However, where to perform fusion in deep learning models is still an open question. It is unclear if there is a single best location to fuse information when analyzing pairs of imperfectly aligned images or if the optimal fusion location depends on the specific model being used. Two main challenges when using multiple imaging modalities to study the pancreas are (1) the pancreas and surrounding abdominal anatomy have a deformable structure, making it difficult to consistently align the images and (2) breathing by the individual during image collection further complicates the alignment between multimodal images. Even after using state-of-the-art deformable image registration techniques, specifically designed to align abdominal images, multimodal images of the abdomen are often not perfectly aligned. We examine how the choice of different fusion points, ranging from early in the image processing pipeline to later stages, impacts the segmentation of the pancreas on imperfectly registered multimodal magnetic resonance (MR) images.</p><p><strong>Approach: </strong>Our dataset consists of 353 pairs of T2-weighted (T2w) and T1-weighted (T1w) abdominal MR images from 163 subjects with accompanying pancreas segmentation labels drawn mainly based on the T2w images. Because the T2w images were acquired in an interleaved manner across two breath holds and the T1w images on one breath hold, there were three different breath holds impacting the alignment of each pair of images. We used deeds, a state-of-the-art deformable abdominal image registration method to align the image pairs. Then, we trained a collection of basic UNets with different fusion points, spanning from early to late layers in the model, to assess how early through late fusion influenced segmentation performance on imperfectly aligned images. To investigate whether performance differences on key fusion points are generalized to other architectures, we expanded our experiments to nnUNet.</p><p><strong>Results: </strong>The single-modality T2w baseline using a basic UNet model had a median Dice score of 0.766, whereas the same baseline on the nnUNet model achieved 0.824. For each fusion approach, we analyzed the differences in performance with Dice residuals, by subtracting the baseline score from the fusion score for each datapoint. For the basic UNet, the best fusion approach was from early/mid fusion and occurred in the middle of the encoder with a median Dice residual of <math><mrow><mo>+</mo> <mn>0.012</mn></mrow> </math> compared with the baseline. For the nnUNet, the best fusion approach was early fusion through naïve image concatenation before the model, with a median Dice residual of <math><mrow><mo>+</mo> <mn>0.004</mn></mrow> </math> compared with the baseline. After Bonferroni correction, the distributions of the Dice scores for these best fusion approaches were found to be statistically significant ( <math><mrow><mi>p</mi> <mo><</mo> <mn>0.05</mn></mrow> </math> ) via the paired Wilcoxon signed-rank test against the baseline.</p><p><strong>Conclusions: </strong>Fusion in specific blocks can improve performance, but the best blocks for fusion are model-specific, and the gains are small. In imperfectly registered datasets, fusion is a nuanced problem, with the art of design remaining vital for uncovering potential insights. Future innovation is needed to better address fusion in cases of imperfect alignment of abdominal image pairs. The code associated with this project is available here https://github.com/MASILab/influence_of_fusion_on_pancreas_segmentation.</p>","PeriodicalId":47707,"journal":{"name":"Journal of Medical Imaging","volume":"12 2","pages":"024008"},"PeriodicalIF":1.7000,"publicationDate":"2025-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12032765/pdf/","citationCount":"0","resultStr":"{\"title\":\"Influence of early through late fusion on pancreas segmentation from imperfectly registered multimodal magnetic resonance imaging.\",\"authors\":\"Lucas W Remedios, Han Liu, Samuel W Remedios, Lianrui Zuo, Adam M Saunders, Shunxing Bao, Yuankai Huo, Alvin C Powers, John Virostko, Bennett A Landman\",\"doi\":\"10.1117/1.JMI.12.2.024008\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<p><strong>Purpose: </strong>Combining different types of medical imaging data, through multimodal fusion, promises better segmentation of anatomical structures, such as the pancreas. Strategic implementation of multimodal fusion could improve our ability to study diseases such as diabetes. However, where to perform fusion in deep learning models is still an open question. It is unclear if there is a single best location to fuse information when analyzing pairs of imperfectly aligned images or if the optimal fusion location depends on the specific model being used. Two main challenges when using multiple imaging modalities to study the pancreas are (1) the pancreas and surrounding abdominal anatomy have a deformable structure, making it difficult to consistently align the images and (2) breathing by the individual during image collection further complicates the alignment between multimodal images. Even after using state-of-the-art deformable image registration techniques, specifically designed to align abdominal images, multimodal images of the abdomen are often not perfectly aligned. We examine how the choice of different fusion points, ranging from early in the image processing pipeline to later stages, impacts the segmentation of the pancreas on imperfectly registered multimodal magnetic resonance (MR) images.</p><p><strong>Approach: </strong>Our dataset consists of 353 pairs of T2-weighted (T2w) and T1-weighted (T1w) abdominal MR images from 163 subjects with accompanying pancreas segmentation labels drawn mainly based on the T2w images. Because the T2w images were acquired in an interleaved manner across two breath holds and the T1w images on one breath hold, there were three different breath holds impacting the alignment of each pair of images. We used deeds, a state-of-the-art deformable abdominal image registration method to align the image pairs. Then, we trained a collection of basic UNets with different fusion points, spanning from early to late layers in the model, to assess how early through late fusion influenced segmentation performance on imperfectly aligned images. To investigate whether performance differences on key fusion points are generalized to other architectures, we expanded our experiments to nnUNet.</p><p><strong>Results: </strong>The single-modality T2w baseline using a basic UNet model had a median Dice score of 0.766, whereas the same baseline on the nnUNet model achieved 0.824. For each fusion approach, we analyzed the differences in performance with Dice residuals, by subtracting the baseline score from the fusion score for each datapoint. For the basic UNet, the best fusion approach was from early/mid fusion and occurred in the middle of the encoder with a median Dice residual of <math><mrow><mo>+</mo> <mn>0.012</mn></mrow> </math> compared with the baseline. For the nnUNet, the best fusion approach was early fusion through naïve image concatenation before the model, with a median Dice residual of <math><mrow><mo>+</mo> <mn>0.004</mn></mrow> </math> compared with the baseline. After Bonferroni correction, the distributions of the Dice scores for these best fusion approaches were found to be statistically significant ( <math><mrow><mi>p</mi> <mo><</mo> <mn>0.05</mn></mrow> </math> ) via the paired Wilcoxon signed-rank test against the baseline.</p><p><strong>Conclusions: </strong>Fusion in specific blocks can improve performance, but the best blocks for fusion are model-specific, and the gains are small. In imperfectly registered datasets, fusion is a nuanced problem, with the art of design remaining vital for uncovering potential insights. Future innovation is needed to better address fusion in cases of imperfect alignment of abdominal image pairs. The code associated with this project is available here https://github.com/MASILab/influence_of_fusion_on_pancreas_segmentation.</p>\",\"PeriodicalId\":47707,\"journal\":{\"name\":\"Journal of Medical Imaging\",\"volume\":\"12 2\",\"pages\":\"024008\"},\"PeriodicalIF\":1.7000,\"publicationDate\":\"2025-03-01\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12032765/pdf/\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Journal of Medical Imaging\",\"FirstCategoryId\":\"3\",\"ListUrlMain\":\"https://doi.org/10.1117/1.JMI.12.2.024008\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"2025/4/26 0:00:00\",\"PubModel\":\"Epub\",\"JCR\":\"Q3\",\"JCRName\":\"RADIOLOGY, NUCLEAR MEDICINE & MEDICAL IMAGING\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Medical Imaging","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1117/1.JMI.12.2.024008","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2025/4/26 0:00:00","PubModel":"Epub","JCR":"Q3","JCRName":"RADIOLOGY, NUCLEAR MEDICINE & MEDICAL IMAGING","Score":null,"Total":0}
引用次数: 0

摘要

目的:结合不同类型的医学影像数据,通过多模态融合,有望更好地分割解剖结构,如胰腺。战略性地实施多模态融合可以提高我们研究糖尿病等疾病的能力。然而,在深度学习模型中如何进行融合仍然是一个悬而未决的问题。当分析不完全对齐的图像对时,是否存在一个最佳的融合位置,或者是否最佳的融合位置取决于所使用的特定模型,目前还不清楚。当使用多种成像模式来研究胰腺时,两个主要的挑战是:(1)胰腺和周围腹部解剖结构具有可变形的结构,使得难以一致地对齐图像;(2)在图像收集过程中个体的呼吸使多模式图像之间的对齐进一步复杂化。即使使用了最先进的可变形图像配准技术,专门设计用于对齐腹部图像,腹部的多模态图像通常也不能完全对齐。我们研究了不同融合点的选择,从图像处理管道的早期到后期阶段,如何影响不完全配准的多模态磁共振(MR)图像上胰腺的分割。方法:我们的数据集由163名受试者的353对t2加权(T2w)和t1加权(T1w)腹部MR图像组成,并附带主要基于T2w图像绘制的胰腺分割标签。由于T2w图像是在两个屏气中以交错的方式获得的,而T1w图像是在一个屏气中获得的,因此有三个不同的屏气会影响每对图像的对齐。我们使用了最先进的可变形腹部图像配准方法来对齐图像对。然后,我们训练了一组具有不同融合点的基本unet,从模型的早期到晚期,以评估早期和晚期融合如何影响不完全对齐图像的分割性能。为了研究关键融合点的性能差异是否可以推广到其他架构,我们将实验扩展到nnUNet。结果:使用基本UNet模型的单模态T2w基线的Dice得分中位数为0.766,而使用nnUNet模型的相同基线的Dice得分中位数为0.824。对于每种融合方法,我们通过从每个数据点的融合评分中减去基线评分,使用Dice残差分析性能差异。对于基本UNet,最好的融合方法是从早期/中期融合,发生在编码器的中间,与基线相比,Dice残差中位数为+ 0.012。对于nnUNet,最好的融合方法是在模型之前通过naïve图像拼接进行早期融合,与基线相比,Dice残差中位数为+ 0.004。在Bonferroni校正后,通过配对Wilcoxon sign -rank检验,发现这些最佳融合方法的Dice分数分布在基线上具有统计学意义(p 0.05)。结论:在特定块内融合可以提高性能,但最佳的融合块是特定模型的,并且收益很小。在不完美注册的数据集中,融合是一个微妙的问题,设计艺术对于发现潜在的见解仍然至关重要。未来的创新需要更好地解决融合的情况下,腹部图像对对齐不完美。与这个项目相关的代码可以在这里获得https://github.com/MASILab/influence_of_fusion_on_pancreas_segmentation。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
Influence of early through late fusion on pancreas segmentation from imperfectly registered multimodal magnetic resonance imaging.

Purpose: Combining different types of medical imaging data, through multimodal fusion, promises better segmentation of anatomical structures, such as the pancreas. Strategic implementation of multimodal fusion could improve our ability to study diseases such as diabetes. However, where to perform fusion in deep learning models is still an open question. It is unclear if there is a single best location to fuse information when analyzing pairs of imperfectly aligned images or if the optimal fusion location depends on the specific model being used. Two main challenges when using multiple imaging modalities to study the pancreas are (1) the pancreas and surrounding abdominal anatomy have a deformable structure, making it difficult to consistently align the images and (2) breathing by the individual during image collection further complicates the alignment between multimodal images. Even after using state-of-the-art deformable image registration techniques, specifically designed to align abdominal images, multimodal images of the abdomen are often not perfectly aligned. We examine how the choice of different fusion points, ranging from early in the image processing pipeline to later stages, impacts the segmentation of the pancreas on imperfectly registered multimodal magnetic resonance (MR) images.

Approach: Our dataset consists of 353 pairs of T2-weighted (T2w) and T1-weighted (T1w) abdominal MR images from 163 subjects with accompanying pancreas segmentation labels drawn mainly based on the T2w images. Because the T2w images were acquired in an interleaved manner across two breath holds and the T1w images on one breath hold, there were three different breath holds impacting the alignment of each pair of images. We used deeds, a state-of-the-art deformable abdominal image registration method to align the image pairs. Then, we trained a collection of basic UNets with different fusion points, spanning from early to late layers in the model, to assess how early through late fusion influenced segmentation performance on imperfectly aligned images. To investigate whether performance differences on key fusion points are generalized to other architectures, we expanded our experiments to nnUNet.

Results: The single-modality T2w baseline using a basic UNet model had a median Dice score of 0.766, whereas the same baseline on the nnUNet model achieved 0.824. For each fusion approach, we analyzed the differences in performance with Dice residuals, by subtracting the baseline score from the fusion score for each datapoint. For the basic UNet, the best fusion approach was from early/mid fusion and occurred in the middle of the encoder with a median Dice residual of + 0.012 compared with the baseline. For the nnUNet, the best fusion approach was early fusion through naïve image concatenation before the model, with a median Dice residual of + 0.004 compared with the baseline. After Bonferroni correction, the distributions of the Dice scores for these best fusion approaches were found to be statistically significant ( p < 0.05 ) via the paired Wilcoxon signed-rank test against the baseline.

Conclusions: Fusion in specific blocks can improve performance, but the best blocks for fusion are model-specific, and the gains are small. In imperfectly registered datasets, fusion is a nuanced problem, with the art of design remaining vital for uncovering potential insights. Future innovation is needed to better address fusion in cases of imperfect alignment of abdominal image pairs. The code associated with this project is available here https://github.com/MASILab/influence_of_fusion_on_pancreas_segmentation.

求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
Journal of Medical Imaging
Journal of Medical Imaging RADIOLOGY, NUCLEAR MEDICINE & MEDICAL IMAGING-
CiteScore
4.10
自引率
4.20%
发文量
0
期刊介绍: JMI covers fundamental and translational research, as well as applications, focused on medical imaging, which continue to yield physical and biomedical advancements in the early detection, diagnostics, and therapy of disease as well as in the understanding of normal. The scope of JMI includes: Imaging physics, Tomographic reconstruction algorithms (such as those in CT and MRI), Image processing and deep learning, Computer-aided diagnosis and quantitative image analysis, Visualization and modeling, Picture archiving and communications systems (PACS), Image perception and observer performance, Technology assessment, Ultrasonic imaging, Image-guided procedures, Digital pathology, Biomedical applications of biomedical imaging. JMI allows for the peer-reviewed communication and archiving of scientific developments, translational and clinical applications, reviews, and recommendations for the field.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信