Expert-centered Evaluation of Deep Learning Algorithms for Brain Tumor Segmentation.
IF 8.1
Q1 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE
Katharina V Hoebel, Christopher P Bridge, Sara Ahmed, Oluwatosin Akintola, Caroline Chung, Raymond Y Huang, Jason M Johnson, Albert Kim, K Ina Ly, Ken Chang, Jay Patel, Marco Pinho, Tracy T Batchelor, Bruce R Rosen, Elizabeth R Gerstner, Jayashree Kalpathy-Cramer
下载PDF
{"title":"Expert-centered Evaluation of Deep Learning Algorithms for Brain Tumor Segmentation.","authors":"Katharina V Hoebel, Christopher P Bridge, Sara Ahmed, Oluwatosin Akintola, Caroline Chung, Raymond Y Huang, Jason M Johnson, Albert Kim, K Ina Ly, Ken Chang, Jay Patel, Marco Pinho, Tracy T Batchelor, Bruce R Rosen, Elizabeth R Gerstner, Jayashree Kalpathy-Cramer","doi":"10.1148/ryai.220231","DOIUrl":null,"url":null,"abstract":"<p><p>Purpose To present results from a literature survey on practices in deep learning segmentation algorithm evaluation and perform a study on expert quality perception of brain tumor segmentation. Materials and Methods A total of 180 articles reporting on brain tumor segmentation algorithms were surveyed for the reported quality evaluation. Additionally, ratings of segmentation quality on a four-point scale were collected from medical professionals for 60 brain tumor segmentation cases. Results Of the surveyed articles, Dice score, sensitivity, and Hausdorff distance were the most popular metrics to report segmentation performance. Notably, only 2.8% of the articles included clinical experts' evaluation of segmentation quality. The experimental results revealed a low interrater agreement (Krippendorff α, 0.34) in experts' segmentation quality perception. Furthermore, the correlations between the ratings and commonly used quantitative quality metrics were low (Kendall tau between Dice score and mean rating, 0.23; Kendall tau between Hausdorff distance and mean rating, 0.51), with large variability among the experts. Conclusion The results demonstrate that quality ratings are prone to variability due to the ambiguity of tumor boundaries and individual perceptual differences, and existing metrics do not capture the clinical perception of segmentation quality. <b>Keywords:</b> Brain Tumor Segmentation, Deep Learning Algorithms, Glioblastoma, Cancer, Machine Learning Clinical trial registration nos. NCT00756106 and NCT00662506 <i>Supplemental material is available for this article.</i> © RSNA, 2023.</p>","PeriodicalId":29787,"journal":{"name":"Radiology-Artificial Intelligence","volume":null,"pages":null},"PeriodicalIF":8.1000,"publicationDate":"2024-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10831514/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Radiology-Artificial Intelligence","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1148/ryai.220231","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE","Score":null,"Total":0}
引用次数: 0
引用
批量引用
Abstract
Purpose To present results from a literature survey on practices in deep learning segmentation algorithm evaluation and perform a study on expert quality perception of brain tumor segmentation. Materials and Methods A total of 180 articles reporting on brain tumor segmentation algorithms were surveyed for the reported quality evaluation. Additionally, ratings of segmentation quality on a four-point scale were collected from medical professionals for 60 brain tumor segmentation cases. Results Of the surveyed articles, Dice score, sensitivity, and Hausdorff distance were the most popular metrics to report segmentation performance. Notably, only 2.8% of the articles included clinical experts' evaluation of segmentation quality. The experimental results revealed a low interrater agreement (Krippendorff α, 0.34) in experts' segmentation quality perception. Furthermore, the correlations between the ratings and commonly used quantitative quality metrics were low (Kendall tau between Dice score and mean rating, 0.23; Kendall tau between Hausdorff distance and mean rating, 0.51), with large variability among the experts. Conclusion The results demonstrate that quality ratings are prone to variability due to the ambiguity of tumor boundaries and individual perceptual differences, and existing metrics do not capture the clinical perception of segmentation quality. Keywords: Brain Tumor Segmentation, Deep Learning Algorithms, Glioblastoma, Cancer, Machine Learning Clinical trial registration nos. NCT00756106 and NCT00662506 Supplemental material is available for this article. © RSNA, 2023.
以专家为中心评估用于脑肿瘤分割的深度学习算法。
目的 介绍有关深度学习分割算法评估实践的文献调查结果,并对脑肿瘤分割的专家质量感知进行研究。材料与方法 共调查了 180 篇报道脑肿瘤分割算法的文章,以进行报告质量评估。此外,还收集了医学专家对 60 个脑肿瘤分割病例的分割质量的四级评分。结果 在调查的文章中,骰子得分、灵敏度和豪斯多夫距离是报告分割性能最常用的指标。值得注意的是,只有 2.8% 的文章包含临床专家对分割质量的评价。实验结果表明,专家对分割质量的感知存在较低的互评一致性(Krippendorff α,0.34)。此外,评分与常用定量质量指标之间的相关性也很低(Dice 分数与平均评分之间的 Kendall tau 值为 0.23;Hausdorff 距离与平均评分之间的 Kendall tau 值为 0.51),专家之间的差异也很大。结论 结果表明,由于肿瘤边界的模糊性和个体感知的差异,质量评分容易产生变异,现有的指标不能反映临床对分割质量的感知。关键词脑肿瘤分割 深度学习算法 胶母细胞瘤 癌症 机器学习 临床试验注册号:NCT00756106 和 NCT00756106。NCT00756106 和 NCT00662506 本文有补充材料。© RSNA, 2023.
本文章由计算机程序翻译,如有差异,请以英文原文为准。
来源期刊
期刊介绍:
Radiology: Artificial Intelligence is a bi-monthly publication that focuses on the emerging applications of machine learning and artificial intelligence in the field of imaging across various disciplines. This journal is available online and accepts multiple manuscript types, including Original Research, Technical Developments, Data Resources, Review articles, Editorials, Letters to the Editor and Replies, Special Reports, and AI in Brief.