Expert-centered Evaluation of Deep Learning Algorithms for Brain Tumor Segmentation.

IF 8.1 Q1 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE
Katharina V Hoebel, Christopher P Bridge, Sara Ahmed, Oluwatosin Akintola, Caroline Chung, Raymond Y Huang, Jason M Johnson, Albert Kim, K Ina Ly, Ken Chang, Jay Patel, Marco Pinho, Tracy T Batchelor, Bruce R Rosen, Elizabeth R Gerstner, Jayashree Kalpathy-Cramer
{"title":"Expert-centered Evaluation of Deep Learning Algorithms for Brain Tumor Segmentation.","authors":"Katharina V Hoebel, Christopher P Bridge, Sara Ahmed, Oluwatosin Akintola, Caroline Chung, Raymond Y Huang, Jason M Johnson, Albert Kim, K Ina Ly, Ken Chang, Jay Patel, Marco Pinho, Tracy T Batchelor, Bruce R Rosen, Elizabeth R Gerstner, Jayashree Kalpathy-Cramer","doi":"10.1148/ryai.220231","DOIUrl":null,"url":null,"abstract":"<p><p>Purpose To present results from a literature survey on practices in deep learning segmentation algorithm evaluation and perform a study on expert quality perception of brain tumor segmentation. Materials and Methods A total of 180 articles reporting on brain tumor segmentation algorithms were surveyed for the reported quality evaluation. Additionally, ratings of segmentation quality on a four-point scale were collected from medical professionals for 60 brain tumor segmentation cases. Results Of the surveyed articles, Dice score, sensitivity, and Hausdorff distance were the most popular metrics to report segmentation performance. Notably, only 2.8% of the articles included clinical experts' evaluation of segmentation quality. The experimental results revealed a low interrater agreement (Krippendorff α, 0.34) in experts' segmentation quality perception. Furthermore, the correlations between the ratings and commonly used quantitative quality metrics were low (Kendall tau between Dice score and mean rating, 0.23; Kendall tau between Hausdorff distance and mean rating, 0.51), with large variability among the experts. Conclusion The results demonstrate that quality ratings are prone to variability due to the ambiguity of tumor boundaries and individual perceptual differences, and existing metrics do not capture the clinical perception of segmentation quality. <b>Keywords:</b> Brain Tumor Segmentation, Deep Learning Algorithms, Glioblastoma, Cancer, Machine Learning Clinical trial registration nos. NCT00756106 and NCT00662506 <i>Supplemental material is available for this article.</i> © RSNA, 2023.</p>","PeriodicalId":29787,"journal":{"name":"Radiology-Artificial Intelligence","volume":null,"pages":null},"PeriodicalIF":8.1000,"publicationDate":"2024-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10831514/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Radiology-Artificial Intelligence","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1148/ryai.220231","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE","Score":null,"Total":0}
引用次数: 0

Abstract

Purpose To present results from a literature survey on practices in deep learning segmentation algorithm evaluation and perform a study on expert quality perception of brain tumor segmentation. Materials and Methods A total of 180 articles reporting on brain tumor segmentation algorithms were surveyed for the reported quality evaluation. Additionally, ratings of segmentation quality on a four-point scale were collected from medical professionals for 60 brain tumor segmentation cases. Results Of the surveyed articles, Dice score, sensitivity, and Hausdorff distance were the most popular metrics to report segmentation performance. Notably, only 2.8% of the articles included clinical experts' evaluation of segmentation quality. The experimental results revealed a low interrater agreement (Krippendorff α, 0.34) in experts' segmentation quality perception. Furthermore, the correlations between the ratings and commonly used quantitative quality metrics were low (Kendall tau between Dice score and mean rating, 0.23; Kendall tau between Hausdorff distance and mean rating, 0.51), with large variability among the experts. Conclusion The results demonstrate that quality ratings are prone to variability due to the ambiguity of tumor boundaries and individual perceptual differences, and existing metrics do not capture the clinical perception of segmentation quality. Keywords: Brain Tumor Segmentation, Deep Learning Algorithms, Glioblastoma, Cancer, Machine Learning Clinical trial registration nos. NCT00756106 and NCT00662506 Supplemental material is available for this article. © RSNA, 2023.

以专家为中心评估用于脑肿瘤分割的深度学习算法。
目的 介绍有关深度学习分割算法评估实践的文献调查结果,并对脑肿瘤分割的专家质量感知进行研究。材料与方法 共调查了 180 篇报道脑肿瘤分割算法的文章,以进行报告质量评估。此外,还收集了医学专家对 60 个脑肿瘤分割病例的分割质量的四级评分。结果 在调查的文章中,骰子得分、灵敏度和豪斯多夫距离是报告分割性能最常用的指标。值得注意的是,只有 2.8% 的文章包含临床专家对分割质量的评价。实验结果表明,专家对分割质量的感知存在较低的互评一致性(Krippendorff α,0.34)。此外,评分与常用定量质量指标之间的相关性也很低(Dice 分数与平均评分之间的 Kendall tau 值为 0.23;Hausdorff 距离与平均评分之间的 Kendall tau 值为 0.51),专家之间的差异也很大。结论 结果表明,由于肿瘤边界的模糊性和个体感知的差异,质量评分容易产生变异,现有的指标不能反映临床对分割质量的感知。关键词脑肿瘤分割 深度学习算法 胶母细胞瘤 癌症 机器学习 临床试验注册号:NCT00756106 和 NCT00756106。NCT00756106 和 NCT00662506 本文有补充材料。© RSNA, 2023.
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
CiteScore
16.20
自引率
1.00%
发文量
0
期刊介绍: Radiology: Artificial Intelligence is a bi-monthly publication that focuses on the emerging applications of machine learning and artificial intelligence in the field of imaging across various disciplines. This journal is available online and accepts multiple manuscript types, including Original Research, Technical Developments, Data Resources, Review articles, Editorials, Letters to the Editor and Replies, Special Reports, and AI in Brief.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信