Evaluation of AI-based auto-contouring tools in radiotherapy: A single-institution study

IF 2 4区医学 Q3 RADIOLOGY, NUCLEAR MEDICINE & MEDICAL IMAGING

Journal of Applied Clinical Medical Physics Pub Date : 2025-01-21 DOI:10.1002/acm2.14620

Tingyu Wang, James Tam, Thomas Chum, Cyril Tai, Deborah C. Marshall, Michael Buckstein, Jerry Liu, Sheryl Green, Robert D. Stewart, Tian Liu, Ming Chao

{"title":"Evaluation of AI-based auto-contouring tools in radiotherapy: A single-institution study","authors":"Tingyu Wang, James Tam, Thomas Chum, Cyril Tai, Deborah C. Marshall, Michael Buckstein, Jerry Liu, Sheryl Green, Robert D. Stewart, Tian Liu, Ming Chao","doi":"10.1002/acm2.14620","DOIUrl":null,"url":null,"abstract":"<div>\n \n \n <section>\n \n <h3> Background</h3>\n \n <p>Accurate delineation of organs at risk (OARs) is crucial yet time-consuming in the radiotherapy treatment planning workflow. Modern artificial intelligence (AI) technologies had made automation of OAR contouring feasible. This report details a single institution's experience in evaluating two commercial auto-contouring software tools and making well-informed decisions about their clinical adoption.</p>\n </section>\n \n <section>\n \n <h3> Methods</h3>\n \n <p>A cohort of 36 patients previously treated at our institution were selected for the software performance assessment. Fifty-eight OAR structures from seven disease sites were automatically segmented with each tool. Five radiation oncologists with different specialties qualitatively scored the automatic OAR contours’ clinical usability by a 4-level scale (0–3), termed as quality score (QS), representing from “0: not usable” to “3: directly usable for a clinic.” Additionally, quantitative comparison with clinically approved contours using Dice similarity coefficient (DSC) and the 95% Hausdorff distance (HD95) was performed in complement to QS from physicians.</p>\n </section>\n \n <section>\n \n <h3> Result</h3>\n \n <p>Software A achieved an average QS of 2.17 ± 0.69, comparable to Software B's average QS of 2.17 ± 0.72. Software B performed better with more OARs (42 vs. 37) that required minor or no modification than Software A. Major modifications were needed for 13 out of 58 automated contours from both tools. Both DSC and HD95 scores for the two tools were comparable to each other, with DSC: 0.67 ± 0.23 versus 0.66 ± 0.21 and HD95: 13.07 ± 15.84 versus 15.55 ± 18.45 for Software A and Software B, respectively. Correlation coefficients between the physician score and the quantitative metrics suggested that the contouring results from Software A aligned more closely with the physician's evaluations.</p>\n </section>\n \n <section>\n \n <h3> Conclusion</h3>\n \n <p>Based on our study, either software tool could produce clinically acceptable contours for about 65% of the OAR structures. However, further refinement is necessary for several challenging OARs to improve model performance.</p>\n </section>\n </div>","PeriodicalId":14989,"journal":{"name":"Journal of Applied Clinical Medical Physics","volume":"26 4","pages":""},"PeriodicalIF":2.0000,"publicationDate":"2025-01-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/acm2.14620","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Applied Clinical Medical Physics","FirstCategoryId":"3","ListUrlMain":"https://onlinelibrary.wiley.com/doi/10.1002/acm2.14620","RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"RADIOLOGY, NUCLEAR MEDICINE & MEDICAL IMAGING","Score":null,"Total":0}

引用次数: 0

Abstract

Background

Accurate delineation of organs at risk (OARs) is crucial yet time-consuming in the radiotherapy treatment planning workflow. Modern artificial intelligence (AI) technologies had made automation of OAR contouring feasible. This report details a single institution's experience in evaluating two commercial auto-contouring software tools and making well-informed decisions about their clinical adoption.

Methods

A cohort of 36 patients previously treated at our institution were selected for the software performance assessment. Fifty-eight OAR structures from seven disease sites were automatically segmented with each tool. Five radiation oncologists with different specialties qualitatively scored the automatic OAR contours’ clinical usability by a 4-level scale (0–3), termed as quality score (QS), representing from “0: not usable” to “3: directly usable for a clinic.” Additionally, quantitative comparison with clinically approved contours using Dice similarity coefficient (DSC) and the 95% Hausdorff distance (HD95) was performed in complement to QS from physicians.

Result

Software A achieved an average QS of 2.17 ± 0.69, comparable to Software B's average QS of 2.17 ± 0.72. Software B performed better with more OARs (42 vs. 37) that required minor or no modification than Software A. Major modifications were needed for 13 out of 58 automated contours from both tools. Both DSC and HD95 scores for the two tools were comparable to each other, with DSC: 0.67 ± 0.23 versus 0.66 ± 0.21 and HD95: 13.07 ± 15.84 versus 15.55 ± 18.45 for Software A and Software B, respectively. Correlation coefficients between the physician score and the quantitative metrics suggested that the contouring results from Software A aligned more closely with the physician's evaluations.

Conclusion

Based on our study, either software tool could produce clinically acceptable contours for about 65% of the OAR structures. However, further refinement is necessary for several challenging OARs to improve model performance.

Abstract Image

查看原文本刊更多论文

放疗中基于人工智能的自动轮廓工具的评估：一项单机构研究。

背景：在放射治疗计划工作流程中，准确描绘危险器官（OARs）是至关重要但耗时的。现代人工智能技术使桨形轮廓的自动化成为可能。本报告详细介绍了一家机构在评估两种商业自动轮廓软件工具并对其临床应用做出明智决定方面的经验。方法：选取我院既往收治的36例患者进行软件性能评估。每个工具自动分割来自7个疾病部位的58个桨叶结构。5位不同专业的放射肿瘤学家对自动OAR轮廓的临床可用性进行了定性评分，分为4个等级（0-3），称为质量评分（QS），代表从“0：不可用”到“3：直接用于临床”。此外，使用Dice相似系数（DSC）和95% Hausdorff距离（HD95）与临床批准的轮廓进行定量比较，以补充医生的QS。结果：软件A的平均QS为2.17±0.69，软件B的平均QS为2.17±0.72。与软件a相比，软件B在更多的桨（42对37）上表现得更好，这些桨需要少量修改或不需要修改。两种工具的58个自动轮廓中有13个需要进行主要修改。两种工具的DSC和HD95评分均具有可比性，软件A和软件B的DSC分别为0.67±0.23和0.66±0.21，HD95分别为13.07±15.84和15.55±18.45。医生评分和定量指标之间的相关系数表明，软件A的轮廓结果与医生的评估更接近。结论：根据我们的研究，任何一种软件工具都可以为大约65%的OAR结构产生临床可接受的轮廓。然而，对于一些具有挑战性的桨，需要进一步改进以提高模型性能。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Journal of Applied Clinical Medical Physics 医学-核医学

CiteScore

3.60

自引率

19.00%

发文量

331

审稿时长

3 months

期刊介绍： Journal of Applied Clinical Medical Physics is an international Open Access publication dedicated to clinical medical physics. JACMP welcomes original contributions dealing with all aspects of medical physics from scientists working in the clinical medical physics around the world. JACMP accepts only online submission. JACMP will publish: -Original Contributions: Peer-reviewed, investigations that represent new and significant contributions to the field. Recommended word count: up to 7500. -Review Articles: Reviews of major areas or sub-areas in the field of clinical medical physics. These articles may be of any length and are peer reviewed. -Technical Notes: These should be no longer than 3000 words, including key references. -Letters to the Editor: Comments on papers published in JACMP or on any other matters of interest to clinical medical physics. These should not be more than 1250 (including the literature) and their publication is only based on the decision of the editor, who occasionally asks experts on the merit of the contents. -Book Reviews: The editorial office solicits Book Reviews. -Announcements of Forthcoming Meetings: The Editor may provide notice of forthcoming meetings, course offerings, and other events relevant to clinical medical physics. -Parallel Opposed Editorial: We welcome topics relevant to clinical practice and medical physics profession. The contents can be controversial debate or opposed aspects of an issue. One author argues for the position and the other against. Each side of the debate contains an opening statement up to 800 words, followed by a rebuttal up to 500 words. Readers interested in participating in this series should contact the moderator with a proposed title and a short description of the topic