Real-world performance evaluation of commercial autocontouring software for head and neck cancer radiotherapy.

Abstract

Objectives: To comprehensively evaluate the ability of ART-Plan (autocontouring software using artificial intelligence [AI]) to contour head and neck cancer (HNC) radiotherapy structures (organs-at-risk and elective nodal volumes) in a real-world setting, as required by NICE guidelines.

Methods: Retrospective evaluation (n = 60) compared clinically used contours to AI-contours, using volumetric dice similarity coefficient (VDSC) and blinded radiation oncologist (RO) contour preference assessment. AI-contours were then generated prospectively for HNC radiotherapy patients (n = 61), before review by ROs. ROs recorded qualitative scoring and review/editing time. Geometric and dosimetric comparison of AI-contours and final contours was undertaken. Correlation coefficients between all metrics were calculated.

Results: Retrospective median VDSC varied widely for different structures (0.23-0.88). 31.4% blinded contour assessments preferred clinician-generated contours, 32.9% preferred AI-generated contours, 35.7% saw no significant difference. Prospective evaluation showed AI-contour yielded significant time-saving in reviewing/editing for all structures compared to manual contouring. Qualitative scores demonstrated most structures had median scoring ≥4 (indicating no editing required). Geometric metrics showed high similarity for all structures except larynx. Dosimetric evaluation demonstrated clinically significant dose differences for larynx and elective nodal volumes. Strong correlation was seen between qualitative scoring and all geometric metrics.

Conclusions: AI-contours showed excellent qualitative performance and facilitated time-saving. Protocol differences exist between commercial AI-solutions and implementing centres. Ongoing final clinician review of AI-contours remains essential.

Advances in knowledge: This is the first study demonstrating ART-Plan's capability for excellent qualitative ratings and significant time-savings for HNC radiotherapy contouring in a real-world workflow setting. 5-point Likert-scale qualitative scoring correlates strongly with geometric metrics.