Observer Variability in CT Angiography Carotid Segmentation: Assessing Variability to Set Minimum Clinical Performance

Abstract

Background and Purpose

This work evaluates carotid atherosclerosis quantification from computed tomography angiography (CTA), by novice and expert human contours. Variability sources are critically assessed to establish the minimum performance of future machine learning (ML) tools.

Methods

We analyzed extra cranial carotid lesions, with no, mild, moderate, and severe atherosclerosis (n = 10/group). CTA datasets of 24 patients (n = 6/group) were re-sampled to 2.5 mm axial thicknesses. Lumen, calcific plaque, and soft plaque were manually contoured by three expert experienced clinicians (neuroradiologist, vascular neurologist, and vascular surgeon), a medical physicist (MP), and a radiographer. Contouring was repeated several months later for intra-operator variability and again after development of a protocol. Clinicians blindly ranked each other's contours for descriptive statistical analysis.

Results

Relative to internal carotid origin, plaque began a median of 3.75 mm inferior (Interquartile Range [IQR] 0.8-7 mm), extended 18 mm superior (IQR: 13.0-29.6 mm), with a median total length of 24.4 mm (IQR: 14.7-37.4 mm). Clinicians and non-clinicians contoured lumen and calcific plaque similarly (dice similarity coefficient [DSC]: 0.87/0.62 respectively), but varied greater for soft plaque (DSC: 0.21). Neuroradiologist contours were consistently smaller, from approaching the partial-volume artifact conservatively. Clinicians favored their own contours, most pronouncedly the neuroradiologist (standard deviation: 0.00). Establishing a contouring protocol was not found to improve the agreement between clinicians.

Conclusions

CTA carotid pathology contouring inherently has limited clinician agreement due to small structure size and poor contrast. The reference-contour datasets produced by experienced clinicians are prone to inter-and intra-variability which must be carefully considered to ensure ML models developed from such datasets are not fatally flawed.