Chris Boyd, Timothy J. Kleinig, Joseph Dawson, Sandy Patel, Wolfgang Mayer, Eva Bezak
{"title":"CT血管造影颈动脉分割中的观察者可变性:评估可变性以设定最低临床表现","authors":"Chris Boyd, Timothy J. Kleinig, Joseph Dawson, Sandy Patel, Wolfgang Mayer, Eva Bezak","doi":"10.1111/jon.70058","DOIUrl":null,"url":null,"abstract":"<div>\n \n \n <section>\n \n <h3> Background and Purpose</h3>\n \n <p>This work evaluates carotid atherosclerosis quantification from computed tomography angiography (CTA), by novice and expert human contours. Variability sources are critically assessed to establish the minimum performance of future machine learning (ML) tools.</p>\n </section>\n \n <section>\n \n <h3> Methods</h3>\n \n <p>We analyzed extra cranial carotid lesions, with no, mild, moderate, and severe atherosclerosis (<i>n</i> = 10/group). CTA datasets of 24 patients (<i>n</i> = 6/group) were re-sampled to 2.5 mm axial thicknesses. Lumen, calcific plaque, and soft plaque were manually contoured by three expert experienced clinicians (neuroradiologist, vascular neurologist, and vascular surgeon), a medical physicist (MP), and a radiographer. Contouring was repeated several months later for intra-operator variability and again after development of a protocol. Clinicians blindly ranked each other's contours for descriptive statistical analysis.</p>\n </section>\n \n <section>\n \n <h3> Results</h3>\n \n <p>Relative to internal carotid origin, plaque began a median of 3.75 mm inferior (Interquartile Range [IQR] 0.8-7 mm), extended 18 mm superior (IQR: 13.0-29.6 mm), with a median total length of 24.4 mm (IQR: 14.7-37.4 mm). Clinicians and non-clinicians contoured lumen and calcific plaque similarly (dice similarity coefficient [DSC]: 0.87/0.62 respectively), but varied greater for soft plaque (DSC: 0.21). Neuroradiologist contours were consistently smaller, from approaching the partial-volume artifact conservatively. Clinicians favored their own contours, most pronouncedly the neuroradiologist (standard deviation: 0.00). Establishing a contouring protocol was not found to improve the agreement between clinicians.</p>\n </section>\n \n <section>\n \n <h3> Conclusions</h3>\n \n <p>CTA carotid pathology contouring inherently has limited clinician agreement due to small structure size and poor contrast. The reference-contour datasets produced by experienced clinicians are prone to inter-and intra-variability which must be carefully considered to ensure ML models developed from such datasets are not fatally flawed.</p>\n </section>\n </div>","PeriodicalId":16399,"journal":{"name":"Journal of Neuroimaging","volume":"35 3","pages":""},"PeriodicalIF":2.3000,"publicationDate":"2025-06-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1111/jon.70058","citationCount":"0","resultStr":"{\"title\":\"Observer Variability in CT Angiography Carotid Segmentation: Assessing Variability to Set Minimum Clinical Performance\",\"authors\":\"Chris Boyd, Timothy J. Kleinig, Joseph Dawson, Sandy Patel, Wolfgang Mayer, Eva Bezak\",\"doi\":\"10.1111/jon.70058\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<div>\\n \\n \\n <section>\\n \\n <h3> Background and Purpose</h3>\\n \\n <p>This work evaluates carotid atherosclerosis quantification from computed tomography angiography (CTA), by novice and expert human contours. Variability sources are critically assessed to establish the minimum performance of future machine learning (ML) tools.</p>\\n </section>\\n \\n <section>\\n \\n <h3> Methods</h3>\\n \\n <p>We analyzed extra cranial carotid lesions, with no, mild, moderate, and severe atherosclerosis (<i>n</i> = 10/group). CTA datasets of 24 patients (<i>n</i> = 6/group) were re-sampled to 2.5 mm axial thicknesses. Lumen, calcific plaque, and soft plaque were manually contoured by three expert experienced clinicians (neuroradiologist, vascular neurologist, and vascular surgeon), a medical physicist (MP), and a radiographer. Contouring was repeated several months later for intra-operator variability and again after development of a protocol. Clinicians blindly ranked each other's contours for descriptive statistical analysis.</p>\\n </section>\\n \\n <section>\\n \\n <h3> Results</h3>\\n \\n <p>Relative to internal carotid origin, plaque began a median of 3.75 mm inferior (Interquartile Range [IQR] 0.8-7 mm), extended 18 mm superior (IQR: 13.0-29.6 mm), with a median total length of 24.4 mm (IQR: 14.7-37.4 mm). Clinicians and non-clinicians contoured lumen and calcific plaque similarly (dice similarity coefficient [DSC]: 0.87/0.62 respectively), but varied greater for soft plaque (DSC: 0.21). Neuroradiologist contours were consistently smaller, from approaching the partial-volume artifact conservatively. Clinicians favored their own contours, most pronouncedly the neuroradiologist (standard deviation: 0.00). Establishing a contouring protocol was not found to improve the agreement between clinicians.</p>\\n </section>\\n \\n <section>\\n \\n <h3> Conclusions</h3>\\n \\n <p>CTA carotid pathology contouring inherently has limited clinician agreement due to small structure size and poor contrast. The reference-contour datasets produced by experienced clinicians are prone to inter-and intra-variability which must be carefully considered to ensure ML models developed from such datasets are not fatally flawed.</p>\\n </section>\\n </div>\",\"PeriodicalId\":16399,\"journal\":{\"name\":\"Journal of Neuroimaging\",\"volume\":\"35 3\",\"pages\":\"\"},\"PeriodicalIF\":2.3000,\"publicationDate\":\"2025-06-04\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"https://onlinelibrary.wiley.com/doi/epdf/10.1111/jon.70058\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Journal of Neuroimaging\",\"FirstCategoryId\":\"3\",\"ListUrlMain\":\"https://onlinelibrary.wiley.com/doi/10.1111/jon.70058\",\"RegionNum\":4,\"RegionCategory\":\"医学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q3\",\"JCRName\":\"CLINICAL NEUROLOGY\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Neuroimaging","FirstCategoryId":"3","ListUrlMain":"https://onlinelibrary.wiley.com/doi/10.1111/jon.70058","RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"CLINICAL NEUROLOGY","Score":null,"Total":0}
引用次数: 0
摘要
背景和目的本研究评估了由新手和专家进行的颈动脉粥样硬化计算机断层血管造影(CTA)量化。对可变性源进行严格评估,以建立未来机器学习(ML)工具的最低性能。方法我们分析颅外颈动脉病变,无、轻度、中度和重度动脉粥样硬化(n = 10/组)。24例患者(n = 6/组)的CTA数据集重新采样至2.5 mm轴向厚度。管腔、钙化斑块和软斑块由三位经验丰富的专家临床医生(神经放射学家、血管神经学家和血管外科医生)、一名医学物理学家(MP)和一名放射技师手工绘制。几个月后重复轮廓,以确定操作者内部的可变性,并在制定协议后再次进行轮廓。临床医生盲目地对彼此的轮廓进行排序,以进行描述性统计分析。结果相对于颈内动脉起源,斑块开始时中位数为3.75 mm(四分位间距[IQR] 0.8-7 mm),延伸至18 mm (IQR: 13.0-29.6 mm),总中位数为24.4 mm (IQR: 14.7-37.4 mm)。临床医生和非临床医生对管腔和钙化斑块的轮廓相似(骰子相似系数[DSC]分别为0.87/0.62),但软斑块的差异更大(DSC: 0.21)。神经放射学家的轮廓一直较小,因为保守地接近部分体积伪影。临床医生喜欢他们自己的轮廓,最明显的是神经放射学家(标准差:0.00)。建立一个轮廓协议没有发现提高临床医生之间的协议。结论CTA颈动脉病理轮廓由于结构尺寸小,造影剂差,临床一致性有限。由经验丰富的临床医生生成的参考轮廓数据集容易出现内部和内部变异,必须仔细考虑,以确保从这些数据集开发的ML模型不会存在致命缺陷。
Observer Variability in CT Angiography Carotid Segmentation: Assessing Variability to Set Minimum Clinical Performance
Background and Purpose
This work evaluates carotid atherosclerosis quantification from computed tomography angiography (CTA), by novice and expert human contours. Variability sources are critically assessed to establish the minimum performance of future machine learning (ML) tools.
Methods
We analyzed extra cranial carotid lesions, with no, mild, moderate, and severe atherosclerosis (n = 10/group). CTA datasets of 24 patients (n = 6/group) were re-sampled to 2.5 mm axial thicknesses. Lumen, calcific plaque, and soft plaque were manually contoured by three expert experienced clinicians (neuroradiologist, vascular neurologist, and vascular surgeon), a medical physicist (MP), and a radiographer. Contouring was repeated several months later for intra-operator variability and again after development of a protocol. Clinicians blindly ranked each other's contours for descriptive statistical analysis.
Results
Relative to internal carotid origin, plaque began a median of 3.75 mm inferior (Interquartile Range [IQR] 0.8-7 mm), extended 18 mm superior (IQR: 13.0-29.6 mm), with a median total length of 24.4 mm (IQR: 14.7-37.4 mm). Clinicians and non-clinicians contoured lumen and calcific plaque similarly (dice similarity coefficient [DSC]: 0.87/0.62 respectively), but varied greater for soft plaque (DSC: 0.21). Neuroradiologist contours were consistently smaller, from approaching the partial-volume artifact conservatively. Clinicians favored their own contours, most pronouncedly the neuroradiologist (standard deviation: 0.00). Establishing a contouring protocol was not found to improve the agreement between clinicians.
Conclusions
CTA carotid pathology contouring inherently has limited clinician agreement due to small structure size and poor contrast. The reference-contour datasets produced by experienced clinicians are prone to inter-and intra-variability which must be carefully considered to ensure ML models developed from such datasets are not fatally flawed.
期刊介绍:
Start reading the Journal of Neuroimaging to learn the latest neurological imaging techniques. The peer-reviewed research is written in a practical clinical context, giving you the information you need on:
MRI
CT
Carotid Ultrasound and TCD
SPECT
PET
Endovascular Surgical Neuroradiology
Functional MRI
Xenon CT
and other new and upcoming neuroscientific modalities.The Journal of Neuroimaging addresses the full spectrum of human nervous system disease, including stroke, neoplasia, degenerating and demyelinating disease, epilepsy, tumors, lesions, infectious disease, cerebral vascular arterial diseases, toxic-metabolic disease, psychoses, dementias, heredo-familial disease, and trauma.Offering original research, review articles, case reports, neuroimaging CPCs, and evaluations of instruments and technology relevant to the nervous system, the Journal of Neuroimaging focuses on useful clinical developments and applications, tested techniques and interpretations, patient care, diagnostics, and therapeutics. Start reading today!