CT血管造影颈动脉分割中的观察者可变性:评估可变性以设定最低临床表现

IF 2.3 4区 医学 Q3 CLINICAL NEUROLOGY
Chris Boyd, Timothy J. Kleinig, Joseph Dawson, Sandy Patel, Wolfgang Mayer, Eva Bezak
{"title":"CT血管造影颈动脉分割中的观察者可变性:评估可变性以设定最低临床表现","authors":"Chris Boyd,&nbsp;Timothy J. Kleinig,&nbsp;Joseph Dawson,&nbsp;Sandy Patel,&nbsp;Wolfgang Mayer,&nbsp;Eva Bezak","doi":"10.1111/jon.70058","DOIUrl":null,"url":null,"abstract":"<div>\n \n \n <section>\n \n <h3> Background and Purpose</h3>\n \n <p>This work evaluates carotid atherosclerosis quantification from computed tomography angiography (CTA), by novice and expert human contours. Variability sources are critically assessed to establish the minimum performance of future machine learning (ML) tools.</p>\n </section>\n \n <section>\n \n <h3> Methods</h3>\n \n <p>We analyzed extra cranial carotid lesions, with no, mild, moderate, and severe atherosclerosis (<i>n</i> = 10/group). CTA datasets of 24 patients (<i>n</i> = 6/group) were re-sampled to 2.5 mm axial thicknesses. Lumen, calcific plaque, and soft plaque were manually contoured by three expert experienced clinicians (neuroradiologist, vascular neurologist, and vascular surgeon), a medical physicist (MP), and a radiographer. Contouring was repeated several months later for intra-operator variability and again after development of a protocol. Clinicians blindly ranked each other's contours for descriptive statistical analysis.</p>\n </section>\n \n <section>\n \n <h3> Results</h3>\n \n <p>Relative to internal carotid origin, plaque began a median of 3.75 mm inferior (Interquartile Range [IQR] 0.8-7 mm), extended 18 mm superior (IQR: 13.0-29.6 mm), with a median total length of 24.4 mm (IQR: 14.7-37.4 mm). Clinicians and non-clinicians contoured lumen and calcific plaque similarly (dice similarity coefficient [DSC]: 0.87/0.62 respectively), but varied greater for soft plaque (DSC: 0.21). Neuroradiologist contours were consistently smaller, from approaching the partial-volume artifact conservatively. Clinicians favored their own contours, most pronouncedly the neuroradiologist (standard deviation: 0.00). Establishing a contouring protocol was not found to improve the agreement between clinicians.</p>\n </section>\n \n <section>\n \n <h3> Conclusions</h3>\n \n <p>CTA carotid pathology contouring inherently has limited clinician agreement due to small structure size and poor contrast. The reference-contour datasets produced by experienced clinicians are prone to inter-and intra-variability which must be carefully considered to ensure ML models developed from such datasets are not fatally flawed.</p>\n </section>\n </div>","PeriodicalId":16399,"journal":{"name":"Journal of Neuroimaging","volume":"35 3","pages":""},"PeriodicalIF":2.3000,"publicationDate":"2025-06-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1111/jon.70058","citationCount":"0","resultStr":"{\"title\":\"Observer Variability in CT Angiography Carotid Segmentation: Assessing Variability to Set Minimum Clinical Performance\",\"authors\":\"Chris Boyd,&nbsp;Timothy J. Kleinig,&nbsp;Joseph Dawson,&nbsp;Sandy Patel,&nbsp;Wolfgang Mayer,&nbsp;Eva Bezak\",\"doi\":\"10.1111/jon.70058\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<div>\\n \\n \\n <section>\\n \\n <h3> Background and Purpose</h3>\\n \\n <p>This work evaluates carotid atherosclerosis quantification from computed tomography angiography (CTA), by novice and expert human contours. Variability sources are critically assessed to establish the minimum performance of future machine learning (ML) tools.</p>\\n </section>\\n \\n <section>\\n \\n <h3> Methods</h3>\\n \\n <p>We analyzed extra cranial carotid lesions, with no, mild, moderate, and severe atherosclerosis (<i>n</i> = 10/group). CTA datasets of 24 patients (<i>n</i> = 6/group) were re-sampled to 2.5 mm axial thicknesses. Lumen, calcific plaque, and soft plaque were manually contoured by three expert experienced clinicians (neuroradiologist, vascular neurologist, and vascular surgeon), a medical physicist (MP), and a radiographer. Contouring was repeated several months later for intra-operator variability and again after development of a protocol. Clinicians blindly ranked each other's contours for descriptive statistical analysis.</p>\\n </section>\\n \\n <section>\\n \\n <h3> Results</h3>\\n \\n <p>Relative to internal carotid origin, plaque began a median of 3.75 mm inferior (Interquartile Range [IQR] 0.8-7 mm), extended 18 mm superior (IQR: 13.0-29.6 mm), with a median total length of 24.4 mm (IQR: 14.7-37.4 mm). Clinicians and non-clinicians contoured lumen and calcific plaque similarly (dice similarity coefficient [DSC]: 0.87/0.62 respectively), but varied greater for soft plaque (DSC: 0.21). Neuroradiologist contours were consistently smaller, from approaching the partial-volume artifact conservatively. Clinicians favored their own contours, most pronouncedly the neuroradiologist (standard deviation: 0.00). Establishing a contouring protocol was not found to improve the agreement between clinicians.</p>\\n </section>\\n \\n <section>\\n \\n <h3> Conclusions</h3>\\n \\n <p>CTA carotid pathology contouring inherently has limited clinician agreement due to small structure size and poor contrast. The reference-contour datasets produced by experienced clinicians are prone to inter-and intra-variability which must be carefully considered to ensure ML models developed from such datasets are not fatally flawed.</p>\\n </section>\\n </div>\",\"PeriodicalId\":16399,\"journal\":{\"name\":\"Journal of Neuroimaging\",\"volume\":\"35 3\",\"pages\":\"\"},\"PeriodicalIF\":2.3000,\"publicationDate\":\"2025-06-04\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"https://onlinelibrary.wiley.com/doi/epdf/10.1111/jon.70058\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Journal of Neuroimaging\",\"FirstCategoryId\":\"3\",\"ListUrlMain\":\"https://onlinelibrary.wiley.com/doi/10.1111/jon.70058\",\"RegionNum\":4,\"RegionCategory\":\"医学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q3\",\"JCRName\":\"CLINICAL NEUROLOGY\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Neuroimaging","FirstCategoryId":"3","ListUrlMain":"https://onlinelibrary.wiley.com/doi/10.1111/jon.70058","RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"CLINICAL NEUROLOGY","Score":null,"Total":0}
引用次数: 0

摘要

背景和目的本研究评估了由新手和专家进行的颈动脉粥样硬化计算机断层血管造影(CTA)量化。对可变性源进行严格评估,以建立未来机器学习(ML)工具的最低性能。方法我们分析颅外颈动脉病变,无、轻度、中度和重度动脉粥样硬化(n = 10/组)。24例患者(n = 6/组)的CTA数据集重新采样至2.5 mm轴向厚度。管腔、钙化斑块和软斑块由三位经验丰富的专家临床医生(神经放射学家、血管神经学家和血管外科医生)、一名医学物理学家(MP)和一名放射技师手工绘制。几个月后重复轮廓,以确定操作者内部的可变性,并在制定协议后再次进行轮廓。临床医生盲目地对彼此的轮廓进行排序,以进行描述性统计分析。结果相对于颈内动脉起源,斑块开始时中位数为3.75 mm(四分位间距[IQR] 0.8-7 mm),延伸至18 mm (IQR: 13.0-29.6 mm),总中位数为24.4 mm (IQR: 14.7-37.4 mm)。临床医生和非临床医生对管腔和钙化斑块的轮廓相似(骰子相似系数[DSC]分别为0.87/0.62),但软斑块的差异更大(DSC: 0.21)。神经放射学家的轮廓一直较小,因为保守地接近部分体积伪影。临床医生喜欢他们自己的轮廓,最明显的是神经放射学家(标准差:0.00)。建立一个轮廓协议没有发现提高临床医生之间的协议。结论CTA颈动脉病理轮廓由于结构尺寸小,造影剂差,临床一致性有限。由经验丰富的临床医生生成的参考轮廓数据集容易出现内部和内部变异,必须仔细考虑,以确保从这些数据集开发的ML模型不会存在致命缺陷。
本文章由计算机程序翻译,如有差异,请以英文原文为准。

Observer Variability in CT Angiography Carotid Segmentation: Assessing Variability to Set Minimum Clinical Performance

Observer Variability in CT Angiography Carotid Segmentation: Assessing Variability to Set Minimum Clinical Performance

Background and Purpose

This work evaluates carotid atherosclerosis quantification from computed tomography angiography (CTA), by novice and expert human contours. Variability sources are critically assessed to establish the minimum performance of future machine learning (ML) tools.

Methods

We analyzed extra cranial carotid lesions, with no, mild, moderate, and severe atherosclerosis (n = 10/group). CTA datasets of 24 patients (n = 6/group) were re-sampled to 2.5 mm axial thicknesses. Lumen, calcific plaque, and soft plaque were manually contoured by three expert experienced clinicians (neuroradiologist, vascular neurologist, and vascular surgeon), a medical physicist (MP), and a radiographer. Contouring was repeated several months later for intra-operator variability and again after development of a protocol. Clinicians blindly ranked each other's contours for descriptive statistical analysis.

Results

Relative to internal carotid origin, plaque began a median of 3.75 mm inferior (Interquartile Range [IQR] 0.8-7 mm), extended 18 mm superior (IQR: 13.0-29.6 mm), with a median total length of 24.4 mm (IQR: 14.7-37.4 mm). Clinicians and non-clinicians contoured lumen and calcific plaque similarly (dice similarity coefficient [DSC]: 0.87/0.62 respectively), but varied greater for soft plaque (DSC: 0.21). Neuroradiologist contours were consistently smaller, from approaching the partial-volume artifact conservatively. Clinicians favored their own contours, most pronouncedly the neuroradiologist (standard deviation: 0.00). Establishing a contouring protocol was not found to improve the agreement between clinicians.

Conclusions

CTA carotid pathology contouring inherently has limited clinician agreement due to small structure size and poor contrast. The reference-contour datasets produced by experienced clinicians are prone to inter-and intra-variability which must be carefully considered to ensure ML models developed from such datasets are not fatally flawed.

求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
Journal of Neuroimaging
Journal of Neuroimaging 医学-核医学
CiteScore
4.70
自引率
0.00%
发文量
117
审稿时长
6-12 weeks
期刊介绍: Start reading the Journal of Neuroimaging to learn the latest neurological imaging techniques. The peer-reviewed research is written in a practical clinical context, giving you the information you need on: MRI CT Carotid Ultrasound and TCD SPECT PET Endovascular Surgical Neuroradiology Functional MRI Xenon CT and other new and upcoming neuroscientific modalities.The Journal of Neuroimaging addresses the full spectrum of human nervous system disease, including stroke, neoplasia, degenerating and demyelinating disease, epilepsy, tumors, lesions, infectious disease, cerebral vascular arterial diseases, toxic-metabolic disease, psychoses, dementias, heredo-familial disease, and trauma.Offering original research, review articles, case reports, neuroimaging CPCs, and evaluations of instruments and technology relevant to the nervous system, the Journal of Neuroimaging focuses on useful clinical developments and applications, tested techniques and interpretations, patient care, diagnostics, and therapeutics. Start reading today!
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信