An approach for determining the reliability of manual and digital scoring of sleep stages.

IF 5.3 2区 医学 Q1 CLINICAL NEUROLOGY
Sleep Pub Date : 2023-11-08 DOI:10.1093/sleep/zsad248
Bethany Gerardy, Samuel T Kuna, Allan Pack, Clete A Kushida, James K Walsh, Bethany Staley, Grace W Pien, Magdy Younes
{"title":"An approach for determining the reliability of manual and digital scoring of sleep stages.","authors":"Bethany Gerardy, Samuel T Kuna, Allan Pack, Clete A Kushida, James K Walsh, Bethany Staley, Grace W Pien, Magdy Younes","doi":"10.1093/sleep/zsad248","DOIUrl":null,"url":null,"abstract":"<p><strong>Study objectives: </strong>Inter-scorer variability in sleep staging is largely due to equivocal epochs that contain features of more than one stage. We propose an approach that recognizes the existence of equivocal epochs and evaluates scorers accordingly.</p><p><strong>Methods: </strong>Epoch-by-epoch staging was performed on 70 polysomnograms by six qualified technologists and by a digital system (Michele Sleep Scoring [MSS]). Probability that epochs assigned the same stage by only two of the six technologists (minority score) resulted from random occurrence of two errors was calculated and found to be <5%, thereby indicating that the stage assigned is an acceptable variant for the epoch. Acceptable stages were identified in each epoch as stages assigned by at least two technologists. Percent agreement between each technologist and the other five technologists, acting as judges, was determined. Agreement was considered to exist if the stage assigned by the tested scorer was one of the acceptable stages for the epoch. Stage assigned by MSS was likewise considered in agreement if included in the acceptable stages made by the technologists.</p><p><strong>Results: </strong>Agreement of technologists tested against five qualified judges increased from 80.8% (range 70.5%-86.4% among technologists) when using the majority rule, to 96.1 (89.8%-98.5%) by the proposed approach. Agreement between unedited MSS and same judges was 90.0% and increased to 92.1% after brief editing.</p><p><strong>Conclusions: </strong>Accounting for equivocal epochs provides a more accurate estimate of a scorer's (human or digital) competence in scoring sleep stages and reduces inter-scorer disagreements. The proposed approach can be implemented in sleep-scoring training and accreditation programs.</p>","PeriodicalId":49514,"journal":{"name":"Sleep","volume":null,"pages":null},"PeriodicalIF":5.3000,"publicationDate":"2023-11-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Sleep","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1093/sleep/zsad248","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"CLINICAL NEUROLOGY","Score":null,"Total":0}
引用次数: 0

Abstract

Study objectives: Inter-scorer variability in sleep staging is largely due to equivocal epochs that contain features of more than one stage. We propose an approach that recognizes the existence of equivocal epochs and evaluates scorers accordingly.

Methods: Epoch-by-epoch staging was performed on 70 polysomnograms by six qualified technologists and by a digital system (Michele Sleep Scoring [MSS]). Probability that epochs assigned the same stage by only two of the six technologists (minority score) resulted from random occurrence of two errors was calculated and found to be <5%, thereby indicating that the stage assigned is an acceptable variant for the epoch. Acceptable stages were identified in each epoch as stages assigned by at least two technologists. Percent agreement between each technologist and the other five technologists, acting as judges, was determined. Agreement was considered to exist if the stage assigned by the tested scorer was one of the acceptable stages for the epoch. Stage assigned by MSS was likewise considered in agreement if included in the acceptable stages made by the technologists.

Results: Agreement of technologists tested against five qualified judges increased from 80.8% (range 70.5%-86.4% among technologists) when using the majority rule, to 96.1 (89.8%-98.5%) by the proposed approach. Agreement between unedited MSS and same judges was 90.0% and increased to 92.1% after brief editing.

Conclusions: Accounting for equivocal epochs provides a more accurate estimate of a scorer's (human or digital) competence in scoring sleep stages and reduces inter-scorer disagreements. The proposed approach can be implemented in sleep-scoring training and accreditation programs.

一种确定人工和数字睡眠阶段评分可靠性的方法。
研究目的:评分者之间睡眠阶段的差异很大程度上是由于包含多个阶段特征的模糊时期。我们提出了一种方法,承认存在模棱两可的时代,并相应地评估得分。方法:采用Michele睡眠评分(MSS)数字系统,由6名专业技术人员对70张多导睡眠图进行分期。计算了由于随机出现两个错误而导致6名技术人员中只有2名(少数分数)分配同一阶段的概率,并发现结果:对5名合格裁判进行测试的技术人员的同意度从使用多数决定原则时的80.8%(技术人员的范围为70.5%-86.4%)增加到使用建议方法时的96.1(89.8%-98.5%)。未编辑的MSS与同一法官的一致性为90.0%,经过简短编辑后提高到92.1%。结论:考虑模棱两可的时代,可以更准确地估计计分员(人或数字)对睡眠阶段进行评分的能力,并减少计分员之间的分歧。所提出的方法可以在睡眠评分培训和认证项目中实施。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
Sleep
Sleep 医学-临床神经学
CiteScore
10.10
自引率
10.70%
发文量
1134
审稿时长
3 months
期刊介绍: SLEEP® publishes findings from studies conducted at any level of analysis, including: Genes Molecules Cells Physiology Neural systems and circuits Behavior and cognition Self-report SLEEP® publishes articles that use a wide variety of scientific approaches and address a broad range of topics. These may include, but are not limited to: Basic and neuroscience studies of sleep and circadian mechanisms In vitro and animal models of sleep, circadian rhythms, and human disorders Pre-clinical human investigations, including the measurement and manipulation of sleep and circadian rhythms Studies in clinical or population samples. These may address factors influencing sleep and circadian rhythms (e.g., development and aging, and social and environmental influences) and relationships between sleep, circadian rhythms, health, and disease Clinical trials, epidemiology studies, implementation, and dissemination research.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信