Bethany Gerardy, Samuel T Kuna, Allan Pack, Clete A Kushida, James K Walsh, Bethany Staley, Grace W Pien, Magdy Younes
{"title":"An approach for determining the reliability of manual and digital scoring of sleep stages.","authors":"Bethany Gerardy, Samuel T Kuna, Allan Pack, Clete A Kushida, James K Walsh, Bethany Staley, Grace W Pien, Magdy Younes","doi":"10.1093/sleep/zsad248","DOIUrl":null,"url":null,"abstract":"<p><strong>Study objectives: </strong>Inter-scorer variability in sleep staging is largely due to equivocal epochs that contain features of more than one stage. We propose an approach that recognizes the existence of equivocal epochs and evaluates scorers accordingly.</p><p><strong>Methods: </strong>Epoch-by-epoch staging was performed on 70 polysomnograms by six qualified technologists and by a digital system (Michele Sleep Scoring [MSS]). Probability that epochs assigned the same stage by only two of the six technologists (minority score) resulted from random occurrence of two errors was calculated and found to be <5%, thereby indicating that the stage assigned is an acceptable variant for the epoch. Acceptable stages were identified in each epoch as stages assigned by at least two technologists. Percent agreement between each technologist and the other five technologists, acting as judges, was determined. Agreement was considered to exist if the stage assigned by the tested scorer was one of the acceptable stages for the epoch. Stage assigned by MSS was likewise considered in agreement if included in the acceptable stages made by the technologists.</p><p><strong>Results: </strong>Agreement of technologists tested against five qualified judges increased from 80.8% (range 70.5%-86.4% among technologists) when using the majority rule, to 96.1 (89.8%-98.5%) by the proposed approach. Agreement between unedited MSS and same judges was 90.0% and increased to 92.1% after brief editing.</p><p><strong>Conclusions: </strong>Accounting for equivocal epochs provides a more accurate estimate of a scorer's (human or digital) competence in scoring sleep stages and reduces inter-scorer disagreements. The proposed approach can be implemented in sleep-scoring training and accreditation programs.</p>","PeriodicalId":49514,"journal":{"name":"Sleep","volume":null,"pages":null},"PeriodicalIF":5.3000,"publicationDate":"2023-11-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Sleep","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1093/sleep/zsad248","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"CLINICAL NEUROLOGY","Score":null,"Total":0}
引用次数: 0
Abstract
Study objectives: Inter-scorer variability in sleep staging is largely due to equivocal epochs that contain features of more than one stage. We propose an approach that recognizes the existence of equivocal epochs and evaluates scorers accordingly.
Methods: Epoch-by-epoch staging was performed on 70 polysomnograms by six qualified technologists and by a digital system (Michele Sleep Scoring [MSS]). Probability that epochs assigned the same stage by only two of the six technologists (minority score) resulted from random occurrence of two errors was calculated and found to be <5%, thereby indicating that the stage assigned is an acceptable variant for the epoch. Acceptable stages were identified in each epoch as stages assigned by at least two technologists. Percent agreement between each technologist and the other five technologists, acting as judges, was determined. Agreement was considered to exist if the stage assigned by the tested scorer was one of the acceptable stages for the epoch. Stage assigned by MSS was likewise considered in agreement if included in the acceptable stages made by the technologists.
Results: Agreement of technologists tested against five qualified judges increased from 80.8% (range 70.5%-86.4% among technologists) when using the majority rule, to 96.1 (89.8%-98.5%) by the proposed approach. Agreement between unedited MSS and same judges was 90.0% and increased to 92.1% after brief editing.
Conclusions: Accounting for equivocal epochs provides a more accurate estimate of a scorer's (human or digital) competence in scoring sleep stages and reduces inter-scorer disagreements. The proposed approach can be implemented in sleep-scoring training and accreditation programs.
期刊介绍:
SLEEP® publishes findings from studies conducted at any level of analysis, including:
Genes
Molecules
Cells
Physiology
Neural systems and circuits
Behavior and cognition
Self-report
SLEEP® publishes articles that use a wide variety of scientific approaches and address a broad range of topics. These may include, but are not limited to:
Basic and neuroscience studies of sleep and circadian mechanisms
In vitro and animal models of sleep, circadian rhythms, and human disorders
Pre-clinical human investigations, including the measurement and manipulation of sleep and circadian rhythms
Studies in clinical or population samples. These may address factors influencing sleep and circadian rhythms (e.g., development and aging, and social and environmental influences) and relationships between sleep, circadian rhythms, health, and disease
Clinical trials, epidemiology studies, implementation, and dissemination research.