An inter-rater variability study between human and automatic scorers in 5-s mini-epochs of sleep

IF 3.8 2区医学 Q1 CLINICAL NEUROLOGY

Sleep medicine Pub Date : 2025-02-06 DOI:10.1016/j.sleep.2025.02.005

Louise Frøstrup Follin , Alexander Neergaard Zahid , Rannveig Viste , Janita Vevelstad , Tobias Kaufmann , Anette Ramm-Pettersen , Hilde T. Juvodden , Berit Hjelde Hansen , Julie Anja Engelhard Christensen , Stine Knudsen-Heier

{"title":"An inter-rater variability study between human and automatic scorers in 5-s mini-epochs of sleep","authors":"Louise Frøstrup Follin , Alexander Neergaard Zahid , Rannveig Viste , Janita Vevelstad , Tobias Kaufmann , Anette Ramm-Pettersen , Hilde T. Juvodden , Berit Hjelde Hansen , Julie Anja Engelhard Christensen , Stine Knudsen-Heier","doi":"10.1016/j.sleep.2025.02.005","DOIUrl":null,"url":null,"abstract":"<div><h3>Study objective</h3><div>Sleep is traditionally scored using 30-s epochs of polysomnographies. As sleep is physiologically dynamic and 30-s epochs may conceal important characteristics, we aim to challenge this standard by scoring sleep in 5-s mini-epochs and analyzing inter-rater variability between human and automatic scorers.</div></div><div><h3>Methods</h3><div>In 40 polysomnography recordings, 120 mini-epochs per polysomnography were scored manually by three human experts (expert1_5s, expert2_5s and expert3_5s) and automatically by a validated sleep classifier (USleep_5s). Additionally, 5-s mini-epochs (clinical_5s) extracted from conventional human-scored 30-s epochs were considered. We assessed inter-rater variability and stage shifting in epochs and mini-epochs and further in narcolepsy type 1 (NT1) patients and siblings.</div></div><div><h3>Results</h3><div>Agreement for mini-epochs was κ = 0.50 ± 0.11 (expert1_5s vs clinical_5s) and κ = 0.51 ± 0.12, (expert1_5s vs USleep_5s). Between human experts, agreement was κ = 0.51 ± 0.16 (expert1_5s vs expert2_5s), and κ = 0.57 ± 0.11 (expert1_5s vs expert3_5s). Stage shift percentages were significantly higher in mini-epochs scored by expert1_5s (27.75 %) and USleep_5s (22.88 %) than corresponding conventional epochs (5.12 %), with no significant difference between NT1 patients and siblings.</div></div><div><h3>Conclusion</h3><div>While mini-epoch scoring agreement was generally high, it was still lower than within epochs, likely due to a lack of standard mini-epoch scoring procedure and the automatic classifier being trained on epochs. However, stage discrepancies between epochs and mini-epochs and increased stage shifting in mini-epochs support that epochs can contain several stages, and that mini-epochs could supplement more detailed sleep characterization potentially enabling more precise diagnosis and finding new polysomnographic biomarkers. Future studies should include larger datasets to refine mini-epoch scoring rules and exploit automatic classifiers e.g. via transfer learning.</div></div>","PeriodicalId":21874,"journal":{"name":"Sleep medicine","volume":"128 ","pages":"Pages 139-150"},"PeriodicalIF":3.8000,"publicationDate":"2025-02-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Sleep medicine","FirstCategoryId":"3","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S1389945725000504","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"CLINICAL NEUROLOGY","Score":null,"Total":0}

引用次数: 0

Abstract

Study objective

Sleep is traditionally scored using 30-s epochs of polysomnographies. As sleep is physiologically dynamic and 30-s epochs may conceal important characteristics, we aim to challenge this standard by scoring sleep in 5-s mini-epochs and analyzing inter-rater variability between human and automatic scorers.

Methods

In 40 polysomnography recordings, 120 mini-epochs per polysomnography were scored manually by three human experts (expert1_5s, expert2_5s and expert3_5s) and automatically by a validated sleep classifier (USleep_5s). Additionally, 5-s mini-epochs (clinical_5s) extracted from conventional human-scored 30-s epochs were considered. We assessed inter-rater variability and stage shifting in epochs and mini-epochs and further in narcolepsy type 1 (NT1) patients and siblings.

Results

Agreement for mini-epochs was κ = 0.50 ± 0.11 (expert1_5s vs clinical_5s) and κ = 0.51 ± 0.12, (expert1_5s vs USleep_5s). Between human experts, agreement was κ = 0.51 ± 0.16 (expert1_5s vs expert2_5s), and κ = 0.57 ± 0.11 (expert1_5s vs expert3_5s). Stage shift percentages were significantly higher in mini-epochs scored by expert1_5s (27.75 %) and USleep_5s (22.88 %) than corresponding conventional epochs (5.12 %), with no significant difference between NT1 patients and siblings.

Conclusion

While mini-epoch scoring agreement was generally high, it was still lower than within epochs, likely due to a lack of standard mini-epoch scoring procedure and the automatic classifier being trained on epochs. However, stage discrepancies between epochs and mini-epochs and increased stage shifting in mini-epochs support that epochs can contain several stages, and that mini-epochs could supplement more detailed sleep characterization potentially enabling more precise diagnosis and finding new polysomnographic biomarkers. Future studies should include larger datasets to refine mini-epoch scoring rules and exploit automatic classifiers e.g. via transfer learning.

Abstract Image

查看原文本刊更多论文

求助全文

约1分钟内获得全文求助全文

来源期刊

Sleep medicine 医学-临床神经学

CiteScore

8.40

自引率

6.20%

发文量

1060

审稿时长

49 days

期刊介绍： Sleep Medicine aims to be a journal no one involved in clinical sleep medicine can do without. A journal primarily focussing on the human aspects of sleep, integrating the various disciplines that are involved in sleep medicine: neurology, clinical neurophysiology, internal medicine (particularly pulmonology and cardiology), psychology, psychiatry, sleep technology, pediatrics, neurosurgery, otorhinolaryngology, and dentistry. The journal publishes the following types of articles: Reviews (also intended as a way to bridge the gap between basic sleep research and clinical relevance); Original Research Articles; Full-length articles; Brief communications; Controversies; Case reports; Letters to the Editor; Journal search and commentaries; Book reviews; Meeting announcements; Listing of relevant organisations plus web sites.