Artificial intelligence or sleep experts: Comparing polysomnographic sleep staging in children and adolescents.

Abstract

Study objectives: The manual annotation of polysomnography (PSG) hypnograms is difficult and time-consuming. U-Sleep is an alternative, fast and publicly available, automated sleep staging system evaluated in adult PSGs. In this study we compare the staging done by sleep experts and U-sleep in a pediatric sample.

Methods: PSGs from 56 children aged 6-17 years old (healthy or with a chronic disease) were compared manually annotated with the result of U-sleep. The two outcomes were compared using F1 overlap scores, accuracy, Cohen's kappa, and correlation coefficients. A qualitative analysis of the most significant systematic differences between the manual and automated scoring was performed.

Results: U-sleep matched the manually scored hypnograms with an overall mean F1 score (predicted performance) of 0.75 and reached an accuracy of 83.9% and an overall kappa value of 0.77. The stage-wise F1 scores, U-sleep achieved an F1 score of 0.79 in stage Wake, 0.40 in N1, 0.86 in N2, 0.84 in N3, and 0.86 in REM. The correlation between U-sleep and the manual scorer was moderately or very strong in all sleep stages (r = 0.57-0.81).

Conclusions: Overall, there is a high degree of agreement between manual and automatic scoring. This suggests that U-sleep is a valid and effective method for identifying sleep stages based on normal PSGs in a pediatric population. The disagreement was within what is expected for interscorer variation. Further evaluation needs of AI sleep scoring models includes analysis of outliers and pathological sleep staging - which is also a challenge in manual annotation.