Necessary for seizure forecasting outcome metrics: Seizure frequency and benchmark model

IF 2 4区医学 Q3 CLINICAL NEUROLOGY

Epilepsy Research Pub Date : 2024-11-08 DOI:10.1016/j.eplepsyres.2024.107474

Chi-Yuan Chang , Boyu Zhang , Robert Moss , Rosalind Picard , M. Brandon Westover , Daniel Goldenholz

{"title":"Necessary for seizure forecasting outcome metrics: Seizure frequency and benchmark model","authors":"Chi-Yuan Chang , Boyu Zhang , Robert Moss , Rosalind Picard , M. Brandon Westover , Daniel Goldenholz","doi":"10.1016/j.eplepsyres.2024.107474","DOIUrl":null,"url":null,"abstract":"<div><h3>Background</h3><div>This study aims to illustrate the connection between seizure frequency (SF) and performance metrics in seizure forecasting, and to compare the effectiveness of a moving average (MA) model versus the commonly used permutation benchmark.</div></div><div><h3>Methods</h3><div>Metrics of calibration and discrimination were computed for each dataset, comparing MA and permutation performance across SF values. Three datasets were used: (1) self-reported seizure diaries from 3994 Seizure Tracker patients, (2) automatically detected and sometimes manually reported or edited generalized tonic-clonic seizures from 2350 Empatica Embrace 2 and Mate App users, and (3) simulated datasets with varying SFs.</div></div><div><h3>Results</h3><div>Most metrics were found to depend on SF. The MA model outperformed or matched the permutation model in all cases. These more advanced metrics show that comparison to permutation will falsely elevate poor forecasting models.</div></div><div><h3>Conclusions</h3><div>The findings highlight SF's role in seizure forecasting accuracy and the MA model's suitability as a benchmark. This study underscores the need for considering patient SF in forecasting studies and suggests the MA model may provide a better standard for evaluating future seizure forecasting models.</div></div>","PeriodicalId":11914,"journal":{"name":"Epilepsy Research","volume":"208 ","pages":"Article 107474"},"PeriodicalIF":2.0000,"publicationDate":"2024-11-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Epilepsy Research","FirstCategoryId":"3","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S092012112400189X","RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"CLINICAL NEUROLOGY","Score":null,"Total":0}

引用次数: 0

Abstract

Background

This study aims to illustrate the connection between seizure frequency (SF) and performance metrics in seizure forecasting, and to compare the effectiveness of a moving average (MA) model versus the commonly used permutation benchmark.

Methods

Metrics of calibration and discrimination were computed for each dataset, comparing MA and permutation performance across SF values. Three datasets were used: (1) self-reported seizure diaries from 3994 Seizure Tracker patients, (2) automatically detected and sometimes manually reported or edited generalized tonic-clonic seizures from 2350 Empatica Embrace 2 and Mate App users, and (3) simulated datasets with varying SFs.

Results

Most metrics were found to depend on SF. The MA model outperformed or matched the permutation model in all cases. These more advanced metrics show that comparison to permutation will falsely elevate poor forecasting models.

Conclusions

The findings highlight SF's role in seizure forecasting accuracy and the MA model's suitability as a benchmark. This study underscores the need for considering patient SF in forecasting studies and suggests the MA model may provide a better standard for evaluating future seizure forecasting models.

查看原文本刊更多论文

癫痫发作预测结果指标的必要条件：癫痫发作频率和基准模型。

背景：本研究旨在说明癫痫发作频率（SF）与癫痫发作预测中的性能指标之间的联系，并比较移动平均（MA）模型与常用的置换基准的有效性：方法：计算每个数据集的校准和判别指标，比较不同 SF 值的移动平均模型和置换模型的性能。使用了三个数据集：(1) 来自 3994 名 Seizure Tracker 患者的自我报告发作日记；(2) 来自 2350 名 Empatica Embrace 2 和 Mate App 用户的自动检测和有时手动报告或编辑的全身强直阵挛发作；(3) 具有不同 SF 的模拟数据集：结果：发现大多数指标取决于 SF。在所有情况下，MA 模型都优于或与 permutation 模型相当。这些更先进的指标表明，与置换模型进行比较会错误地抬高差劲的预测模型：研究结果强调了 SF 在癫痫发作预测准确性中的作用以及 MA 模型作为基准的适用性。这项研究强调了在预测研究中考虑患者 SF 的必要性，并建议 MA 模型可为评估未来的癫痫发作预测模型提供更好的标准。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Epilepsy Research 医学-临床神经学

CiteScore

0.10

自引率

4.50%

发文量

143

审稿时长

62 days

期刊介绍： Epilepsy Research provides for publication of high quality articles in both basic and clinical epilepsy research, with a special emphasis on translational research that ultimately relates to epilepsy as a human condition. The journal is intended to provide a forum for reporting the best and most rigorous epilepsy research from all disciplines ranging from biophysics and molecular biology to epidemiological and psychosocial research. As such the journal will publish original papers relevant to epilepsy from any scientific discipline and also studies of a multidisciplinary nature. Clinical and experimental research papers adopting fresh conceptual approaches to the study of epilepsy and its treatment are encouraged. The overriding criteria for publication are novelty, significant clinical or experimental relevance, and interest to a multidisciplinary audience in the broad arena of epilepsy. Review articles focused on any topic of epilepsy research will also be considered, but only if they present an exceptionally clear synthesis of current knowledge and future directions of a research area, based on a critical assessment of the available data or on hypotheses that are likely to stimulate more critical thinking and further advances in an area of epilepsy research.