Necessary for seizure forecasting outcome metrics: Seizure frequency and benchmark model

Background

This study aims to illustrate the connection between seizure frequency (SF) and performance metrics in seizure forecasting, and to compare the effectiveness of a moving average (MA) model versus the commonly used permutation benchmark.

Methods

Metrics of calibration and discrimination were computed for each dataset, comparing MA and permutation performance across SF values. Three datasets were used: (1) self-reported seizure diaries from 3994 Seizure Tracker patients, (2) automatically detected and sometimes manually reported or edited generalized tonic-clonic seizures from 2350 Empatica Embrace 2 and Mate App users, and (3) simulated datasets with varying SFs.

Results

Most metrics were found to depend on SF. The MA model outperformed or matched the permutation model in all cases. These more advanced metrics show that comparison to permutation will falsely elevate poor forecasting models.

Conclusions

The findings highlight SF's role in seizure forecasting accuracy and the MA model's suitability as a benchmark. This study underscores the need for considering patient SF in forecasting studies and suggests the MA model may provide a better standard for evaluating future seizure forecasting models.