AI language model applications for early diagnosis of childhood epilepsy based on unstructured first-visit patient narratives: A cohort study.

Abstract

Objective: Language serves as an indispensable source of information for diagnosing epilepsy, and its computational analysis is increasingly explored. This study assessed - and compared - the diagnostic value of different language model applications in extracting information. The aim is to identify language patterns that may contain useful clinical information that is not overtly considered by the clinician from first-visit documentation to improve the early diagnosis of childhood epilepsy.

Methods: We analyzed 1561 patient letters from the first two seizure clinics. The dataset was divided into training and test sets to evaluate performance and generalizability. We employed an established Naïve Bayes model as a natural language processing technique and a sentence-embedding (large language) model based on the Bidirectional Encoder Representations from Transformers (BERT) architecture. Both models analyzed anamnesis texts as noted by the treating physician only. Within the training sets, we identified predictive features consisting of keywords indicative of 'epilepsy' or 'no epilepsy.' Model outputs were compared to the clinician's final diagnosis (gold standard) after a two-year follow-up period. We computed accuracy, sensitivity, and specificity for both models.

Results: The Naïve Bayes model achieved an accuracy of 0.73 (95% CI: 0.68-0.78), with a sensitivity of 0.79 (95% CI: 0.74-0.85) and a specificity of 0.62 (95% CI: 0.52-0.72). The sentence-embedding model demonstrated comparable performance with an accuracy of 0.74 (95% CI: 0.68-0.79), a sensitivity of 0.74 (95% CI: 0.68-0.80), and a specificity of 0.73 (95% CI: 0.61-0.84).

Significance: Both models demonstrated relatively good performance in diagnosing childhood epilepsy solely based on the first-visit patient anamnesis text. Notably, the more advanced sentence-embedding model showed no improvement over the computationally simpler Naïve Bayes model. This suggests that modeling of anamnesis data does depend on word order for this particular classification task. Further refinement and exploration of language models and computational linguistic approaches are necessary to enhance diagnostic accuracy in clinical practice.