Tree-based machine learning models recover rule-based flare labels from virtual symptom diaries in pediatric Behçet's disease: an in silico proof-of-concept study.

Abstract

Behçet's disease (BD) in childhood is characterised by recurrent inflammatory flares that can result in significant morbidity, most notably with ocular, mucocutaneous, and musculoskeletal manifestations. Timely detection of impending flares is difficult in routine practice, and digital health technologies coupled with patient-reported symptom diaries have been proposed as a candidate route to real-time disease surveillance. As a first methodological step before any prospective evaluation in real patients, we set out to demonstrate that a diary-based machine learning (ML) pipeline can recover a transparent, rule-based flare definition under fully controlled in silico conditions, and to characterise the behaviour of such a pipeline before any clinical deployment is considered. A synthetic, longitudinally structured BD dataset was generated in silico, comprising 100 virtual pediatric patients each followed daily for 90 days (9000 patient-day observations). No real patient data were collected; the dataset was constructed solely as a methodological proof-of-concept and is not presented as a substitute for prospectively collected clinical observations. All simulation parameters were drawn from published pediatric BD distributions and are fully documented in Supplementary Table S1. Recorded features comprised oral and genital ulcers, mucocutaneous lesions, ocular involvement, joint pain or swelling, fever, fatigue, gastrointestinal and vascular involvement, treatment class, and adherence score. Flare was operationalised by a transparent composite rule: ≥2 of {oral ulcer, genital ulcer, joint pain/swelling, ocular involvement, skin lesion} accompanied by either fever >38°C or severe fatigue (≥3 on a 0-3 ordinal scale). Six ML algorithms were compared: Logistic Regression, Random Forest, XGBoost, LightGBM, Support Vector Machine (RBF kernel), and Multilayer Perceptron. The dataset was split at the patient level (80/20) with stratification by patient-level flare prevalence to prevent within-patient leakage. Model performance was evaluated by 5-fold cross-validation using accuracy, sensitivity, specificity, F1-score, AUC, Matthews correlation coefficient (MCC), Brier score, and decision curve analysis. SHAP analysis and unsupervised k-means clustering were used for interpretability. Three pre-specified robustness analyses were performed: label-noise injection (5% and 10% flips on the training partition only), flare-prevalence shifts (5% and 30%), and AR-1 mis-specification (φ ± 0.20); results appear in Supplementary Table S2. Reporting followed the TRIPOD+AI and GAMER guidelines (Supplementary Checklist). Across all six classifiers, tree-based ensembles recovered the labelling function most efficiently. Random Forest achieved the best overall performance (accuracy 0.86, recall 1.00, AUC 0.85, MCC 0.72, Brier 0.11), closely followed by XGBoost (AUC 0.81, MCC 0.66, F1 0.83) and LightGBM (AUC 0.80, MCC 0.63, F1 0.81). Logistic regression, SVM (RBF), and MLP clustered at AUC 0.75-0.77. SHAP analysis ranked oral ulcers, joint pain, and fever as the strongest predictors, with fatigue and ocular involvement contributing to a lesser degree. Because these features are explicit constituents of the labelling rule, their SHAP rankings index pipeline-recovery rather than independent biological discovery. Unsupervised clustering identified three phenotypic groupings consistent with the simulated symptom distributions, serving as an internal consistency check on the data-generating mechanism rather than a novel clinical finding. Decision curve analysis showed a mathematically well-defined positive net benefit across threshold probabilities from 0.10 to 0.70 under the synthetic data-generating distribution, which we do not interpret as evidence of real-world clinical utility. Robustness analyses confirmed that pipeline performance degraded gracefully under modest label noise and prevalence shift but did not establish clinical generalisability. These in silico results provide a methodological proof-of-concept showing that a diary-based ML pipeline can faithfully recover a transparent, rule-based flare definition in pediatric BD with sensible calibration and interpretable feature attribution. They do not, in themselves, establish that such a pipeline would generalise to real patients, in whom flare adjudication requires clinician evaluation and major-organ surveillance that simple diary-based composites cannot replicate. Prospective multicentre validation against independently adjudicated flare labels and post-hoc probability calibration will be required before any clinical role can be considered.