Benchmarking of pre-training strategies for electronic health record foundation models.

Objective: Our objective is to compare different pre-training strategies for electronic health record (EHR) foundation models.

Materials and methods: We evaluated three approaches using a transformer-based architecture: baseline (no pre-training), self-supervised pre-training with masked language modeling, and supervised pre-training. The models were assessed on their ability to predict both major adverse cardiac events and mortality occurring within 12 months. The pre-training cohort was 405 679 patients prescribed antihypertensives and the fine tuning cohort was 5525 patients who received doxorubicin.

Results: Task-specific supervised pre-training achieved superior performance (AUROC 0.70, AUPRC 0.23), outperforming both self-supervised pre-training and the baseline. However, when the model was evaluated on the task of 12-month mortality prediction, the self-supervised model performed best.

Discussion: While supervised pre-training excels when aligned with downstream tasks, self-supervised approaches offer more generalized utility.

Conclusion: Pre-training strategy selection should consider intended applications, data availability, and transferability requirements.