Evaluating Locally Run Large Language Models for Obstructive Sleep Apnea Diagnosis and Treatment: A Real-World Polysomnography Study.

Abstract

Purpose: Sleep medicine is a highly resource-intensive field where large language models (LLMs) could offer a promising solution by supporting diagnostic processes. As web-based LLMs have obvious data protection constraints, locally run LLMs are essential for clinical implementation. This study is the first to investigate the performance of locally run LLMs in the interpretation of real-world polysomnographic (PSG) results.

Methods: We randomly selected N=30 patients (18 male, 12 female, mean age 50.5 ± 11.1 years, mean body mass index 29.7 ± 5.5 kg/m², mean apnea hypopnea index 30.9 ± 23.8) from the clinical database of our sleep laboratory who underwent PSG due to clinical complaints typical of obstructive sleep apnea (OSA). The board-certified sleep physician's interpretations of diagnosis, suitable first-line therapy or alternative therapy were compared with those of three locally run LLMs (Gemma2, Llama3 and Mistral Nemo) assessing the level of concordance.

Results: Gemma2 showed the lowest concordance of 33% (10/30 patients) with the board-certified sleep physician regarding OSA severity, followed by Mistral Nemo at 47% (14/30 patients) and Llama3 at 50% (15/30 patients). For automatic positive airway pressure (aPAP) recommendations, Mistral Nemo showed the highest concordance at 90% (27/30 patients), followed by Gemma2 and Llama3 with 83% (25/30 patients) each.

Conclusion: Although locally run LLMs bypass data security constraints and show promising potential for clinical practice, their performance needs significant improvement prior to real-world implementation. Therefore, at present, the routine implementation of locally run LLMs in sleep medicine needs more refinement and fine tuning before they can be used for interpretation of real-world PSG results.