Large language model derived regular expressions for sleep phenotyping from electronic health record: a feasibility study.

Abstract

Study objectives: Large language models may efficiently develop Regular Expression (regex)-based natural language processing for electronic health record phenotyping. We present PromptNLP to assess feasibility by evaluating performance and generalizability to clinical notes across multiple institutions.

Methods: Sleep-related clinical notes from Kaiser Permanente Southern California, Beth Israel Deaconess Medical Center, and Emory University were used to extract apnea-hypopnea index and Epworth Sleepiness Scale. PromptNLP involves two steps: (1) build an apnea-hypopnea index/Epworth Sleepiness Scale phrase library ("Targeted" via manual, "General" via large language models); (2) input the library into an large language model with prompts to generate regex. For model development (Phase 1), we compared PromptNLP regexes with manual regex. For external validation (Phase 2), we adapted the Phase 1 Targeted regex to Beth Israel Deaconess Medical Center and Emory notes. For application (Phase 3), we applied adapted the regex to 207 404 Kaiser Permanente Southern California notes with validation against 8602 structured apnea-hypopnea index.

Results: Phase 1: All Targeted PromptNLP regexes outperformed manual regex for recall (apnea-hypopnea index: 95%-96% vs 80.8%; Epworth Sleepiness Scale: 98%-100% vs 88%; p < .05). General regexes surpassed manual, but with lower recall. Phase 2: Direct Beth Israel Deaconess Medical Center application performed poorly (apnea-hypopnea index/Epworth Sleepiness Scale: 30%/12%), but adapting the phrase library improved recall to 96%/84%. For Emory, recall improved from 85%/86% to 94%/89% after adaptation. Across Phases 1 and 2, the PromptNLP regexes achieved 100% precision. Phase 3: 8216 (95.5%) apnea-hypopnea index values were correctly extracted with 99.8% precision.

Conclusions: PromptNLP demonstrated feasibility as an efficient, accurate, and generalizable method, outperforming manual approaches and enabling iterative cross-system improvement. Systematic comparisons to direct large language model approaches are needed.