CUI-Curate: a GraphRAG-based framework for automated clinical concept curation for NLP applications.

Abstract

Background: Clinical named entity recognition tools commonly map free text to Unified Medical Language System (UMLS) Concept Unique Identifiers (CUIs). For many downstream tasks, however, the clinically meaningful unit is not a single CUI but a concept set comprising related synonyms, subtypes, and associated concepts. Constructing these sets is labour-intensive, inconsistently performed, and poorly supported by existing tools.

Methods: We present CUI-Curate, a graph-based retrieval-augmented-generation (GraphRAG) framework for automated UMLS concept set curation. A UMLS knowledge graph was constructed and embedded for semantic retrieval. Candidate CUIs were retrieved using graph-based expansion and then filtered and classified using large language models (GPT-5 and Qwen3-32B). The framework was evaluated on five lexically heterogeneous clinical concepts against manually curated concept sets and gold-standard concept sets.

Results: CUI-Curate produced substantially larger and more complete concept sets than the manual benchmarks. A single retrieval configuration across concepts achieved high recall of definitive concepts with manageable candidate sets. GPT-5 outperformed manual curation for all concepts and retained at least 95% of definitive gold-standard CUIs, while Qwen3-32B achieved comparable but slightly lower performance. Many missed concepts were not observed in 10,000 MIMIC-III notes. CUI-Curate infrastructure and end-to-end processing were inexpensive and stable across runs.

Conclusions: CUI-Curate offers a scalable, reproducible, and cost-efficient approach for generating clinician-reviewable UMLS concept sets tailored to clinical natural language processing and phenotyping applications.