Exploratory Implementation and Feasibility Report of CLASS (Clinical LLM Abstraction & Structuring System), A Large Language Model Pipeline for Extracting Unstructured Data From Clinical Notes.

Abstract

Electronic medical records contain rich clinical detail, but much of it resides in unstructured notes, making manual abstraction for research and quality improvement labor-intensive and difficult to scale. This report describes the development of the Clinical LLM Abstraction & Structuring System (CLASS), a Python-based modular pipeline that uses a large language model (LLM) to extract structured data from clinical notes within a secure institutional environment. CLASS combines subject matter expert-curated concept lists, a task-specific prompt suite, and a schema-constrained output format, and exports results to a dashboard for expert review and analysis. The pipeline classifies predefined concepts and flags potential variants or novel concepts for expert consideration. Its modular design allows configuration for different abstraction tasks and can process large note volumes within standard institutional infrastructure. CLASS was applied to a retrospective corpus of pediatric esophageal airway treatment surgery (EATS) operative notes at a single center. In an exploratory evaluation comparing CLASS outputs against surgeon adjudication on the 20 longest notes (3,960 note-procedure pairs), observed concordance was high (F1 0.9967). CLASS also proposed 28 candidate procedure variants or additions, 18 (64.3%) of which were judged clinically useful, though the surgeon identified 12 additional procedures that CLASS did not surface. These findings suggest that an LLM-based pipeline can feasibly extract complex, non-coded procedure information from unstructured clinical notes within this specific single-center, single-service context, and may assist clinicians and informatics teams in curating and expanding specialized concept lists, though broader validation is needed to assess generalizability to other tasks and settings.