Natural Language Processing to Extract Head and Neck Cancer Data From Unstructured Electronic Health Records

Abstract

Aims

Patient data is frequently stored as unstructured data within Electronic Health Records (EHRs), requiring manual curation. AI tools using Natural Language Processing (NLP) may rapidly curate accurate real-world unstructured EHRs to enrich datasets. We evaluated this approach for Head and Neck Cancer (HNC) patient data extraction using an open-source general-purpose healthcare NLP tool (CogStack).

Materials and Methods

CogStack was applied to extract relevant SNOMED-CT concepts from HNC patients' documents, generating outputs denoting the identifications of each concept for each patient. Outputs were compared to manually curated ground truth HNC datasets to calculate pre-training performance. Supervised model training was then performed using SNOMED-CT concept annotation on clinical documents, and the updated model was re-evaluated. A second training cycle was performed before the final evaluation. A thresholding approach (multiple detections needed to qualify a concept as ‘present’) was used to increase precision. The final model was evaluated on an unseen test cohort. F1 score (harmonic mean of precision and recall) was used for evaluation.

Results

Pre-training, the F1 score was incalculable for 19.5% of concepts due to insufficient recall. Following one training cycle, F1 score became calculable for all concepts (median 0.692). After further training, the final model demonstrated improvement in the median F1 score (0.708). Test cohort median F1 score was 0.750. Thresholding analysis developed a concept-specific best threshold approach, resulting in a median F1 score of 0.778 in the test cohort, where 50 out of 109 SNOMED-CT concepts met pre-set criteria to be considered adequately fine-tuned.

Conclusions

NLP can mine unstructured cancer data following limited training. Certain concepts such as histopathology terms remained poorly retrieved. Model performance is maintained when applied to a test cohort, demonstrating good generalisability. Concept-specific thresholding strategy improved performance. Fine-tuning annotations were incorporated into the NLP parent model for future performance. CogStack has been applied to extract data for 50 concepts with validated performance for our entire retrospective HNC cohort.