GPT-4 in Clinical Practice: Assessing Its Capability for Symptom Extraction from Cancer Patient Notes.

Abstract

Accurate extraction of patient symptoms and signs from clinical notes is essential for effective diagnosis, treatment planning, and research. In this study, we evaluate the capability of GPT-4, specifically GPT-4o, in extracting symptoms and signs from nursing notes within the MIMIC-III dataset. We experimented with two temperature settings (1 and 0.3) to explore the impact of model diversity and consistency on extraction accuracy. Performance metrics include precision, specificity, recall, and F1-score. The results show that a higher temperature (1) led to more creative and varied outputs, with a mean precision of 79% and specificity of 96%, but also exhibited variability, with a minimum precision of 24%. Conversely, at a lower temperature (0.3), precision was more conservative but dropped significantly, with a mean precision of 45% and minimum of 0%. High recall and specificity at optimal temperature setting indicates that GPT-4 holds promise as an assistive tool in clinical practice for symptom and sign extraction tasks.