A Large Language Model Approach to Identifying Preoperative Frailty Among Older Adults From Clinical Notes

Abstract

Background

Patients with frailty have a higher risk of major postoperative mortality and morbidity. Identifying frailty from the medical record, however, is not straightforward since it is a multifactorial state based on multiple organ systems and a sum of factors accumulated over time. The objective of this study was to develop a large language model-based binary classifier using accurately phenotyped datasets to identify preoperative frailty from clinical notes.

Methods

We trained various large language models to identify frailty from anesthesia preoperative clinic notes. There were two development datasets used: (1) patients undergoing spine surgery whose frailty was characterized by patient responses to the Vulnerable Elders-13 Survey (VES-13); and (2) patients undergoing surgery whose frailty was characterized by their calculated electronic frailty index (eFI) score.

Results

When trained on our VES-13 development set and tested on our VES-13 validation set, the area under the receiver operating characteristics curve (AUC) for the RoBERTa, BERT, BioBERT, and PubMedBERT models was 0.99, 0.64, 0.67, and 0.73, respectively. When tested on the eFI validation set, the AUCs were 0.63, 0.83, 0.87, and 0.87, respectively. Models trained on the eFI development dataset did not discriminate frailty adequately when tested on the VES-13 validation set.

Conclusion

We report the development and validation of a classifier that detects older adults at risk for preoperative frailty from preoperative anesthesia clinical notes. Large language models can be used to accurately identify a difficult-to-quantify and multifactorial characteristic such as frailty in patients by using readily available unstructured information from clinical notes.