Using natural language processing and machine learning to identify social needs in patient medical notes: model training with balanced versus real-world datasets.

Abstract

Objective: To evaluate natural language processing (NLP) and machine learning (ML) approaches for identifying social needs in electronic health record (EHR) notes, using balanced versus real-world imbalanced datasets.

Materials and methods: A rule-based NLP framework using definite, non-negated keywords flagged social needs domains and "Any Social Need" in unstructured notes. Logistic regression, random forest, and XGBoost models were trained under balanced and imbalanced prevalence and evaluated using area under the receiver operating characteristic curve (AUROC), area under the precision-recall curve (AUPRC), F1, precision, recall, and specificity, with threshold sensitivity analyses.

Results: On the imbalanced test set for any social need, the rule-based model achieved F1 0.373 (precision 0.484; recall 0.303; and specificity 0.978). On the balanced test set for any social need, XGBoost and logistic models achieved AUPRC 0.719 and 0.718, F1 0.438, and precision 0.944; random forest had AUPRC 0.714, F1 0.423, and precision 0.945. Under imbalanced evaluation, XGBoost led (AUPRC 0.319, F1 0.312, precision 0.839, recall 0.191, and specificity 0.998), followed by logistic (AUPRC 0.312 and F1 0.289) and random forest (AUPRC 0.305 and F1 0.282).

Discussion: Balanced training inflated apparent performance, while under realistic class imbalance the ML models traded recall for very high precision, with XGBoost performing best but still lagging in sensitivity.

Conclusion: These findings underscore the need to train and tune under real-world prevalence and to set thresholds based on use-case priorities (eg, maximizing precision for referrals versus improving recall for screening).