A multi-metric evaluation of readability in psychiatric discharge summaries.

Abstract

BACKGROUND: Readability in medical notes is critical for patient engagement and comprehension, especially in mental health, where readable documentation reduces stigmatizing language, promotes patient-friendly communication, and strengthens trust. The importance of readability has been amplified by the federal OpenNotes policy (mandated by the 21st Century Cures Act since April 2021), which requires providers to offer patients immediate electronic access to their clinical notes. This mandate makes it imperative that mental health documentation is both clinically useful and easily understood. METHODS: This study evaluated psychiatric discharge summaries from the MIMIC-IV dataset. Specifically, we compared patient-facing “Discharge Instructions” against provider-facing “Brief Hospital Course” sections to assess documentation practices. We analyzed 1,745 notes associated with four major psychiatric diagnoses (Major Depression, Bipolar Disorder, Schizophrenia, and Eating Disorders), with a paired subset of 880 notes containing both sections. Readability was assessed using a multidimensional framework of 12 metrics, spanning traditional grade-level formulas (e.g., SMOG, KFGL) and deep learning–based models (ClinicalBERT, MedReadMe). The internal correlation across these metrics was also evaluated. RESULTS: Discharge Instructions were significantly shorter, structurally simpler, and scored lower on difficulty indices compared to Brief Hospital Course sections. Additionally, complexity varied significantly by diagnosis: the Eating Disorder cohort uniquely displayed a “readability inversion,” having the simplest instructions but the most complex hospital course summaries. Furthermore, traditional statistical measures showed moderate to high internal consistency. While they demonstrated moderate correlation with MedReadMe, they showed weaker correlation with ClinicalBERT rankings, suggesting that deep learning models capture semantic dimensions of text difficulty distinct from surface-level syntactic complexity. CONCLUSION: Psychiatric documentation practices successfully simplify patient-facing text. However, the observed variability across diagnostic groups and the divergence between traditional and neural readability metrics highlight the necessity of using multipronged evaluation frameworks to accurately assess the accessibility and comprehension of clinical text in the OpenNotes era.