GPT-4o Translation of Pediatric Discharge Instructions Across Languages.

Abstract

Background and objectives: Patients and caregivers who use languages other than English in health care settings rarely receive language-concordant discharge instructions. Large language models show promise for improving access to translations but must be validated to ensure their output is safe for clinical use. This study measured the quality of GPT-4o medical translations to Portuguese, Haitian Creole, and Arabic.

Methods: Previously translated personalized discharge instructions (performed by professional human translators) were extracted from inpatient admissions at a large US pediatric academic medical center. The discharge instructions were translated to Portuguese, Haitian Creole, and Arabic by GPT-4o. Medical translators then evaluated the translations using both the Multidimensional Quality Metrics (MQM) framework and a preference scale.

Results: Translations of 14 to 20 discharge instructions for each of the 3 languages were evaluated. Noninferiority testing showed no significant difference in translation quality between GPT-4o and human translations for Portuguese and Arabic, but worse performance by GPT-4o than human translations for Haitian Creole (mean MQM score 90.5 ± SD 5.5 vs 97.0 ± SD 2.1). Linguists preferred the human translation over GPT-4o in 69% (SE = 7%) of the Haitian Creole cases.

Conclusions: In this cross-sectional study, GPT-4o translation performance of pediatric discharge instructions was noninferior for Portuguese and Arabic but was worse in Haitian Creole. Human validation of medical translations remains necessary for all languages, but large language models can be used to draft high-quality translations in languages with proven high performance, while resources are shifted toward more intensive post-editing of translations in languages with lower performance.