GPT Versus ERNIE for National Traditional Chinese Medicine Licensing Examination: Does Cultural Background Matter?

Abstract

Purpose: This study evaluates the performance of large language models (LLMs) in the context of the Chinese National Traditional Chinese Medicine Licensing Examination (TCMLE). Materials and Methods: We compared the performances of different versions of Generative Pre-trained Transformer (GPT) and Enhanced Representation through Knowledge Integration (ERNIE) using historical TCMLE questions. Results: ERNIE-4.0 outperformed all other models with an accuracy of 81.7%, followed by ERNIE-3.5 (75.2%), GPT-4o (74.8%), and GPT-4 turbo (50.7%). For questions related to Western internal medicine, all models showed high accuracy above 86.7%. Conclusion: The study highlights the significance of cultural context in training data, influencing the performance of LLMs in specific medical examinations.