Artificial Intelligence in Academic Translation: A Comparative Study of Large Language Models and Google Translate

Abstract

Purpose. The advent of Large Language Model (LLM), a generative artificial intelligence (AI) model, in November 2022 has had a profound impact on various domains, including the field of translation studies. This motivated this study to conduct a rigorous evaluation of the effectiveness and precision of machine translation, represented by Google Translate (GT), in comparison to Large Language Models (LLMs), specifically ChatGPT 3.5 and 4, when translating academic abstracts bidirectionally between English and Arabic. Methods. Employing a mixed-design approach, this study utilizes a corpus comprising 20 abstracts sourced from peer-reviewed journals indexed in the Clarivate Web of Science, specifically the Journal of Arabic Literature and Al-Istihlal Journal. The abstracts are equally divided to represent both English-Arabic and Arabic-English translation directionality. The study’s design is rooted in a comprehensive evaluation rubric adapted from Hurtado Albir and Taylor (2015), focusing on semantic integrity, syntactic coherence, and technical adequacy. Three independent raters carried out assessments of the translation outputs generated by both GT and LLM models. Results. Results from quantitative and qualitative analyses indicated that LLM tools significantly outperformed MT outputs in both Arabic and English translation directions. Additionally, ChatGPT 4 demonstrated a significant advantage over ChatGPT 3.5 in Arabic-English translation, while no statistically significant difference was observed in the English-Arabic translation directionality. Qualitative analysis findings indicated that AI tools exhibited the capacity to comprehend contextual nuances, recognize city names, and adapt to the target language's style. Conversely, GT displayed limitations in handling specific contextual aspects and often provided literal translations for certain terms.