Assessing text comprehension proficiency: Indonesian higher education students vs ChatGPT

Abstract

AI has developed rapidly. However, AI research in applied linguistics in the field of language education in Indonesia still needs to be expanded to reading and writing skills. This research aims to explore students' skills in writing summaries and understanding the historical theme of the development of the Indonesian language with AI based on gender and university aspects. The quantitative method uses descriptive statistical analysis techniques, independent sample t-test, and Welch One-Way ANOVA. The research sample was 288 students from Makassar State University, Timor University, and Makassar Health Polytechnic. The results show that ChatGPT is significantly better at reprocessing text than students based on the aspects measured. ChatGPT outperforms almost every aspect of the assessment. However, in the MCT_Score aspect, the average for Universitas Negeri Makassar students is slightly higher than ChatGPT and the other two universities. Meanwhile, the Makassar Health Polytechnic almost matches the average ChatGPT score. Apart from that, the Universitas Timor average seems significantly different, with a score range of only 6.00 – 7.00. This research contributes to developing the Indonesian curriculum using Artificial Intelligence (AI) technology. The government can use these findings as a basis for making better policies to improve the quality of education. This research implies that Indonesian students have a gap in understanding texts compared to ChatGPT. The first implication is the need to revise and develop the educational curriculum. Therefore, future research can examine text comprehension abilities in more specific contexts, such as scientific texts, journalism, literature, or specific scientific disciplines. It can provide more detailed insight into students' strategies for overcoming difficulties in understanding texts. In addition, future research will be conducted on the broader impact of using Artificial Intelligence technology in language education on the development of student text comprehension and the potential social and ethical impacts.