Examining the Effects of a Real-Time, Knowledge-Aware Tool for Academic Writing Assessment

The computer-supported writing assessment (CSWA) has been widely used to reduce instructor workload and provide real-time feedback. Interpretability of CSWA draws extensive attention because it can benefit the validity, transparency, and knowledge-aware feedback of academic writing assessments. This study proposes a novel assessment tool, EduNERScore, which enables raters to identify different types of domain knowledge from unstructured educational texts and strengthen the interpretation of assessment through knowledge-aware evidence. This research uses an educational experiment to examine raters’ validity, efficiency, interpretability and user perception with and without the support of EduNERScore. The findings show that the experimental group with the support of EduNERScore achieves human-expert validity, high efficiency, interpretability and good use perception, compared with the control group. Finally, based on the empirical results, this research discusses the technological and educational implications for developing and applying the interpretable CSWA system in education.