A diachronic language model for long-time span classical Chinese

Classical Chinese literature, with its long history spanning thousands of years, serves as an invaluable resource for historical and humanistic studies. Previous classical Chinese language models have achieved significant progress in semantic understanding. However, they largely neglected the dynamic evolution of language across different historical eras. In this paper, we introduce a novel diachronic pre-trained language model tailored for classical Chinese texts. This model utilizes a time-based transformer architecture that captures the continuous evolution of semantics over time. Moreover, it adeptly balances the contextual and temporal information, minimizing semantic ambiguities from excessive time-related inputs. A high-quality diachronic corpus for classical Chinese is developed for training. This corpus spans from the pre-Qin dynasty to the Qing dynasty and includes a diverse array of genres. We validate its effectiveness by enriching a well-known classical Chinese word sense disambiguation dataset with additional temporal annotations. The results demonstrate the state-of-the-art performance of our model in discerning classical Chinese word meanings across different historical periods. Our research helps linguists to rapidly grasp the extent of semantic changes across different periods from vast corpora.¹