ForestryBERT: A pre-trained language model with continual learning adapted to changing forestry text

Efficient utilization and enhancement of the growing volume of forestry-related textual data is crucial for advancing smart forestry. Pre-trained language models (PLMs) have demonstrated strong capabilities in processing large unlabeled text. To adapt a general PLM to a specific domain, existing studies typically employ a single target corpus for one-time pre-training to incorporate domain-specific knowledge. However, this approach fails to align with the dynamic processes of continuous adaptation and knowledge accumulation that are essential in real-world scenarios. Here, this study proposes ForestryBERT, a BERT model that is continually pre-trained on three Chinese forestry corpora comprising 204,636 texts (19.66 million words) using a continual learning method called DAS.¹ We evaluate the model on both text classification and extractive question answering tasks using five datasets for each task. Experimental results show that ForestryBERT outperforms five general-domain PLMs and further pre-trained PLMs (without DAS) across eight custom-built forestry datasets. Moreover, PLMs using DAS exhibit a forgetting rate of 0.65, which is 1.41 lower than PLMs without DAS, and demonstrate superior performance on both new and old tasks. These findings indicate that ForestryBERT, based on continual learning, effectively mitigates catastrophic forgetting and facilitates the continuous acquisition of new knowledge. It expands its forestry knowledge by continually absorbing new unlabeled forestry corpora, showcasing its potential for sustainability and scalability. Our study provides a strategy for handling the growing volume of forestry text during PLM construction, a strategy that is also applicable to other domains.