Encoder models at the European Patent Office: Pre-training and use cases

Abstract

Patents are organized using systems of technical concepts like the Cooperative Patent Classification. Classification information is extremely valuable for patent professionals, particularly for patent search. Language models have proven useful in Natural Language Processing tasks, including document classification. Generally, pre-training on a domain is essential for optimal downstream performance. Currently, there are no models pre-trained on patents with sequence length above 512. We pre-trained a RoBERTa model with sequence length 1024, increasing the fully covered claims sections from 12% to 53%. It has a ‘base’ configuration, reducing free parameters compared to ‘large’ models in the patent domain three-fold. We fine-tuned the model on classification tasks in the CPC, up to leaf level. Our tokenizer produces sequences on average 5% and up to 10% shorter than the general English RoBERTa tokenizer. With our pre-trained ‘base’ size model, we reach classification performance better than general English models, comparable to ‘large’ models pre-trained on patents. On the finest CPC granularity, 88% of test documents have at least one ground truth symbol in the top 10 predictions. Our CPC prediction models and data sets are publicly accessible. With the described procedures, we can periodically repeat pre-training and fine-tuning to cope with drift effects.