Domain over size: Clinical ELECTRA surpasses general BERT for bleeding site classification in the free text of electronic health records

Bleeding can be a life-threatening condition which occurs for 3.2% of medical patients. Information about previous bleeding and bleeding site is used to predict the risk of future bleeding and guide anticoagulant treatment. However, obtaining this information is a time-consuming task as it is contained in the free text of electronic health records. Previous research has mainly been focused on extracting bleeding events but does not classify the bleeding site which is important for assessing the severity of the bleeding. This study creates the first dataset for developing and evaluating machine learning models for classification of bleeding site. The dataset consists of sentences annotated by medical doctors as belonging to one of ten bleeding sites. The sentences were annotated in 149,523 electronic health record notes from 1,533 patients of Odense University Hospital, Denmark, between 2015 and 2020. We compare different deep learning models on classifying bleeding site and find that a ∼13M parameter ELECTRA model pretrained on clinical text achieves higher accuracy ($0.905\ \pm 0.002$) than a ∼110M parameter general BERT model ($0.884 \pm 0.001$) on a balanced test set of 1,500 sentences. We furthermore test different methods for dealing with unbalanced data without finding any significant differences between methods.