Predicting chronic diseases using clinical notes and fine-tuned transformers

Abstract

Electronic health records(EHR) have been used extensively by researchers lately to gain insights and use them as clinical informatics. EHR data contains structured data, as a result of having information systems in-place, and also unstructured data like clinical notes. These unstructured data have a huge scope of exploration and can derive meaningful insights. Challenges exists like the heterogeneous and multi modal nature of such data. This work provides insights into the EHR data, the datasets available for research, the tasks that can be performed on them, the methods that can be applied on them, and then demonstrates how BERT and DistilBERT can be fine-tuned on the medical datasets to predict chronic diseases like asthma, renal diseases, heart diseases and arthritis and how DISTILBERT can be a preferred option over BERT. Both the models BERT and DISTILBERT have been pre-trained and then fine tuned to predict the chronic diseases from the clinical notes.