How concept drift can impair the classification of fake news

Abstract

Fake news is a serious problem that can influence political choices, harm people's physical and mental health, promote treatments without scientific evidence, and even incite violence. Machine learning methods are one of the leading solutions that have been studied for filtering fake news automatically. However, most studies do not consider the dynamic nature of news, creating static models and evaluating them offline through the traditional holdout or cross-validation. These studies naively assume that news characteristics do not change over time and, therefore, the performance of offline models is preserved as time goes on. In this study, we show how concept drift can impair the classification of fake news. We aim to verify whether the conclusions obtained in studies that disregarded the dynamic nature of the news are sustained. We analyzed how the performance of methods trained in an offline fashion is affected by the news update over time, including concept drift due to impacting events like the Covid-19 pandemic and the United States presidential election. The results showed that the performance of offline models is over-optimistic. Incremental learning methods should be preferred because they can adapt to changes in textual patterns over time.