FactRank: Developing automated claim detection for Dutch-language fact-checkers

Abstract

Fact-checking has always been a central task of journalism, but given the ever-growing amount and speed of news offline and online, as well as the growing amounts of misinformation and disinformation, it is becoming increasingly important to support human fact-checkers with (semi-)automated methods to make their work more efficient. Within fact-checking, the detection of check-worthy claims is a crucial initial step, since it limits the number of claims that require or deserve to be checked for their truthfulness.

In this paper, we present FactRank, a novel claim detection tool for journalists specifically created for the Dutch language. To the best of our knowledge, this is the first and still the only such tool for Dutch. FactRank thus complements existing online claim detection tools for English and (a small number of) other languages. FactRank performs similarly to claim detection in ClaimBuster, the state-of-the-art fact-checking tool for English. Our comparisons with a human baseline also indicate that given how much even expert human fact-checkers disagree, there may be a natural “upper bound” on the accuracy of check-worthiness detection by machine-learning methods.

The specific quality of FactRank derives from the interdisciplinary and iterative process in which it was created, which includes not only a high-performance deep-learning neural network architecture, but also a principled approach to defining and operationalising the concept of check-worthiness via a detailed codebook. This codebook was created jointly by expert fact-checkers from the two countries that have Dutch as an official language (Belgium/Flanders and the Netherlands). We expect FactRank to be very useful exactly because of the way we defined check-worthiness, and because of how we have made this explicit and traceable.