Measuring Media Bias via Masked Language Modeling

Bias in news reporting can lead to tribalism and division on important issues. Scalable and reliable measurement of such biases is an important first step in addressing them. In this work, based on the intuition that media bias is captured by the tone and word choices in articles, we propose a framework for modeling the relative bias of media outlets through masked token prediction via large-scale pretrained masked language models fine-tuned on articles form news outlets. Through experiments on five diverse and politically polarized topics we show that our framework can capture media bias towards these topics with high reliability. Additionally, our experiments show that our framework is general, in that language models fine-tuned on one topic can be applied to other topics with little drop in performance.