EmoThreat@FIRE2022: Shared Track on Emotions and Threat Detection in Urdu

Abstract

Many languages with a wealth of resources have been researched to solve the challenges of emotion and targeted abuse detection, i.e. threat. But when it comes to languages, such as Urdu, it is noted that there is a severe lack of both resources and approaches in terms of Urdu language processing. Therefore, this study concentrated on offering resources for Urdu by organizing a shared task called “EmoThreat: Emotions and Threat detection in Urdu". The task offered two tasks: (i) multi-label emotion classification (Task A), and (ii) binary threat detection (Task B). Task B was a multi-class problem since it was further subdivided into the identification of threats posed by groups and individuals. This paper provides an overview of the methodology and results obtained by each of the 10 distinct teams who participated in the shared task. In addition, each group presented a detailed error analysis as part of their submission for the best model. The top-performing system in Task A received a macro-F1 score of 0.687. In contrast, subtask 1 of Task B received a score of 0.716 macro-F1 while subtask 2 of Task B obtained a 0.539 macro-F1 score.