New Public Dataset for Classification of Inappropriate Comments in Slovak language

Discussion sections under online news articles enable the exchange of opinions and information between discussants and the provision of information to potential visitors and readers. Discussion moderators have the task of guarding the level of discussions and preventing the publication of harmful content (vulgarity, hate speech, abuse, etc.). The process of moderating discussions should ideally be predictable and based on clear, predetermined rules of conduct. Discussion participants should not have the feeling of either rigid censorship or passive toleration of inappropriate content. Manual moderation is time-consuming and not particularly popular among newspaper workers, especially when there is a large number of comments. For the above reasons, we decided to propose attributes that would help to detect inappropriate content automatically and perform machine learning for discussion comments, in order to guarantee both a certain degree of predictability and uni-formity in the classification of inappropriate comments and, secondly, to offer a list of potential machine learning models that are appropriate for the task. For evaluation purposes, we collected an entirely new dataset with 2,283 inappropri-ate comments (marked as such by human moderators) and 10,000 ordinary comments in the Slovak language from past discussions on a major Slovak online news portal. We offer this dataset freely to other researchers to develop and test their own algorithms. By way of illustrative example we have been able to classify inappropriate comments with a macro F-measure of 67.3% on the basis of various statistical attributes, which we consider an encouraging result given that Slovak is a highly inflected synthetic language.