Detecting covert disruptive behavior in online interaction by analyzing conversational features and norm violations

Abstract

Disruptive behavior is a prevalent threat to constructive online engagement. Covert behaviors, like trolling, are especially challenging to detect automatically, because they utilize deceptive strategies to manipulate conversation. We illustrate a novel approach to their detection: analyzing conversational structures instead of focusing only on messages in isolation. Building on conversation analysis, we demonstrate that 1) conversational actions and their norms provide concepts for a deeper understanding of covert disruption, and that 2) machine learning, natural language processing and structural analysis of conversation can complement message-level features to create models that surpass earlier approaches to trolling detection. Our models, developed for detecting overt (aggression) as well as covert (trolling) behaviors using prior studies’ message-level features and new conversational action features, achieved high accuracies (0.90 and 0.92, respectively). The findings offer a theoretically grounded approach to computationally analyzing social media interaction, and novel methods for effectively detecting covert disruptive conversations online.