How to measure disagreement as a premise for learning from controversy in a social media context

Learning scenarios building on disagreement in a learning group or a whole classroom are well established in modern pedagogy. In the specific tradition of collaborative learning, such approaches have been traced back to theories of socio-cognitive conflict and have been associated with argumentative learning interactions. An important premise for these types of learning scenarios is the identification of disagreement. In the spirit of learning analytics, this calls for analytic tools and mechanisms to detect and measure disagreement in learning groups.Our mathematical analysis of several methods shows that methods of different origin are largely equivalent, only differing in the normalization factors and ensuing scaling properties. We have selected a measure that scales best and applied it to a target scenario in which learners judged types and levels of “toxicity” of social media content using an interactive tagging tool. Due restrictions imposed by the pandemic, we had to replace the originally envisaged classroom scenario by online experiments. We report on two consecutive experiments involving 42 students in the first and 89 subjects in the second instance. The results corroborate the adequacy of the measure in combination with the interactive, game-based approach to collecting judgements. We also saw that a revision of categories after the first study reduced the ambiguity. In addition to applying the disagreement measure to the learner judgements, we also assessed several personality traits, such as authoritarianism and social closeness. Regarding the dependency of the learner judgements on personality traits, we could only observe a weak influence of authoritarianism.