Using In-game Analytics to Explore Learning Dynamics of Information Literacy in a Social Media Simulator

The emergence of Game-Based Learning (GBL) strategies to promote critical thinking has generated a growing need for analytical tools able to assess their effectiveness. Current methods typically apply qualitative approaches such as focus groups and survey-based questionnaires evaluating players’ knowledge, skills, and attitudes before and after playing the game; these methods are flexible, and they provide valuable information, yet new methods are needed to improve our understanding of what is happening during a gameplay session. The challenge of understanding the internal dynamics of GBL are illustrated by the open debate on gamified disinformation inoculators. These educational tools teach players how to identify disinformation by training them with a small set of fake news displayed within a gaming experience. There is an ongoing debate on what exactly is being learnt with these inoculators as some studies suggest a positive effect while other ones reveal that they promote scepticism instead of resistance against fake news. It is argued that new analytical methods are required to capture the learning process: detailed data collection on gameplay would be extremely useful to identify the most useful traits of Game-Based Learning, while detecting potential limitations or negative effects of this valuable learning resource. This work presents a novel framework to assess GBL dynamics grounded on data analytics. The approach uses the potential of the Unity in-game analytics platform to collect detailed data on how players tackle the challenges posed by the game mechanics; this diverse information may include the full set of decisions and interactions of the player as well as additional information such as the number of tries or time lapse between interactions. The behavioural data is then combined with content metadata information to infer general learning dynamics amongst players. This analytical framework is applied to a social media simulator included in the video game Julia: A Science Journey. Results suggest that the framework can reveal novel insights including aspects such as the level of engagement of the players, the impact of the type of content on the correct assessment of fake news, and the relation between reading speed and performance.