Provoking critical thinking: Using counter-arguments in online discussion summarisation

Generative AI systems based on Large Language Models (LLMs), like ChatGPT, have brought profound convenience to users thanks to their ability to summarise existing documents and to generate new text. This shows the potential to summarise online human discussions or debates for new entrants to quickly comprehend the ongoing matters and arguments and to get efficiently involved in the opinion deliberation process. However, generative AI has frequently been associated with negatively affecting users’ decision making. In this paper, we study a novel approach based on generative AI to trigger users’ critical thinking by challenging fresh counter-arguments after summarising existing online discussions for incoming users. We conduct a user study with 558 participants to determine the effectiveness and fairness of AI summarisation across three online platforms — Reddit, Kialo, and Debatewise. Our results show that the intervention methods and platform differences are strongly associated with participants’ level of opinion change and the strength of their belief. We found that participants’ opinion changes affected their perceived usefulness of the AI system. Our work opens the door to LLM applications helping Web users participate in online opinion deliberation more efficiently, with a higher level of critical thinking, and with a reduced negative attitude.