Zero-Shot Prediction of Conversational Derailment With Large Language Models

Abstract

Online discussion platforms often show a tendency for conversations to stray from the topic and devolve into personal attacks. Previous studies have trained machine learning algorithms to detect conversational derailment using supervised methods. However, creating the datasets required for supervised training is very costly. To address this challenge, we focus on the zero-shot performance of large language models (LLMs), which have advanced rapidly in recent years. This study aims to evaluate the zero-shot prediction performance of conversational derailment using LLMs. First, we measured the performance of the most commonly used LLMs in predicting conversational derailments and found that the zero-shot prediction performance is comparable to that of traditional fine-tuning approaches. Secondly, we explored the effect of inserting prior knowledge into the prompts on the behavior of the LLMs. We discovered that this practice does not necessarily improve the performance of LLMs, resulting in unexpected changes in prediction timing. This study’s contributions are as follows: We demonstrate that zero-shot inference with LLMs is an effective method for predicting conversational derailment in the absence of training data. Additionally, we found that altering the prompt to include prior knowledge leads to unintended results in the conversational derailment prediction task. This observation emphasizes the need to evaluate the impact of prompt changes from various perspectives, going beyond simple performance metrics.