Engagement, Metrics and Personalisation: the Good, the Bad and the Ugly

Abstract

User engagement plays a central role in companies and organisations operating online services. A main challenge is to leverage knowledge about the online interaction of users to understand what engage them short-term and more importantly long-term. Two critical steps of improving user engagement are defining the right metrics and properly optimising for them. A common way that engagement is measured and understood is through the definition and development of metrics of user satisfaction, which can act as proxy of short-term user engagement, mostly at session level. In the context of recommender systems, developing a better understanding of how users interact (implicit signals) with them during their online session is important for developing metrics of user satisfaction. Detecting and understanding implicit signals of user satisfaction are essential for enhancing the quality of the recommendations. When users interact with the recommendations served to them, they leave behind fine-grained traces of interaction patterns, which can be leveraged to predict how satisfying their experience was. This talk will present various works and personal thoughts on how to measure user engagement. It will discuss the definition and development of metrics of user satisfaction that can be used as proxy of user engagement, and will include cases of good, bad and ugly scenarios. An important message will be to show that, when aiming to personalise the recommendations, it is important to consider the heterogeneity of both user and content to formalise the notion of satisfaction, and in turn design the appropriate satisfaction metrics to capture these.