Extending the Evaluation Paradigm for Personalisation: A Categorisation Use Case

The Conlan whitepaper makes a compelling case for a community-scale means of evaluating algorithms for user modeling, adaptation and personalisation (UMAP). The authors propose an evaluation paradigm focused on a personalisation use case within an open modeling environment. Their use case is one where a mobile interface learns a user's preferences for different notifications in different contexts. Additional use cases could be incorporated in this paradigm to provide contrasting kinds of challenge for personalisation evaluation. In this position paper we summarise a use case where personalisation is achieved by categorising a user and switching their interface to a variant. The user interface is a mobile news app within a platform that also comprises a user modeling function and an interface personalisation service. Comparison of the use cases helps to map the space for the evaluation paradigm.