Online Complex Action Learning and User State Estimation for Adaptive Dialogue Systems

Dialogue Systems (DS) have been rapidly evolving during the last few years. In order for them to be able to adapt to their surroundings and to individual users, researchers have focused on adaptation techniques, giving rise to the field of Adaptive Dialogue Systems (ADS). One important sub field of ADS is learning what the system should say or do next. Most of the work done in this direction assumes the system performs simple actions, rather than behaviours described by sub-dialogues, i.e. complex actions. To this end we propose a novel online algorithm that ranks complex actions according to their performance and selects the top-k performing ones. To the best of our knowledge there is currently no work describing methods to learn policies for complex actions in an online fashion. We also propose an online algorithm able to estimate the effects of system actions on the user's state, in order to make more informed decisions about which action to take and guide the learning algorithm towards a desired final user state. Our results show that the proposed complex action learning algorithm outperforms simple Hierarchical Reinforcement Learning algorithms and that we are able to successfully estimate the effect an action will have on the user's state, using emotional states as a case study.