Improved Consistency in Price Negotiation Dialogue System Using Parameterized Action Space with Generative Adversarial Imitation Learning

Abstract

A price negotiation dialogue system should not only gain profit but also pay attention to the negotiation process, such as the consistency of price proposals. In particular, when using a parameterized action space to make price proposals as a continuous action, the proposals can be more flexible but potentially inconsistent. In this study, we propose introducing Generative Adversarial Imitation Learning (GAIL) to price negotiation dialogues with a parameterized action space. To the best of our knowledge, this is the first case study to introduce GAIL in parameterized action space. In addition, we work on extending the dialogue act to maintain consistency, and on combining parameteried action reinforcement learning(RL) and GAIL by using Multi-Critic. The proposed method is applied to the CRAIGSLISTBARGAIN negotiation task, which is a practical negotiation task and is trained in a multi-agent format as a seller and buyer without using a simulator, and is evaluated by interacting with agents that combine supervised learning and rule-based methods. The results show that GAIL can reduce price inconsistencies better than RL with the designed reward function for consistency maintenance or the combination of RL and behavior cloning. Furthermore, we confirmed that the combined method of RL with the designed reward function and GAIL can reduce price inconsistencies the most and also enhance profit-seeking abilities.