Toward Automatic Market Making: An Imitative Reinforcement Learning Approach With Predictive Representation Learning

Market making (MM) is a crucial trading problem, where a market maker stands ready to buy and sell the asset at a publicly quoted price to provide market liquidity continuously. The primary challenges in market making include position risk, liquidity risk, and adverse selection. Emerging research works investigate applying reinforcement learning (RL) techniques to derive automatic MM strategies. However, existing methods mainly focus on addressing inventory risk using only single-level quotes, which restricts the trading flexibility. In this paper, we shed light on the optimization of market makers' returns under a smaller risk while ensuring market liquidity and depth. This paper proposes a novel RL-based market-making strategy Predictive and Imitative Market Making Agent (PIMMA). First, to ensure adequate liquidity, we design an action space to enable stably allocating orders of multi-level volumes and prices. Beyond that, we apply queue position information from these multi-price levels to encode them in the state representations. Second, aiming at alleviating adverse selection, we draw auxiliary signals into state representation and design a representation learning network structure to catch implicit information from the price-volume fluctuations. Finally, we develop a novel reward function to earn a fortune while avoiding holding a large inventory. With a provided expert demonstration, our method augments the RL objective with imitation learning and learns an effective MM policy. Experiments are conducted to evaluate the proposed method based on realistic historical data, and the results demonstrate PIMMA outperforms RL-based strategy in the perspectives of earning decent revenue and information by adopting the multi-risk aversion strategy.