Deep Q network with action retention for going long and short selling

Abstract

In computer-simulated games, the primary objective of adopting reinforcement learning is to achieve victory by attaining the highest hand-crafted reward, considering the optimal state-value functions across the promising trajectories. However, in the context of algorithmic trading, there is no clear goal for hand-crafting an extremely high reward for the state-value function. Besides, the exploration and exploitation of the reinforcement learning could generate a high number of unexpected buy and sell actions. These actions could lead to overlapped transactions which cannot provide a fair reward function. In order to alleviate these problems, we propose a novel trading algorithm named Deep Q Network with Action Retention (DQN-AR). Firstly, the action retention mechanism is proposed to avoid the overlapped transactions. Secondly, the divide-and-conquer approach is employed to break down the profit maximization goal into several sub-goals, with the aim of optimizing the annualized returns from all transactions throughout the entire trading period. Thirdly, we evaluate the effectiveness of the proposed approach by implementing the DQN-AR model for both long and short selling in algorithmic trading. In the experiments, we compare DQN-AR with DQN, Gated-DQN (GDQN), Simple Moving Average (SMA) and Dual Moving Average Crossover (DMAC). The experimental result shows that DQN-AR is superior to DQN, GDQN, SMA and DMAC and achieves the state-of-art trading performance both for long and short positions. In summary, our DQN-AR achieves 15.4% higher profit on average than the second top competitor approach for the long position and 101.03% higher on average for the short position.