Deep Reinforcement Learning Approach Using Customized Technical Indicators for A Pre-emerging Market: A Case Study of Vietnamese Stock Market

Abstract

The Vietnamese stock market is a challenge for applying algorithmic trading. However, the advance of Machine Learning, especially Reinforcement Learning, has provided a new opportunity to develop better trading models. In this work, we proposed an ensemble strategy, namely S-A-P, combining three deep reinforcement learning models (i.e., SAC, PPO, and A2C). The best model at each quarter is used for trading in the incoming quarter based on the Sharpe ratio. Moreover, the list of technical indicators is also proposed to represent the variation in this market. Our approach shows better performance than the baseline and VN30INDEX in both profits (55% in cumulative return) and risk management (0.77 in Sharpe ratio). Additionally, this approach can perform appropriately during two high-turbulence periods, which the baseline cannot detect. The extension of this work may consider a novel Machine Learning approach for representing the stock market and a different metric for building ensemble strategy.