A Reinforcement Learning Based 3D Guided Drilling Method: Beyond Ground Control

Abstract

The current drilling guide operation relies on the two-way transmission of signals between the downhole drilling tools and the ground control center. However, the downhole environment is sometimes not conducive to such real-time signal transmission, and the analysis and decision-making in the ground involves complex human expert analysis and fine management. To deal with these problems, this paper proposes a downhole self-steering guided drilling method based on a reinforcement learning framework to achieve the 3D well trajectory design and control in real-time. In every time interval of the drilling process, the proposed system evaluates the drilling status and gives the adjustment action of drill bit in 3D space according to the received data, guiding the drill bit to the target reservoir without the involvement of human. The main module is a modified deep Q network using Sarsa algorithm for online self-learning. The experimental results show that after training, the drill bit is increasingly able to select control actions closer to the target reservoir. The frequency of effective actions is approximately 258% higher after the algorithm converges. The proposed system has the ability of online self-learning, which can automatically adjust the evaluation and decision models without manual monitoring.