Graduated non-convex feature-metric-based 6D object pose refinement via deep reinforcement learning

Recently, many works focus on 6D object pose refinement with a single RGB image. Most of them apply the differentiable Levenberg–Marquardt (LM) algorithm as the solver. However, they may easily ignore the importance of the damping parameter denoted by $\lambda$, which affects the accuracy and efficiency of prediction. In this paper, we present a coarse-to-fine feature-metric-based 6D object pose refinement framework, which utilizes the intermediate layers to predict $\lambda$ combined with Region of Interest (ROI) alignment and eigenvalues. To facilitate better convergence during the training process, we propose to leverage graduated non-convexity (GNC) to handle uncertainty and feature residual learning in a pixel-level manner. Moreover, current works have not analyzed the control process during the whole iteration process. We propose to use deep reinforcement learning to fit this non-differentiable process, which can reduce redundant steps during the prediction stage. Finally, with a Transformer-based backbone, our algorithm with no iteration control learning (ICL) achieves better performance with Shape-constraint Recurrent Flow (SRF, state-of-the-art object pose refinement method) (Hai et al. 2023) on Linear Model for Object Detection (LineMOD), LineMOD Occlusion and YCB-Video datasets. Moreover, our full algorithm with VGG-16 as the backbone, accelerated with TensorRT, runs at about 94 FPS. It exhibits superior speed compared to RePose (Iwase et al. 2021), and notably surpasses its accuracy, especially for initial poses with large errors. The code will be available at https://github.com/NiPeiyuan/EARePOSE.git.