A Feature Analysis Tool for Batch RL Datasets

Batch RL is concerned about learning a decision policy from a given dataset without interacting with the environment. Although research is actively conducted on learning-related issues (e.g., convergence speed, stability, and safety), empirical challenges before learning are largely ignored. Many RL practitioners face the challenge of determining whether a designed Markov Decision Process (MDP) is valid and meaningful. This study proposes a model-based method to check whether an MDP designed for a given dataset is well formulated through a heuristic-based feature analysis. We tested our method in constructed as well as more realistic environments. Our results show that our approach can identify potential problems of data. As far as we know, performing validity analysis on batch RL data is a novel direction, and we envision that our tool serves as a motivational example to help practitioners apply RL more easily.