Caged CrafText: Language-Grounded Safety Constraints for Multimodal Reinforcement Learning in CrafText

Abstract

Safe reinforcement learning under natural language instructions remains challenging. Current benchmarks are limited by providing only safety constraints in natural language while using structured goal representations, and by focusing on simple navigation tasks in static environments. We introduce Caged CrafText, the first benchmark where both task objectives and safety constraints are specified entirely through natural language. Our benchmark features complex, long-horizon tasks requiring reasoning in dynamic environments, and is structured around two dataset levels: the Main dataset, designed to evaluate language understanding and safe exploration capabilities across aggregated constraint types, and the Debug dataset, which enables detailed diagnosis of specific failure modes in individual constraint scenarios. Evaluation of existing safety approaches reveals their insufficient safe exploration capability, highlighting the need for improved methods capable of handling complex language-guided constraints. Our experiments also demonstrate that while Large Language Model(LLM) agents show promising reasoning capabilities, they suffer from grounding issues that lead to frequent constraint violations.