MACR-afford: Weakly supervised multimodal affordance grounding via multi-branch attention enhancement and CoT multi-stage reasoning

Multimodal affordance grounding plays a crucial role in enabling computer systems to understand and recognize the functions and potential uses of objects. Affordance grounding involves not only identifying the shape and appearance of objects, but also understanding how they interact with the environment and users. However, current research faces challenges in accurately localizing affordance regions and addressing previously unseen scenarios. To address these challenges, we propose a weakly supervised multimodal affordance grounding framework, termed MACR-Afford, which combines a Multi-Branch Attention Enhancement module and a multi-stage Chain-of-Thought reasoning module. Under the guidance of a large language model, MACR-Afford improves the ability of intelligent agents to recognize and utilize objects in complex environments. First, we introduce a Multi-Branch Attention Enhancement (MBAE) module to improve the complementarity among object features. By enhancing cross-branch attention and extracting complementary discriminative features, MBAE enables more accurate localization of affordance regions. Subsequently, we introduce a Chain-of-Thought multi-stage reasoning module to generates general affordance knowledge units, which are used to guide the model in localizing affordance-relevant regions. Comprehensive experiments demonstrate that MACR-Afford consistently achieves superior performance in both seen and unseen scenarios, surpassing state-of-the-art baselines across multiple evaluation metrics. The code is publicly available at: https://github.com/HuiHui-Robot/MACR-Afford.