CL-HOI: Cross-level human–object interaction distillation from multimodal large language models

Abstract

Human–object interaction (HOI) detection often relies on labor-intensive annotations, but multimodal large language models (MLLMs) show potential for recognizing and reasoning about image-level interactions. However, MLLMs are typically computationally heavy and lack instance-level HOI detection capabilities. In this paper, we propose a cross-level HOI distillation (CL-HOI) framework that distills instance-level HOI detection from MLLMs, expanding HOI detection without labor-intensive and expensive manual annotations. Our approach uses CL-HOI as a student model to distill HOIs from a teacher MLLM in two stages: context distillation, where a visual-linguistic translator (VLT) converts visual information into linguistic form, and interaction distillation, where an interaction cognition network (ICN) facilitates interaction reasoning. Contrastive distillation losses transfer image-level context and interactions to the VLT and ICN for instance-level HOI detection. Evaluations on the HICO-DET and V-COCO datasets show that our method outperforms existing weakly supervised approaches, demonstrating its effectiveness in HOI detection without manual annotations.