MGCA-Net: Multi-Grained Category-Aware Network for Open-Vocabulary Temporal Action Localization

Abstract

Open-Vocabulary Temporal Action Localization (OV-TAL) aims to recognize and localize instances of any desired action categories in videos without explicitly curating training data for all categories. Existing methods mostly recognize action categories at a single granularity, which degrades localization performance. To address this issue, we propose a Multi-Grained Category-Aware Network (MGCA-Net) comprising a localizer, an action presence predictor, a base classifier, and a coarse-to-fine classifier. Specifically, the localizer generates category-agnostic action proposals. For these proposals, the action presence predictor and base classifier estimate the action presence score and probabilities over base categories, respectively. Based on these predictions, proposals are partitioned into base action instances, novel proposals, or discarded ones. The category of each base action instance is determined by the class with the highest probability from the base classifier. In contrast, the categories of novel proposals are inferred by a coarse-to-fine classifier, which first identifies action presence at the video level—i.e., all action categories occurring in the input video—to obtain coarse categories, and then assigns each proposal to one category from them at the proposal level. Through coarse-to-fine awareness from video to proposal granularity for novel actions, and base classifier awareness of base actions at snippet granularity, MGCA-Net achieves multi-grained category modeling, effectively improving localization performance. Comprehensive evaluations on the THUMOS’14, ActivityNet-1.3, and FineAction benchmarks demonstrate competitive performance. Moreover, MGCA-Net also yields strong results under the Zero-Shot Temporal Action Localization (ZS-TAL) setting, with its advantage becoming more pronounced at higher tIoU thresholds. Our code is available at https://github.com/zhenyingfang/MGCA-Net