CITAL: Counterfactual intervention for temporal action localization with point-level annotation

Abstract

Point-supervised temporal action localization (PTAL) requires only a timestamp annotated on each action instance for training. Most existing PTAL methods use multiple instances learning (MIL) paradigm that localize actions according to contributions of the snippets to the classification results. The gap between classification and localization tasks causes the models to focus more on clues than pure actions. And the models are prone to localize fake actions when there are biased clues between training and test datasets. Inspired by earlier efforts on causal inference, we propose a counterfactual intervention framework for PTAL, CITAL for short. Counterfactual intervention simulates how models respond to counterfactual inputs that contains the same clues without action instances. Intuitively, we can obtain the real response to the pure actions by comparing responses to the inputs before and after counterfactual intervention. Specifically, we propose a background suppression (BS) block to suppresses the background response by guiding the model pay more attention to action instances rather than clues. To fuse the output scores of the various inputs, we propose a fusing by imitation (FI) strategy that further modifies the scores to have a high response to actions and low response to the background segments, generating more accurate proposals. Besides, we propose a counterfactual example generation (CEG) block to generate counterfactual examples with only clues and background contents based on the point labels and snippet-level action scores. Our approach achieves significant mAP gains on THUMOS14, BEOID and GTEA benchmarks comparing to various CAS-based methods without introducing additional parameters.