Disentangled adaptive fusion transformer using adversarial perturbation for egocentric action anticipation

Abstract

In recent years, egocentric action anticipation for wearable egocentric cameras has gained significant attention due to its ability to interpret objects and behaviors from a first-person perspective. However, the field faces challenges due to uncertainties arising from several sources: action-irrelevant information, semantically mixed representations of behaviors and objects, and the abrupt motion of the user. To address these challenges, we propose Ego-$A^4$ to enhance the robustness and reliability of egocentric action anticipation systems. First, Ego-$A^4$ selectively extracts action-relevant information to make efficient use of additional data beyond visual information. Second, Ego-$A^4$ generates effective disentangled representations for verbs and nouns using learnable behavior and object queries. Finally, Ego-$A^4$ enhances the continuity between the present and future using adversarial perturbation. The experimental results on the EPIC-Kitchens-100 and EGTEA Gaze＋ datasets demonstrate that Ego-$A^4$ outperforms existing methods in terms of mean top-5 recall and top-1 accuracy, respectively.