Hand-Object Interaction Reasoning

Abstract

This paper proposes an interaction reasoning network for modelling spatio-temporal relationships between hands and objects in egocentric video. The proposed interaction unit utilises a Transformer-style module to reason about each acting hand, and its spatio-temporal relations to the other hand as well as objects being interacted with. We show that modelling two-handed interactions are critical for action recognition in egocentric video, and demonstrate that by using positionally-encoded trajectories, the network can better recognise observed interactions. We train and evaluate our proposed network on large-scale egocentric EPIC-KITCHENS-100 and crowd-sourced Something-Else datasets, with an ablation study to showcase our proposal.