Long and Short-Term Collaborative Decision-Making Transformer for Online Action Detection and Anticipation

Abstract

Online Action Detection (OAD) and Online Action Anticipation (OAA) are all based on recognizing recent actions in historical videos without utilizing any future information. Existing methods use remote videos to obtain additional visual clues. However, remote videos may also include irrelevant videos that could attract attention, causing misinterpretation of recent actions. To this end, we propose a dual-path collaborative decision-making framework, which integrates one path that exclusively accesses recent videos with another path that can access both recent and remote videos, enabling it to correct low-confidence results misled by irrelevant remote content. On this basis, we propose a unified model for OAD and OAA, named Collaborative Decision-Making Transformer (CDM-Tr), which includes (1) a long-term history-based LT-Path that utilizes remote videos to assist in recognizing actions in recent videos, (2) a short-term history-based ST-Path that relies only on recent videos to recognize recent actions, and (3) a Multi-Task Classifier that makes collaborative decisions based on the weighted summation of these two paths. CDM-Tr achieves state-of-the-art performance on THUMOS’14 (OAD:72.6%, OAA:59.2%) and TVSeries (OAD:89.8%, OAA:84.2%). Meanwhile, the effectiveness and flexibility of the collaborative decision-making framework are further demonstrated.