DEFT-DPT: A Lightweight Multimodal Framework for Egocentric Video Action Recognition

Abstract

Egocentric video analysis has emerged as an active and rapidly growing area of research in computer vision and multimedia due to its ability to capture human actions from a first-person perspective rich in visual and motion cues. This paper presents a lightweight multimodal framework for action localization and recognition in egocentric videos. Existing approaches often fail to effectively leverage complementary information from RGB, motion, and depth modalities, and even when multiple cues are fused, they typically lack adaptive integration, accurate localization, and scalability for large datasets. To overcome these challenges, we introduce the Dynamic Egocentric Feature Transformation (DEFT) module, which adaptively encodes features while preserving the distinctive motion and viewpoint characteristics of egocentric videos. DEFT emphasizes on central and action-relevant regions through spatial alignment and Gaussian-based weighting, enabling precise action localization and robust recognition. Furthermore, a lightweight Co-Attention module adaptively fuses complementary modality features through a cross-modal agreement. Next, we construct a Sparse Video Similarity Graph (SVSG) in the deep feature space using ResNet-50, where the central frame regions serve as graph nodes. To maintain computational efficiency, Dynamic Percentile Thresholding (DPT) retains only the most informative graph edges, yielding a sparse, yet meaningful graph representation. The resulting graph is then processed by a Graph Attention Network (GAT) for context-aware learning and action classification. Our lightweight framework, containing only 34.22M parameters, achieves 46.25% lower computation time compared to large-scale pretrained foundation models, while maintaining competitive accuracy. Extensive experiments on four benchmark egocentric video datasets—EPIC-Kitchens (RGB + Optical Flow), MECCANO (RGB + Depth), GTEA (RGB), and ADL (RGB)—demonstrate the superiority of the proposed method, showing consistent improvements in different modalities and settings. Our implementation and code are publicly available at https://github.com/pawanesh-mnnit/deft_dpt