SportSummarizer: A unified multimodal fusion transformer for context-aware sports video summarization

Abstract

Automated sports video summarization faces critical challenges due to the complexity of dynamic gameplay, event variability, and the intricate rules governing sports like cricket and soccer. Existing methods often struggle to capture key moments accurately, resulting in false positives, redundant content like replays, ineffective multimodal data integration, and challenges in spatio-temporal modeling and semantic event understanding. To overcome these limitations, a novel Unified Multimodal Fusion Transformer is proposed for the summarization of cricket and soccer videos. This approach utilizes advanced feature encoding across multiple modalities: ViViT for video, OpenL3 for audio, and DistilBERT for text, ensuring robust multimodal representations. A multimodal fusion transformer with contextual cross-quadrimodal attention is introduced to address weak multimodal integration, enabling the model to capture complex interactions across visual, audio, and textual data for precise event detection. Further, Hierarchical Temporal Convolutional Networks (Hierarchical TCN) module integrates hierarchical temporal and metadata-enhanced positional to model both short and long-term game sequences effectively. Additionally, replay and redundancy elimination mechanisms are applied to remove repetitive content, generating concise and high-quality video summaries that reflect the game's critical moments. The proposed method achieves state-of-the-art results, with the highest precision (99.2 %) and recall (98.9 %), and a low error rate (4 %). It also demonstrates superior ROC-AUC performance (0.88) and maintains peak accuracy (89.5 %) with strong performance in mIoU (0.82) and highlight diversity (0.93), highlighting its robustness across various event detection metrics.