Average Sparse Attention for Dense Video Captioning From Multiperspective Edge-Computing Cameras

Abstract

In recent years, the artificial intelligence of things (AIoT) has accelerated the development of edge computing. Since existing edge computing for dense video captioning has only explored single-camera decision-making, we propose a lightweight image stitching model that uses a proposed inverted pruned residual model to realize multicamera decision-making to generate more accurate captions. Existing dense video captioning uses an intensive attention mechanism, which readily results in the loss of important information. Thus, our study proposes an average sparse attention mechanism such that the resultant dense video-captioning model is better able to focus on important information and improve the quality of its generated captions. The experiments show that the lightweight video stitching model can reduce model parameters by 13.40% and increase frames per second by 28.96% on an edge platform when compared to the latest studies. Furthermore, a dense video caption network with the average sparse attention mechanism yielded improvements of 22.97% for BLEU3, 35.04% for BLEU4, and 7.51% for METEOR.