{"title":"面向任务的实时语义分割视频压缩流","authors":"Xuedou Xiao;Yingying Zuo;Mingxuan Yan;Wei Wang;Jianhua He;Qian Zhang","doi":"10.1109/TMC.2024.3446185","DOIUrl":null,"url":null,"abstract":"Real-time semantic segmentation (SS) is a major task for various vision-based applications such as self-driving. Due to the limited computing resources and stringent performance requirements, streaming videos from camera-embedded mobile devices to edge servers for SS is a promising approach. While there are increasing efforts on task-oriented video compression, most SS-applicable algorithms apply more uniform compression, as the sensitive regions are less obvious and concentrated. Such processing results in low compression performance and significantly limits the capacity of edge servers supporting real-time SS. In this paper, we propose STAC, a novel task-oriented DNN-driven video compressive streaming algorithm tailed for SS, to strike accuracy-bitrate balance and adapt to time-varying bandwidth. It exploits DNN's gradients as sensitivity metrics for fine-grained spatial adaptive compression and includes a temporal adaptive scheme that integrates spatial adaptation with predictive coding. Furthermore, we design a new bandwidth-aware neural network, serving as a compatible configuration tuner to fit time-varying bandwidth and content. STAC is evaluated in a system with a commodity mobile device and an edge server with real-world network traces. Experiments show that STAC can save up to 63.7–75.2% of bandwidth or improve accuracy by 3.1–9.5% compared to state-of-the-art algorithms, while capable of adapting to time-varying bandwidth.","PeriodicalId":50389,"journal":{"name":"IEEE Transactions on Mobile Computing","volume":"23 12","pages":"14396-14413"},"PeriodicalIF":7.7000,"publicationDate":"2024-09-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Task-Oriented Video Compressive Streaming for Real-Time Semantic Segmentation\",\"authors\":\"Xuedou Xiao;Yingying Zuo;Mingxuan Yan;Wei Wang;Jianhua He;Qian Zhang\",\"doi\":\"10.1109/TMC.2024.3446185\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"Real-time semantic segmentation (SS) is a major task for various vision-based applications such as self-driving. Due to the limited computing resources and stringent performance requirements, streaming videos from camera-embedded mobile devices to edge servers for SS is a promising approach. While there are increasing efforts on task-oriented video compression, most SS-applicable algorithms apply more uniform compression, as the sensitive regions are less obvious and concentrated. Such processing results in low compression performance and significantly limits the capacity of edge servers supporting real-time SS. In this paper, we propose STAC, a novel task-oriented DNN-driven video compressive streaming algorithm tailed for SS, to strike accuracy-bitrate balance and adapt to time-varying bandwidth. It exploits DNN's gradients as sensitivity metrics for fine-grained spatial adaptive compression and includes a temporal adaptive scheme that integrates spatial adaptation with predictive coding. Furthermore, we design a new bandwidth-aware neural network, serving as a compatible configuration tuner to fit time-varying bandwidth and content. STAC is evaluated in a system with a commodity mobile device and an edge server with real-world network traces. Experiments show that STAC can save up to 63.7–75.2% of bandwidth or improve accuracy by 3.1–9.5% compared to state-of-the-art algorithms, while capable of adapting to time-varying bandwidth.\",\"PeriodicalId\":50389,\"journal\":{\"name\":\"IEEE Transactions on Mobile Computing\",\"volume\":\"23 12\",\"pages\":\"14396-14413\"},\"PeriodicalIF\":7.7000,\"publicationDate\":\"2024-09-18\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"IEEE Transactions on Mobile Computing\",\"FirstCategoryId\":\"94\",\"ListUrlMain\":\"https://ieeexplore.ieee.org/document/10684013/\",\"RegionNum\":2,\"RegionCategory\":\"计算机科学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"COMPUTER SCIENCE, INFORMATION SYSTEMS\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"IEEE Transactions on Mobile Computing","FirstCategoryId":"94","ListUrlMain":"https://ieeexplore.ieee.org/document/10684013/","RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMPUTER SCIENCE, INFORMATION SYSTEMS","Score":null,"Total":0}
引用次数: 0
摘要
实时语义分割(SS)是自动驾驶等各种基于视觉的应用的一项重要任务。由于有限的计算资源和严格的性能要求,从嵌入摄像头的移动设备流式传输视频到边缘服务器进行语义分割是一种很有前景的方法。虽然人们在面向任务的视频压缩方面做出了越来越多的努力,但大多数适用于 SS 的算法都采用了更均匀的压缩方式,因为敏感区域不太明显和集中。这种处理方式导致压缩性能低下,极大地限制了支持实时 SS 的边缘服务器的能力。在本文中,我们提出了 STAC,这是一种新颖的面向任务的 DNN 驱动视频压缩流算法,专为 SS 量身定制,以实现精度与比特率之间的平衡,并适应随时间变化的带宽。该算法利用 DNN 的梯度作为灵敏度度量,实现细粒度空间自适应压缩,并包含一个将空间自适应与预测编码相结合的时间自适应方案。此外,我们还设计了一种新的带宽感知神经网络,作为兼容配置调节器,以适应时变带宽和内容。STAC 在一个使用商品移动设备和边缘服务器的系统中通过真实的网络跟踪进行了评估。实验表明,与最先进的算法相比,STAC 能节省高达 63.7-75.2% 的带宽,或将准确率提高 3.1-9.5%,同时还能适应随时间变化的带宽。
Task-Oriented Video Compressive Streaming for Real-Time Semantic Segmentation
Real-time semantic segmentation (SS) is a major task for various vision-based applications such as self-driving. Due to the limited computing resources and stringent performance requirements, streaming videos from camera-embedded mobile devices to edge servers for SS is a promising approach. While there are increasing efforts on task-oriented video compression, most SS-applicable algorithms apply more uniform compression, as the sensitive regions are less obvious and concentrated. Such processing results in low compression performance and significantly limits the capacity of edge servers supporting real-time SS. In this paper, we propose STAC, a novel task-oriented DNN-driven video compressive streaming algorithm tailed for SS, to strike accuracy-bitrate balance and adapt to time-varying bandwidth. It exploits DNN's gradients as sensitivity metrics for fine-grained spatial adaptive compression and includes a temporal adaptive scheme that integrates spatial adaptation with predictive coding. Furthermore, we design a new bandwidth-aware neural network, serving as a compatible configuration tuner to fit time-varying bandwidth and content. STAC is evaluated in a system with a commodity mobile device and an edge server with real-world network traces. Experiments show that STAC can save up to 63.7–75.2% of bandwidth or improve accuracy by 3.1–9.5% compared to state-of-the-art algorithms, while capable of adapting to time-varying bandwidth.
期刊介绍:
IEEE Transactions on Mobile Computing addresses key technical issues related to various aspects of mobile computing. This includes (a) architectures, (b) support services, (c) algorithm/protocol design and analysis, (d) mobile environments, (e) mobile communication systems, (f) applications, and (g) emerging technologies. Topics of interest span a wide range, covering aspects like mobile networks and hosts, mobility management, multimedia, operating system support, power management, online and mobile environments, security, scalability, reliability, and emerging technologies such as wearable computers, body area networks, and wireless sensor networks. The journal serves as a comprehensive platform for advancements in mobile computing research.