先知:实现自动驾驶汽车可预测的实时感知管道

Liangkai Liu, Zheng-hong Dong, Yanzhi Wang, Weisong Shi
{"title":"先知:实现自动驾驶汽车可预测的实时感知管道","authors":"Liangkai Liu, Zheng-hong Dong, Yanzhi Wang, Weisong Shi","doi":"10.1109/RTSS55097.2022.00034","DOIUrl":null,"url":null,"abstract":"We have witnessed the broad adoption of Deep Neu-ral Networks (DNNs) in autonomous vehicles (AV). As a safety-critical system, deadline-based scheduling is used to guarantee the predictability of the AV system. However, non-negligible time variations exist for most DNN models in an AV system, even when the whole system is just running one model. The fact that multiple DNNs are running on the same platform makes the time variations issue even more severe. However, none of the existing works have thoroughly studied the root cause of the time variation issue. In the first part of the paper, we conducted a comprehensive empirical study. We found that the inference time variations for a single DNN model are mainly caused by the DNN's multi-stage/multi-branch structure, which has a dynamic number of proposals or raw points. In addition, we found that the uncoordinated contention and cooperation are the roots of the time variations for multi-tenant DNNs inference. Second, based on these insights, we proposed the Prophet system that addresses the time variations in the AV perception system in two steps. The first step is to predict the time variations based on the intermediate results like proposals and raw points. The second step is coordinating the multi-tenant DNNs to ensure the execution progress is close to each other. From the evaluation results on the KITTI dataset, the time prediction of a single model all achieve higher than 91% accuracy for Faster R-CNN, LaneNet, and PINet. Besides, the perception fusion delay is bounded to 150ms, and the fusion drop ratio is reduced from 5.4% to less than 1 percent.","PeriodicalId":202402,"journal":{"name":"2022 IEEE Real-Time Systems Symposium (RTSS)","volume":"235 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2022-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"3","resultStr":"{\"title\":\"Prophet: Realizing a Predictable Real-time Perception Pipeline for Autonomous Vehicles\",\"authors\":\"Liangkai Liu, Zheng-hong Dong, Yanzhi Wang, Weisong Shi\",\"doi\":\"10.1109/RTSS55097.2022.00034\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"We have witnessed the broad adoption of Deep Neu-ral Networks (DNNs) in autonomous vehicles (AV). As a safety-critical system, deadline-based scheduling is used to guarantee the predictability of the AV system. However, non-negligible time variations exist for most DNN models in an AV system, even when the whole system is just running one model. The fact that multiple DNNs are running on the same platform makes the time variations issue even more severe. However, none of the existing works have thoroughly studied the root cause of the time variation issue. In the first part of the paper, we conducted a comprehensive empirical study. We found that the inference time variations for a single DNN model are mainly caused by the DNN's multi-stage/multi-branch structure, which has a dynamic number of proposals or raw points. In addition, we found that the uncoordinated contention and cooperation are the roots of the time variations for multi-tenant DNNs inference. Second, based on these insights, we proposed the Prophet system that addresses the time variations in the AV perception system in two steps. The first step is to predict the time variations based on the intermediate results like proposals and raw points. The second step is coordinating the multi-tenant DNNs to ensure the execution progress is close to each other. From the evaluation results on the KITTI dataset, the time prediction of a single model all achieve higher than 91% accuracy for Faster R-CNN, LaneNet, and PINet. Besides, the perception fusion delay is bounded to 150ms, and the fusion drop ratio is reduced from 5.4% to less than 1 percent.\",\"PeriodicalId\":202402,\"journal\":{\"name\":\"2022 IEEE Real-Time Systems Symposium (RTSS)\",\"volume\":\"235 1\",\"pages\":\"0\"},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2022-12-01\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"3\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"2022 IEEE Real-Time Systems Symposium (RTSS)\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.1109/RTSS55097.2022.00034\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"2022 IEEE Real-Time Systems Symposium (RTSS)","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/RTSS55097.2022.00034","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 3

摘要

我们已经见证了深度神经网络(dnn)在自动驾驶汽车(AV)中的广泛应用。作为一种安全关键系统,基于截止日期调度是保证自动驾驶系统可预测性的重要手段。然而,对于自动驾驶系统中的大多数DNN模型来说,即使整个系统只运行一个模型,也存在不可忽略的时间变化。多个dnn在同一平台上运行的事实使得时间变化问题更加严重。然而,现有的工作都没有深入研究时间变化问题的根本原因。在论文的第一部分,我们进行了全面的实证研究。我们发现单个深度神经网络模型的推理时间变化主要是由深度神经网络的多阶段/多分支结构引起的,该结构具有动态数量的建议或原始点。此外,我们发现不协调的竞争和合作是多租户dnn推理时间变化的根源。其次,基于这些见解,我们提出了Prophet系统,该系统分两步解决自动驾驶感知系统的时间变化问题。第一步是根据提案和原始点等中间结果预测时间变化。第二步是协调多租户dnn,以确保执行进度彼此接近。从KITTI数据集上的评价结果来看,Faster R-CNN、LaneNet和PINet的单模型时间预测准确率均达到91%以上。此外,感知融合延迟被限制在150ms以内,融合下降率从5.4%降低到小于1%。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
Prophet: Realizing a Predictable Real-time Perception Pipeline for Autonomous Vehicles
We have witnessed the broad adoption of Deep Neu-ral Networks (DNNs) in autonomous vehicles (AV). As a safety-critical system, deadline-based scheduling is used to guarantee the predictability of the AV system. However, non-negligible time variations exist for most DNN models in an AV system, even when the whole system is just running one model. The fact that multiple DNNs are running on the same platform makes the time variations issue even more severe. However, none of the existing works have thoroughly studied the root cause of the time variation issue. In the first part of the paper, we conducted a comprehensive empirical study. We found that the inference time variations for a single DNN model are mainly caused by the DNN's multi-stage/multi-branch structure, which has a dynamic number of proposals or raw points. In addition, we found that the uncoordinated contention and cooperation are the roots of the time variations for multi-tenant DNNs inference. Second, based on these insights, we proposed the Prophet system that addresses the time variations in the AV perception system in two steps. The first step is to predict the time variations based on the intermediate results like proposals and raw points. The second step is coordinating the multi-tenant DNNs to ensure the execution progress is close to each other. From the evaluation results on the KITTI dataset, the time prediction of a single model all achieve higher than 91% accuracy for Faster R-CNN, LaneNet, and PINet. Besides, the perception fusion delay is bounded to 150ms, and the fusion drop ratio is reduced from 5.4% to less than 1 percent.
求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信