Real-time safe validation of autonomous landing in populated areas: from virtual environments to Robot-In-The-Loop

IF 4.4 3区 计算机科学 Q1 COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS
Hector Tovanche-Picon, Javier González-Trejo, Ángel Flores-Abad, Miguel Ángel García-Terán, Diego Mercado-Ravell
{"title":"Real-time safe validation of autonomous landing in populated areas: from virtual environments to Robot-In-The-Loop","authors":"Hector Tovanche-Picon, Javier González-Trejo, Ángel Flores-Abad, Miguel Ángel García-Terán, Diego Mercado-Ravell","doi":"10.1007/s10055-024-00965-6","DOIUrl":null,"url":null,"abstract":"<p>Safe autonomous landing for Unmanned Aerial Vehicles (UAVs) in populated areas is a crucial aspect for successful integration of UAVs in populated environments. Nonetheless, validating autonomous landing in real scenarios is a challenging task with a high risk of injuring people. In this work, we propose a framework for safe real-time and thorough evaluation of vision-based autonomous landing in populated scenarios, using photo-realistic virtual environments and physics-based simulation. The proposed evaluation pipeline includes the use of Unreal graphics engine coupled with AirSim for realistic drone simulation to evaluate landing strategies. Then, Software-/Hardware-In-The-Loop can be used to test beforehand the performance of the algorithms. The final validation stage consists in a Robot-In-The-Loop evaluation strategy where a real drone must perform autonomous landing maneuvers in real-time, with an avatar drone in a virtual environment mimicking its behavior, while the detection algorithms run in the virtual environment (virtual reality to the robot). This method determines the safe landing areas based on computer vision and convolutional neural networks to avoid colliding with people in static and dynamic scenarios. To test the robustness of the algorithms in adversary conditions, different urban-like environments were implemented, including moving agents and different weather conditions. We also propose different metrics to quantify the performance of the landing strategies, establishing a baseline for comparison with future works on this challenging task, and analyze them through several randomized iterations. The proposed approach allowed us to safely validate the autonomous landing strategies, providing an evaluation pipeline, and a benchmark for comparison. An extensive evaluation showed a 99% success rate in static scenarios and 87% in dynamic cases, demonstrating that the use of autonomous landing algorithms considerably prevents accidents involving humans, facilitating the integration of drones in human-populated spaces, which may help to unleash the full potential of drones in urban environments. Besides, this type of development helps to increase the safety of drone operations, which would advance drone flight regulations and allow their use in closer proximity to humans.</p>","PeriodicalId":23727,"journal":{"name":"Virtual Reality","volume":null,"pages":null},"PeriodicalIF":4.4000,"publicationDate":"2024-03-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Virtual Reality","FirstCategoryId":"94","ListUrlMain":"https://doi.org/10.1007/s10055-024-00965-6","RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS","Score":null,"Total":0}
引用次数: 0

Abstract

Safe autonomous landing for Unmanned Aerial Vehicles (UAVs) in populated areas is a crucial aspect for successful integration of UAVs in populated environments. Nonetheless, validating autonomous landing in real scenarios is a challenging task with a high risk of injuring people. In this work, we propose a framework for safe real-time and thorough evaluation of vision-based autonomous landing in populated scenarios, using photo-realistic virtual environments and physics-based simulation. The proposed evaluation pipeline includes the use of Unreal graphics engine coupled with AirSim for realistic drone simulation to evaluate landing strategies. Then, Software-/Hardware-In-The-Loop can be used to test beforehand the performance of the algorithms. The final validation stage consists in a Robot-In-The-Loop evaluation strategy where a real drone must perform autonomous landing maneuvers in real-time, with an avatar drone in a virtual environment mimicking its behavior, while the detection algorithms run in the virtual environment (virtual reality to the robot). This method determines the safe landing areas based on computer vision and convolutional neural networks to avoid colliding with people in static and dynamic scenarios. To test the robustness of the algorithms in adversary conditions, different urban-like environments were implemented, including moving agents and different weather conditions. We also propose different metrics to quantify the performance of the landing strategies, establishing a baseline for comparison with future works on this challenging task, and analyze them through several randomized iterations. The proposed approach allowed us to safely validate the autonomous landing strategies, providing an evaluation pipeline, and a benchmark for comparison. An extensive evaluation showed a 99% success rate in static scenarios and 87% in dynamic cases, demonstrating that the use of autonomous landing algorithms considerably prevents accidents involving humans, facilitating the integration of drones in human-populated spaces, which may help to unleash the full potential of drones in urban environments. Besides, this type of development helps to increase the safety of drone operations, which would advance drone flight regulations and allow their use in closer proximity to humans.

Abstract Image

人口密集区自主着陆的实时安全验证:从虚拟环境到机器人在环运行
无人驾驶飞行器(UAV)在人口密集地区安全自主着陆是成功将无人驾驶飞行器融入人口密集环境的一个重要方面。然而,在真实场景中验证自主着陆是一项极具挑战性的任务,极有可能造成人员受伤。在这项工作中,我们提出了一个框架,利用逼真的虚拟环境和基于物理的模拟,对人口密集场景中基于视觉的自主着陆进行安全、实时和全面的评估。建议的评估流程包括使用虚幻图形引擎和 AirSim 进行逼真的无人机模拟,以评估着陆策略。然后,可以使用软件/硬件在环测试算法的性能。最后的验证阶段包括 "机器人在环"(Robot-In-The-Loop)评估策略,即真实无人机必须实时执行自主着陆操作,虚拟环境中的无人机化身模仿其行为,同时检测算法在虚拟环境(机器人的虚拟现实)中运行。该方法基于计算机视觉和卷积神经网络确定安全着陆区域,以避免在静态和动态场景中与人相撞。为了测试算法在不利条件下的鲁棒性,我们实施了不同的城市类环境,包括移动代理和不同的天气条件。我们还提出了不同的指标来量化着陆策略的性能,为与未来在这一具有挑战性任务上的工作进行比较建立了基线,并通过多次随机迭代对其进行了分析。所提出的方法使我们能够安全地验证自主着陆策略,提供了一个评估管道和比较基准。广泛的评估表明,在静态情况下成功率为 99%,在动态情况下成功率为 87%,这表明使用自主着陆算法可以大大防止涉及人类的事故,促进无人机与人类居住空间的融合,这可能有助于释放无人机在城市环境中的全部潜力。此外,这种类型的开发有助于提高无人机操作的安全性,这将推进无人机飞行法规,并允许在更接近人类的地方使用无人机。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
Virtual Reality
Virtual Reality COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS-COMPUTER SCIENCE, SOFTWARE ENGINEERING
CiteScore
8.30
自引率
14.30%
发文量
95
审稿时长
>12 weeks
期刊介绍: The journal, established in 1995, publishes original research in Virtual Reality, Augmented and Mixed Reality that shapes and informs the community. The multidisciplinary nature of the field means that submissions are welcomed on a wide range of topics including, but not limited to: Original research studies of Virtual Reality, Augmented Reality, Mixed Reality and real-time visualization applications Development and evaluation of systems, tools, techniques and software that advance the field, including: Display technologies, including Head Mounted Displays, simulators and immersive displays Haptic technologies, including novel devices, interaction and rendering Interaction management, including gesture control, eye gaze, biosensors and wearables Tracking technologies VR/AR/MR in medicine, including training, surgical simulation, rehabilitation, and tissue/organ modelling. Impactful and original applications and studies of VR/AR/MR’s utility in areas such as manufacturing, business, telecommunications, arts, education, design, entertainment and defence Research demonstrating new techniques and approaches to designing, building and evaluating virtual and augmented reality systems Original research studies assessing the social, ethical, data or legal aspects of VR/AR/MR.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信