MT-Nod: Metamorphic testing for detecting non-optimal decisions of autonomous driving systems in interactive scenarios

IF 3.8 2区计算机科学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

Information and Software Technology Pub Date : 2025-01-03 DOI:10.1016/j.infsof.2024.107659

Zhen Yang , Song Huang , Xingya Wang , Tongtong Bai , Yang Wang

{"title":"MT-Nod: Metamorphic testing for detecting non-optimal decisions of autonomous driving systems in interactive scenarios","authors":"Zhen Yang , Song Huang , Xingya Wang , Tongtong Bai , Yang Wang","doi":"10.1016/j.infsof.2024.107659","DOIUrl":null,"url":null,"abstract":"<div><h3>Context:</h3><div>Autonomous driving technology advances into daily life, with expectations for autonomous driving systems (ADSs) to make optimal, human-like decisions. However, ADSs often exhibit “unintelligent” behaviors like inefficient path choices, significantly impacting travel efficiency and potentially causing delays. Therefore, testing the decision optimality of ADSs is critically urgent. However, the testing process faces a significant “testing oracle” problem, and current methods overlook behavior interactions, which do not reflect real-world traffic scenarios.</div></div><div><h3>Objective:</h3><div>To assess the performance and reliability of ADSs in optimal decision-making, mitigate test oracle problems, and detect non-optimal decisions without calculating the optimal path.</div></div><div><h3>Method:</h3><div>This paper proposes a metamorphic testing method for optimal decision-making in autonomous driving under interactive scenarios, MT-Nod. Our method introduces a novel metamorphic relation to evaluate the optimality of path directions, along with a follow-up scenario generation method. The scenario generation method includes mutation points selection based on key behaviors, scenario mutation based on behavioral interactions, and road accessibility assessment, to generate scenarios with dynamic interactions. Additionally, a scenario scheduling strategy is designed to prioritize and schedule scenarios based on the priority of mutation points.</div></div><div><h3>Results:</h3><div>We evaluate MT-Nod extensively on the advanced Apollo ADS. Across four source scenarios, it generates 69.3 non-optimal decision scenarios (NoDSs), classified into eight types. Compared to baselines, MT-Nod efficiently produces and detects a greater variety and quantity of NoDSs.</div></div><div><h3>Conclusion:</h3><div>The proposed method for optimal decision testing under interactive scenarios, MT-Nod, effectively detects non-optimal decisions of ADSs. These “unintelligent” behaviors are crucial for enhancing the performance and reliability of ADSs.</div></div>","PeriodicalId":54983,"journal":{"name":"Information and Software Technology","volume":"180 ","pages":"Article 107659"},"PeriodicalIF":3.8000,"publicationDate":"2025-01-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Information and Software Technology","FirstCategoryId":"94","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0950584924002647","RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"COMPUTER SCIENCE, INFORMATION SYSTEMS","Score":null,"Total":0}

引用次数: 0

Abstract

Context:

Autonomous driving technology advances into daily life, with expectations for autonomous driving systems (ADSs) to make optimal, human-like decisions. However, ADSs often exhibit “unintelligent” behaviors like inefficient path choices, significantly impacting travel efficiency and potentially causing delays. Therefore, testing the decision optimality of ADSs is critically urgent. However, the testing process faces a significant “testing oracle” problem, and current methods overlook behavior interactions, which do not reflect real-world traffic scenarios.

Objective:

To assess the performance and reliability of ADSs in optimal decision-making, mitigate test oracle problems, and detect non-optimal decisions without calculating the optimal path.

Method:

This paper proposes a metamorphic testing method for optimal decision-making in autonomous driving under interactive scenarios, MT-Nod. Our method introduces a novel metamorphic relation to evaluate the optimality of path directions, along with a follow-up scenario generation method. The scenario generation method includes mutation points selection based on key behaviors, scenario mutation based on behavioral interactions, and road accessibility assessment, to generate scenarios with dynamic interactions. Additionally, a scenario scheduling strategy is designed to prioritize and schedule scenarios based on the priority of mutation points.

Results:

We evaluate MT-Nod extensively on the advanced Apollo ADS. Across four source scenarios, it generates 69.3 non-optimal decision scenarios (NoDSs), classified into eight types. Compared to baselines, MT-Nod efficiently produces and detects a greater variety and quantity of NoDSs.

Conclusion:

The proposed method for optimal decision testing under interactive scenarios, MT-Nod, effectively detects non-optimal decisions of ADSs. These “unintelligent” behaviors are crucial for enhancing the performance and reliability of ADSs.

查看原文本刊更多论文

求助全文

约1分钟内获得全文求助全文

来源期刊

Information and Software Technology 工程技术-计算机：软件工程

CiteScore

9.10

自引率

7.70%

发文量

164

审稿时长

9.6 weeks

期刊介绍： Information and Software Technology is the international archival journal focusing on research and experience that contributes to the improvement of software development practices. The journal''s scope includes methods and techniques to better engineer software and manage its development. Articles submitted for review should have a clear component of software engineering or address ways to improve the engineering and management of software development. Areas covered by the journal include: • Software management, quality and metrics, • Software processes, • Software architecture, modelling, specification, design and programming • Functional and non-functional software requirements • Software testing and verification & validation • Empirical studies of all aspects of engineering and managing software development Short Communications is a new section dedicated to short papers addressing new ideas, controversial opinions, "Negative" results and much more. Read the Guide for authors for more information. The journal encourages and welcomes submissions of systematic literature studies (reviews and maps) within the scope of the journal. Information and Software Technology is the premiere outlet for systematic literature studies in software engineering.