Ranking guidance actions to support engineers in fulfilling process constraints

Abstract

In safety-critical systems engineering, regulations such as Automotive SPICE, ISO26262, or ED-109A mandate software quality assurance measures to provide evidence that the developed system is high quality. The constraints that define quality assurance conditions during the engineering life cycle are often non-trivial. This paper addresses the challenges, engineers face who are unfamiliar with the precise constraints of various projects (e.g., when newly joining a company or switching between departments). Understanding how to fulfill a constraint is a time-consuming and challenging task as an engineer needs to determine the most suitable option (out of potentially many) to fulfill a constraint violation. To this end, we propose a guidance action ranking framework to provide engineers with the most relevant guidance actions. Our primary ranking algorithm analyzes in the background the actions that engineers have made in the past to resolve a constraint violation without requiring explicit feedback from them. We evaluated our framework on two real-world data sets: an open-source drone management and an industrial air traffic control software system. Concretely, we replay past engineering activities and measured whether, in the case of a constraint violation, our suggested guidance actions were indeed selected by the engineer. The evaluation results revealed that learning from prior guidance actions effectively identifies the most appropriate guidance actions (ranked top 1 or 2) when compared to ranking algorithms based on action simplicity and artifact property change frequency. Specifically, we achieve a median MRR of 0.95 for the first case study and 0.94 for the second case study: an improvement of 80% and 100% over the baseline. Additionally, we observed that the simplicity of a guidance action does not reliably indicate its suitability for fulfilling a constraint, whereas learning from prior change operation property out-performed simplicity-based ranking but did not surpass guidance frequency-based ranking.