Studying SATD in drone systems with Human-AI collaboration

Abstract

Background:

Self-Admitted Technical Debt (SATD) refers to sub-optimal solutions that developers acknowledge within the source code. SATD research originated on Java projects but is expanding to other domains. We focus on SATD in drones, which are used for various critical tasks.

Aims:

The primary objective is to investigate SATD in drone systems. The second aim is to explore the integration of AI and human collaboration for SATD labelling and classification.

Method:

Method: We conducted a sample study of SATD comments in drone systems (14 open source, 4 SDKs) to analyse the quantity and types of SATD comments present. Our study incorporates collaboration between AI and humans by utilising LLM for SATD classification. Additionally, we classified a sample of 385 SATD comments as either drone-specific or non-drone-specific.

Results:

The most prevalent SATD categories in drone software are Code Debt (35%), Unclassifiable Debt (16%), and Design Debt (15%). We found that 22% of SATD is specific to drones. Drone-specific SATD is proportionally more focused on Requirements and Design Debt compared to non-drone-specific SATD. We found that using both human and LLM for SATD classification can improve accuracy, as both LLM and human revised their initial ratings. After two rounds, a “near-perfect agreement” (Fleiss’ kappa 0.83) was achieved.

Conclusions:

Future studies should investigate whether our observation that domain-specific (drone) SATD comments relate more to Requirement Debt holds true in other domains. We propose a workflow that integrates AI into classification tasks, enhancing the accuracy of both human and AI classifications.