IoT systems testing: Taxonomy, empirical findings, and recommendations

Abstract

The Internet of Things (IoT) is reshaping our lives, increasing the need for thorough pre-deployment testing. However, traditional software testing may not address the testing requirements of IoT systems, leading to quality challenges. A specific testing taxonomy is crucial, yet no widely recognized taxonomy exists for IoT system testing. We introduced an IoT-specific testing taxonomy that categorizes aspects of IoT systems testing into seven distinct categories. We mined testing aspects from 83 primary studies in IoT systems testing and built an initial taxonomy. This taxonomy was refined and validated through two rounds of surveys involving 16 and then 204 IoT industry practitioners. We assessed its effectiveness by conducting an empirical evaluation on two separate IoT systems, each involving 12 testers. Our findings categorize seven testing aspects: (1) testing objectives, (2) testing tools and artifacts, (3) testers, (4) testing stage, (5) testing environment, (6) Object Under Test (OUT) and metrics, and (7) testing approaches. The evaluation showed that testers equipped with the taxonomy could more effectively identify diverse test cases and scenarios. Additionally, we recommend new research opportunities to enhance the testing of IoT systems.