Evaluating test characteristics and effectiveness of FSM-based testing methods on RBAC systems

Access control mechanisms demand rigorous software testing approaches, otherwise they can end up with security flaws. Finite state machines (FSM) have been used for testing Role-Based Access Control (RBAC) mechanisms and complete, but significantly large, test suites can be obtained. Experimental studies have shown that recent FSM testing methods can reduce the overall test suite length for random FSMs. However, since the similarity between random FSMs and these specifying RBAC mechanisms is unclear, these outcomes cannot be necessarily generalized to RBAC. In this paper, we compare the characteristics and effectiveness of test suites generated by traditional and recent FSM testing methods for RBAC policies specified as FSM models. The methods W, HSI and SPY were applied on RBAC policies specified as FSMs and the test suites obtained were evaluated considering test characteristics (number of resets, average test case length, and test suite length) and effectiveness on the RBAC fault domain. Our results corroborate some outcomes of previous investigations in which test suites presented different characteristics. On average, the SPY method generated test suites with 32% less resets, average test case length 78% greater than W and HSI, and overall length 46% lower. There were no differences among FSM testing methods for RBAC regarding effectiveness. However, the SPY method significantly reduced the overall test suite length and the number of resets.