Quality evaluation meta-model for open-source software: multi-method validation study

Abstract

In recent years, open-source software (OSS) has attracted increasing attention due to its easy accessibility via cloud repositories, voluntary community, no vendor lock-in, and low total cost of ownership. In turn, specifying and evaluating OSS quality has become a significant challenge for OSS adoption in organizations that are inclined to use them. Although many OSS quality models have been proposed in the literature, the dynamic and diverse nature of OSS has caused these models to be heterogeneous in terms of structure and content. This has adversely affected the standardization of evaluations and led to the evaluation results obtained from different OSS quality models for the same purpose being incomparable and sometimes unreliable. Therefore, in this study, a meta-model for OSS quality (OSS-QMM), which employs a unified structure from existing quality models and enables the derivation of homogeneous models, has been proposed. For this purpose, a systematic and laborious effort has been spent via a step-based meta-model creation process including review-and-revise iterations. In order to validate the OSS-QMM, case study and expert opinion methods have been applied to answer three research questions (RQs) targeted to investigate practical applicability, results comparability, and effectiveness of using the meta-model. Multiple and embedded case study designs have been employed for evaluating three real ERP systems, and 20 subject matter experts have been interviewed during the validation process. The results of multi-faceted empirical studies have indicated that the OSS-QMM has addressed solving problems in OSS quality evaluation and its adoption with high degrees of confidence.