Prioritization Method for Crowdsourced Test Report by Integrating Text and Image Information

Crowdsourcing testing has the advantages of efficiency, speed, and reliability, but an excessive number of test reports makes it a challenge for report reviewers to select high-quality test reports in a limited time. Test reports submitted by crowd workers often tend to be short textual descriptions with a large number of screenshots attached. Most traditional processing methods of test reports target reports that only contain text information, which cannot meet the defect detection requirements of crowdsourced test reports. In view of this, this paper proposes a prioritization method of crowdsourced test reports that integrates text and image information. First, we extract the text and image information from the test reports, based on which the defect detection abilities of the test reports are measured and the similarities between test reports are calculated. Then, a multi-stage prioritization method of the test reports is presented based on the defect detection levels and similarities of the test reports. In the first stage, based on the defect detection levels and the similarities, the test report set is sorted and clustered to obtain the sorting results of partial reports and the similar set for each sorted report; in the second stage, the similar test report set is sorted with the criteria of minimizing the similarity and maximizing the defect detection level; the sorting results of the two stages are combined to form the final priorities of test reports. To validate our approach, we conducted experiments on five crowdsourced test datasets. The results and the analysis show that our approach can detect all faults faster in a limited time. By comprehensively utilizing text and image information to prioritize test reports, better sorting results can be obtained than state-of-the-art methods.