Text–image fusion template for large language model assisted crowdsourcing test aggregation

Abstract

Mobile crowdsourced testing leverages a varied group to enhance software quality through screenshots and text feedback. Examining the multitude of reports is tedious but crucial, often necessitating a combined analysis of both visual and textual information. However, professionals employ detailed judgment beyond mere similarity, which poses a challenge given the limited textual data and abundance of images in the reports.

We introduce a framework that guides large language models to handle missing data and inconsistencies in crowdsourced reports by using a triplet template $〈$ Scene, Operation, Defect $〉$ for bug identification. The framework leverages the element independence of the triplet for clustering ensemble and designs an algorithm to generate potential operation paths, aggregating reports within the cluster through constructed graphs. Our method, validated on 5115 reports, employs a clustering ensemble and graph aggregation, improving the clustering V-measure to 0.722. It also reduces the annotation time per report by 39. 3%, thereby improving the quality of the tagging. Source code available at https://github.com/Boomnana/Text-Image-Fusion.