RPerf: Mining user reviews using topic modeling to assist performance testing: An industrial experience report

IF 3.7 2区计算机科学 Q1 COMPUTER SCIENCE, SOFTWARE ENGINEERING

Journal of Systems and Software Pub Date : 2024-11-28 DOI:10.1016/j.jss.2024.112283

Zehao Wang , Wei Liu , Jinfu Chen , Tse-Hsun (Peter) Chen

{"title":"RPerf: Mining user reviews using topic modeling to assist performance testing: An industrial experience report","authors":"Zehao Wang , Wei Liu , Jinfu Chen , Tse-Hsun (Peter) Chen","doi":"10.1016/j.jss.2024.112283","DOIUrl":null,"url":null,"abstract":"<div><div>Software performance affects the user-perceived quality of software. Therefore, it is important to analyze the performance issues that users are concerned with. In this paper, we document our experience working with our industry partner on analyzing user reviews to identify and analyze performance issues users are concerned with. In particular, we designed an approach, RPerf, which automatically analyzes unstructured user reviews and generates a performance analysis report that can assist performance engineers with performance testing. In particular, RPerf uses BERTopic to uncover performance-related topics in user reviews. RPerf then maps the derived topics to performance KPIs (key performance indicators) such as response time. Such performance KPIs better help performance test design and allocate performance testing resources. Finally, RPerf extracts user usage scenarios from user reviews to help identify the causes. Through a manual evaluation, we find RPerf achieves a high accuracy (over 93%) in identifying the performance-related topics and performance KPIs from user reviews. RPerf can also accurately extract usage scenarios in over 80% of user reviews. We discuss the performance analysis report that is generated based on RPerf. We believe that our findings can assist practitioners with analyzing performance-related user reviews and inspire future research on user review analysis.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"222 ","pages":"Article 112283"},"PeriodicalIF":3.7000,"publicationDate":"2024-11-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Systems and Software","FirstCategoryId":"94","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0164121224003273","RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMPUTER SCIENCE, SOFTWARE ENGINEERING","Score":null,"Total":0}

引用次数: 0

Abstract

Software performance affects the user-perceived quality of software. Therefore, it is important to analyze the performance issues that users are concerned with. In this paper, we document our experience working with our industry partner on analyzing user reviews to identify and analyze performance issues users are concerned with. In particular, we designed an approach, RPerf, which automatically analyzes unstructured user reviews and generates a performance analysis report that can assist performance engineers with performance testing. In particular, RPerf uses BERTopic to uncover performance-related topics in user reviews. RPerf then maps the derived topics to performance KPIs (key performance indicators) such as response time. Such performance KPIs better help performance test design and allocate performance testing resources. Finally, RPerf extracts user usage scenarios from user reviews to help identify the causes. Through a manual evaluation, we find RPerf achieves a high accuracy (over 93%) in identifying the performance-related topics and performance KPIs from user reviews. RPerf can also accurately extract usage scenarios in over 80% of user reviews. We discuss the performance analysis report that is generated based on RPerf. We believe that our findings can assist practitioners with analyzing performance-related user reviews and inspire future research on user review analysis.

查看原文本刊更多论文

RPerf：使用主题建模挖掘用户评论以辅助性能测试：一份行业经验报告

软件性能影响用户感知的软件质量。因此，分析用户关心的性能问题非常重要。在本文中，我们记录了我们与行业合作伙伴一起分析用户评论以识别和分析用户关心的性能问题的经验。特别是，我们设计了一种方法RPerf，它可以自动分析非结构化的用户评论，并生成一个性能分析报告，可以帮助性能工程师进行性能测试。特别是，RPerf使用BERTopic在用户评论中发现与性能相关的主题。然后，RPerf将派生主题映射到性能kpi（关键性能指标），例如响应时间。这样的性能kpi可以更好地帮助性能测试设计和分配性能测试资源。最后，RPerf从用户评论中提取用户使用场景，以帮助确定原因。通过手动评估，我们发现RPerf在从用户评论中识别与性能相关的主题和性能kpi方面达到了很高的准确性（超过93%）。RPerf还可以在超过80%的用户评论中准确提取使用场景。我们将讨论基于RPerf生成的性能分析报告。我们相信我们的发现可以帮助从业者分析与性能相关的用户评论，并激发未来对用户评论分析的研究。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Journal of Systems and Software 工程技术-计算机：理论方法

CiteScore

8.60

自引率

5.70%

发文量

193

审稿时长

16 weeks

期刊介绍： The Journal of Systems and Software publishes papers covering all aspects of software engineering and related hardware-software-systems issues. All articles should include a validation of the idea presented, e.g. through case studies, experiments, or systematic comparisons with other approaches already in practice. Topics of interest include, but are not limited to: •Methods and tools for, and empirical studies on, software requirements, design, architecture, verification and validation, maintenance and evolution •Agile, model-driven, service-oriented, open source and global software development •Approaches for mobile, multiprocessing, real-time, distributed, cloud-based, dependable and virtualized systems •Human factors and management concerns of software development •Data management and big data issues of software systems •Metrics and evaluation, data mining of software development resources •Business and economic aspects of software development processes The journal welcomes state-of-the-art surveys and reports of practical experience for all of these topics.