Evaluating large language models for software testing

IF 4.1 2区计算机科学 Q1 COMPUTER SCIENCE, HARDWARE & ARCHITECTURE

Computer Standards & Interfaces Pub Date : 2024-11-13 DOI:10.1016/j.csi.2024.103942

Yihao Li , Pan Liu , Haiyang Wang , Jie Chu , W. Eric Wong

引用次数: 0

Abstract

Large language models (LLMs) have demonstrated significant prowess in code analysis and natural language processing, making them highly valuable for software testing. This paper conducts a comprehensive evaluation of LLMs applied to software testing, with a particular emphasis on test case generation, error tracing, and bug localization across twelve open-source projects. The advantages and limitations, as well as recommendations associated with utilizing LLMs for these tasks, are delineated. Furthermore, we delve into the phenomenon of hallucination in LLMs, examining its impact on software testing processes and presenting solutions to mitigate its effects. The findings of this work contribute to a deeper understanding of integrating LLMs into software testing, providing insights that pave the way for enhanced effectiveness in the field.

查看原文本刊更多论文

评估用于软件测试的大型语言模型

大型语言模型（LLMs）在代码分析和自然语言处理方面表现出了巨大的优势，这使它们在软件测试方面具有很高的价值。本文对应用于软件测试的 LLM 进行了全面评估，重点是十二个开源项目的测试用例生成、错误跟踪和错误定位。本文阐述了在这些任务中使用 LLM 的优势和局限性，以及相关建议。此外，我们还深入探讨了 LLM 中的幻觉现象，研究了它对软件测试过程的影响，并提出了减轻其影响的解决方案。这项工作的发现有助于加深对将 LLM 纳入软件测试的理解，为提高该领域的效率提供了真知灼见。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Computer Standards & Interfaces 工程技术-计算机：软件工程

CiteScore

11.90

自引率

16.00%

发文量

审稿时长

6 months

期刊介绍： The quality of software, well-defined interfaces (hardware and software), the process of digitalisation, and accepted standards in these fields are essential for building and exploiting complex computing, communication, multimedia and measuring systems. Standards can simplify the design and construction of individual hardware and software components and help to ensure satisfactory interworking. Computer Standards & Interfaces is an international journal dealing specifically with these topics. The journal • Provides information about activities and progress on the definition of computer standards, software quality, interfaces and methods, at national, European and international levels • Publishes critical comments on standards and standards activities • Disseminates user''s experiences and case studies in the application and exploitation of established or emerging standards, interfaces and methods • Offers a forum for discussion on actual projects, standards, interfaces and methods by recognised experts • Stimulates relevant research by providing a specialised refereed medium.