Python Symbolic Execution with LLM-powered Code Generation

Wenhan Wang, Kaibo Liu, An Ran Chen, Ge Li, Zhi Jin, Gang Huang, Lei Ma
{"title":"Python Symbolic Execution with LLM-powered Code Generation","authors":"Wenhan Wang, Kaibo Liu, An Ran Chen, Ge Li, Zhi Jin, Gang Huang, Lei Ma","doi":"arxiv-2409.09271","DOIUrl":null,"url":null,"abstract":"Symbolic execution is a key technology in software testing, which generates\ntest cases by collecting symbolic path constraints and then solving constraints\nwith SMT solvers. Symbolic execution has been proven helpful in generating\nhigh-coverage test cases, but its limitations, e.g., the difficulties in\nsolving path constraints, prevent it from broader usage in software testing.\nMoreover, symbolic execution has encountered many difficulties when applied to\ndynamically typed languages like Python, because it is extremely challenging to\ntranslate the flexible Python grammar into rigid solvers. To overcome the main challenges of applying symbolic execution in Python, we\nproposed an LLM-empowered agent, LLM-Sym, that automatically calls an SMT\nsolver, Z3, to solve execution path constraints. Based on an introductory-level\nsymbolic execution engine, our LLM agent can extend it to supporting programs\nwith complex data type `list'. The core contribution of LLM-Sym is translating\ncomplex Python path constraints into Z3 code. To enable accurate path-to-Z3\ntranslation, we design a multiple-step code generation pipeline including type\ninference, retrieval and self-refine. Our experiments demonstrate that LLM-Sym\nis capable of solving path constraints on Leetcode problems with complicated\ncontrol flows and list data structures, which is impossible for the backbone\nsymbolic execution engine. Our approach paves the way for the combination of\nthe generation ability of LLMs with the reasoning ability of symbolic solvers,\nand opens up new opportunities in LLM-augmented test case generation.","PeriodicalId":501278,"journal":{"name":"arXiv - CS - Software Engineering","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2024-09-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - CS - Software Engineering","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2409.09271","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

Abstract

Symbolic execution is a key technology in software testing, which generates test cases by collecting symbolic path constraints and then solving constraints with SMT solvers. Symbolic execution has been proven helpful in generating high-coverage test cases, but its limitations, e.g., the difficulties in solving path constraints, prevent it from broader usage in software testing. Moreover, symbolic execution has encountered many difficulties when applied to dynamically typed languages like Python, because it is extremely challenging to translate the flexible Python grammar into rigid solvers. To overcome the main challenges of applying symbolic execution in Python, we proposed an LLM-empowered agent, LLM-Sym, that automatically calls an SMT solver, Z3, to solve execution path constraints. Based on an introductory-level symbolic execution engine, our LLM agent can extend it to supporting programs with complex data type `list'. The core contribution of LLM-Sym is translating complex Python path constraints into Z3 code. To enable accurate path-to-Z3 translation, we design a multiple-step code generation pipeline including type inference, retrieval and self-refine. Our experiments demonstrate that LLM-Sym is capable of solving path constraints on Leetcode problems with complicated control flows and list data structures, which is impossible for the backbone symbolic execution engine. Our approach paves the way for the combination of the generation ability of LLMs with the reasoning ability of symbolic solvers, and opens up new opportunities in LLM-augmented test case generation.
利用 LLM 驱动的代码生成实现 Python 符号执行
符号执行是软件测试中的一项关键技术,它通过收集符号路径约束,然后用 SMT 求解器求解约束来生成测试用例。符号执行已被证明有助于生成高覆盖率的测试用例,但其局限性,如路径约束求解的困难,阻碍了它在软件测试中的广泛应用。此外,符号执行在应用于像 Python 这样的动态类型语言时也遇到了很多困难,因为将灵活的 Python 语法转换为刚性求解器是一项极具挑战性的工作。为了克服在 Python 中应用符号执行所面临的主要挑战,我们提出了一种由 LLM 驱动的代理 LLM-Sym,它可以自动调用 SMT 解算器 Z3 来解决执行路径约束。我们的 LLM 代理以入门级符号执行引擎为基础,可以扩展到支持具有复杂数据类型 "列表 "的程序。LLM-Sym 的核心贡献是将复杂的 Python 路径约束翻译成 Z3 代码。为了实现路径到 Z3 的精确翻译,我们设计了一个多步骤代码生成流水线,包括类型推理、检索和自我精炼。我们的实验证明,LLM-Sym 能够解决具有复杂控制流和列表数据结构的 Leetcode 问题的路径约束,而这是骨干符号执行引擎无法做到的。我们的方法为将 LLM 的生成能力与符号求解器的推理能力相结合铺平了道路,并为 LLM 增强测试用例生成开辟了新的机遇。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信