Random test generators demystified: Differences and potential for compiler reliability

IF 1.5 4区计算机科学 Q3 COMPUTER SCIENCE, SOFTWARE ENGINEERING

Science of Computer Programming Pub Date : 2025-07-03 DOI:10.1016/j.scico.2025.103359

Yang Wang, Zeyu Lu, Beining Wu, Yibiao Yang, Hongmin Lu, Yuming Zhou

{"title":"Random test generators demystified: Differences and potential for compiler reliability","authors":"Yang Wang, Zeyu Lu, Beining Wu, Yibiao Yang, Hongmin Lu, Yuming Zhou","doi":"10.1016/j.scico.2025.103359","DOIUrl":null,"url":null,"abstract":"<div><div>Compiler testing requires diverse programs as inputs. Various random program generators that can produce programs from scratch have been developed for this purpose. However, there is a gap in understanding (1) the differences among the generated programs and (2) how to make better use of these generators. To fill this gap, we selected five C random program generators and conducted the first comprehensive empirical analysis. For generated programs, our study focuses on three key areas: comparing the variations in features from multiple perspectives, analyzing the impact of compiling these programs on open-source compilers, and exploring their application potential in non-traditional testing scenarios. Programs from different generators show distinctive differences in various program features. Each has unique abilities to increase coverage of specific compiler components. Moreover, they can spot inconsistencies in the coverage statistics provided by different compilers, indicating promising application potential. Our study demonstrates that existing generators involve trade-offs in their design, making it challenging for any single implementation to balance efficiency, usability, and diversity for all scenarios. This motivates us to both maximize the potential of current generators and innovate to create more high-quality test programs for modern compiler quality assurance.</div></div>","PeriodicalId":49561,"journal":{"name":"Science of Computer Programming","volume":"247 ","pages":"Article 103359"},"PeriodicalIF":1.5000,"publicationDate":"2025-07-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Science of Computer Programming","FirstCategoryId":"94","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S016764232500098X","RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"COMPUTER SCIENCE, SOFTWARE ENGINEERING","Score":null,"Total":0}

引用次数: 0

Abstract

Compiler testing requires diverse programs as inputs. Various random program generators that can produce programs from scratch have been developed for this purpose. However, there is a gap in understanding (1) the differences among the generated programs and (2) how to make better use of these generators. To fill this gap, we selected five C random program generators and conducted the first comprehensive empirical analysis. For generated programs, our study focuses on three key areas: comparing the variations in features from multiple perspectives, analyzing the impact of compiling these programs on open-source compilers, and exploring their application potential in non-traditional testing scenarios. Programs from different generators show distinctive differences in various program features. Each has unique abilities to increase coverage of specific compiler components. Moreover, they can spot inconsistencies in the coverage statistics provided by different compilers, indicating promising application potential. Our study demonstrates that existing generators involve trade-offs in their design, making it challenging for any single implementation to balance efficiency, usability, and diversity for all scenarios. This motivates us to both maximize the potential of current generators and innovate to create more high-quality test programs for modern compiler quality assurance.

查看原文本刊更多论文

揭秘随机测试生成器：编译器可靠性的差异和潜力

编译器测试需要不同的程序作为输入。为了这个目的，已经开发了各种可以从头开始生成程序的随机程序生成器。然而，在理解(1)生成的程序之间的差异和(2)如何更好地利用这些生成器方面存在差距。为了填补这一空白，我们选择了五个C随机程序生成器，并进行了第一次全面的实证分析。对于生成的程序，我们的研究主要集中在三个关键领域：从多个角度比较特征的变化，分析编译这些程序对开源编译器的影响，以及探索它们在非传统测试场景中的应用潜力。来自不同生成器的程序在各种程序特性上表现出明显的差异。每个都有独特的能力来增加特定编译器组件的覆盖率。此外，它们还可以发现不同编译器提供的覆盖率统计数据中的不一致之处，从而表明应用程序的潜力。我们的研究表明，现有的生成器在其设计中涉及权衡，使得任何单一实现都难以平衡所有场景的效率，可用性和多样性。这促使我们最大化当前生成器的潜力，并为现代编译器质量保证创造更多高质量的测试程序。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Science of Computer Programming 工程技术-计算机：软件工程

CiteScore

3.80

自引率

0.00%

发文量

审稿时长

67 days

期刊介绍： Science of Computer Programming is dedicated to the distribution of research results in the areas of software systems development, use and maintenance, including the software aspects of hardware design. The journal has a wide scope ranging from the many facets of methodological foundations to the details of technical issues andthe aspects of industrial practice. The subjects of interest to SCP cover the entire spectrum of methods for the entire life cycle of software systems, including • Requirements, specification, design, validation, verification, coding, testing, maintenance, metrics and renovation of software; • Design, implementation and evaluation of programming languages; • Programming environments, development tools, visualisation and animation; • Management of the development process; • Human factors in software, software for social interaction, software for social computing; • Cyber physical systems, and software for the interaction between the physical and the machine; • Software aspects of infrastructure services, system administration, and network management.