Piloting Copilot, Codex, and StarCoder2: Hot temperature, cold prompts, or black magic?

IF 4.1 2区计算机科学 Q1 COMPUTER SCIENCE, SOFTWARE ENGINEERING

Journal of Systems and Software Pub Date : 2025-07-25 DOI:10.1016/j.jss.2025.112562

Jean-Baptiste Döderlein , Nguessan Hermann Kouadio , Mathieu Acher , Djamel Eddine Khelladi , Benoit Combemale

{"title":"Piloting Copilot, Codex, and StarCoder2: Hot temperature, cold prompts, or black magic?","authors":"Jean-Baptiste Döderlein , Nguessan Hermann Kouadio , Mathieu Acher , Djamel Eddine Khelladi , Benoit Combemale","doi":"10.1016/j.jss.2025.112562","DOIUrl":null,"url":null,"abstract":"<div><div>Language models are promising solutions for tackling increasing complex problems. In software engineering, they recently gained attention in code assistants, which generate programs from a natural language task description (prompt). They have the potential to save time and effort but remain poorly understood, limiting their optimal use. In this article, we investigate the impact of input variations on two configurations of a language model, focusing on parameters such as task description, surrounding context, model creativity, and the number of generated solutions. We design specific operators to modify these inputs and apply them to three LLM-based code assistants (Copilot, Codex, StarCoder2) and two benchmarks representing algorithmic problems (HumanEval, LeetCode). Our study examines whether these variations significantly affect program quality and how these effects generalize across models.</div><div>Our results show that varying input parameters can greatly improve performance, achieving up to 79.27% success in one-shot generation compared to 22.44% for Codex and 31.1% for Copilot in default settings. Actioning this potential in practice is challenging due to the complex interplay in our study—the optimal settings for temperature, prompt, and number of generated solutions vary by problem.</div><div>Reproducing our study with StarCoder2 confirms these findings, indicating they are not model-specific. We also uncover surprising behaviors (e.g., fully removing the prompt can be effective), revealing model brittleness and areas for improvement.</div><div>Overall, this work opens opportunities to envision (automated) strategies for enhancing performance of language model-based code assistants, but also questions their reliability and robustness.</div><div><em>Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board</em>.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"230 ","pages":"Article 112562"},"PeriodicalIF":4.1000,"publicationDate":"2025-07-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Systems and Software","FirstCategoryId":"94","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0164121225002316","RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMPUTER SCIENCE, SOFTWARE ENGINEERING","Score":null,"Total":0}

引用次数: 0

Abstract

Language models are promising solutions for tackling increasing complex problems. In software engineering, they recently gained attention in code assistants, which generate programs from a natural language task description (prompt). They have the potential to save time and effort but remain poorly understood, limiting their optimal use. In this article, we investigate the impact of input variations on two configurations of a language model, focusing on parameters such as task description, surrounding context, model creativity, and the number of generated solutions. We design specific operators to modify these inputs and apply them to three LLM-based code assistants (Copilot, Codex, StarCoder2) and two benchmarks representing algorithmic problems (HumanEval, LeetCode). Our study examines whether these variations significantly affect program quality and how these effects generalize across models.

Our results show that varying input parameters can greatly improve performance, achieving up to 79.27% success in one-shot generation compared to 22.44% for Codex and 31.1% for Copilot in default settings. Actioning this potential in practice is challenging due to the complex interplay in our study—the optimal settings for temperature, prompt, and number of generated solutions vary by problem.

Reproducing our study with StarCoder2 confirms these findings, indicating they are not model-specific. We also uncover surprising behaviors (e.g., fully removing the prompt can be effective), revealing model brittleness and areas for improvement.

Overall, this work opens opportunities to envision (automated) strategies for enhancing performance of language model-based code assistants, but also questions their reliability and robustness.

Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board.

查看原文本刊更多论文

驾驶副驾驶，法典，和StarCoder2：高温，冷提示，或黑魔法？

语言模型是解决日益复杂的问题的有希望的解决方案。在软件工程中，它们最近在代码助手中引起了关注，代码助手从自然语言任务描述（提示）生成程序。它们有节省时间和精力的潜力，但仍然知之甚少，限制了它们的最佳使用。在本文中，我们研究了输入变化对语言模型的两种配置的影响，重点关注任务描述、周围上下文、模型创造力和生成的解决方案数量等参数。我们设计了特定的操作符来修改这些输入，并将它们应用于三个基于llm的代码助手（Copilot, Codex, StarCoder2）和两个代表算法问题的基准（HumanEval, LeetCode）。我们的研究考察了这些变化是否会显著影响节目质量，以及这些影响如何在各个模型中普遍化。我们的研究结果表明，不同的输入参数可以大大提高性能，在默认设置下，Codex和Copilot的单次生成成功率分别为22.44%和31.1%，而Codex和Copilot的单次生成成功率高达79.27%。由于我们的研究中复杂的相互作用，在实践中发挥这一潜力是具有挑战性的——温度、提示和生成解决方案的数量的最佳设置因问题而异。用StarCoder2复制我们的研究证实了这些发现，表明它们不是特定于模型的。我们还发现了令人惊讶的行为（例如，完全删除提示符可能是有效的），揭示了模型的脆弱性和需要改进的地方。总的来说，这项工作为设想（自动化）策略来增强基于语言模型的代码助手的性能提供了机会，但也质疑了它们的可靠性和健壮性。编者注：开放科学材料由系统与软件开放科学委员会杂志验证。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Journal of Systems and Software 工程技术-计算机：理论方法

CiteScore

8.60

自引率

5.70%

发文量

193

审稿时长

16 weeks

期刊介绍： The Journal of Systems and Software publishes papers covering all aspects of software engineering and related hardware-software-systems issues. All articles should include a validation of the idea presented, e.g. through case studies, experiments, or systematic comparisons with other approaches already in practice. Topics of interest include, but are not limited to: •Methods and tools for, and empirical studies on, software requirements, design, architecture, verification and validation, maintenance and evolution •Agile, model-driven, service-oriented, open source and global software development •Approaches for mobile, multiprocessing, real-time, distributed, cloud-based, dependable and virtualized systems •Human factors and management concerns of software development •Data management and big data issues of software systems •Metrics and evaluation, data mining of software development resources •Business and economic aspects of software development processes The journal welcomes state-of-the-art surveys and reports of practical experience for all of these topics.