Do comments and expertise still matter? An experiment on programmers’ adoption of AI-generated JavaScript code

IF 4.1 2区计算机科学 Q1 COMPUTER SCIENCE, SOFTWARE ENGINEERING

Journal of Systems and Software Pub Date : 2025-09-23 DOI:10.1016/j.jss.2025.112634

Changwen Li , Christoph Treude , Ofir Turel

{"title":"Do comments and expertise still matter? An experiment on programmers’ adoption of AI-generated JavaScript code","authors":"Changwen Li , Christoph Treude , Ofir Turel","doi":"10.1016/j.jss.2025.112634","DOIUrl":null,"url":null,"abstract":"<div><div>This paper investigates the factors influencing programmers’ adoption of AI-generated JavaScript code recommendations within the context of lightweight, function-level programming tasks. It extends prior research by (1) utilizing objective (as opposed to the typically self-reported) measurements for programmers’ adoption of AI-generated code and (2) examining whether AI-generated comments added to code recommendations and development expertise drive AI-generated code adoption. We tested these potential drivers in an online experiment with 173 programmers. Participants were asked to answer some questions to demonstrate their level of development expertise. Then, they were asked to solve a LeetCode problem without AI support. After attempting to solve the problem on their own, they received an AI-generated solution to assist them in refining their solutions. The solutions provided were manipulated to include or exclude AI-generated comments (a between-subjects factor). Programmers’ adoption of AI-generated code was gauged by code similarity between AI-generated solutions and participants’ submitted solutions, providing a behavioral measurement of code adoption behaviors. Our findings revealed that, within the context of function-level programming tasks, the presence of comments significantly influences programmers’ adoption of AI-generated code regardless of the participants’ development expertise.</div></div>","PeriodicalId":51099,"journal":{"name":"Journal of Systems and Software","volume":"231 ","pages":"Article 112634"},"PeriodicalIF":4.1000,"publicationDate":"2025-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Systems and Software","FirstCategoryId":"94","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0164121225003036","RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMPUTER SCIENCE, SOFTWARE ENGINEERING","Score":null,"Total":0}

引用次数: 0

Abstract

This paper investigates the factors influencing programmers’ adoption of AI-generated JavaScript code recommendations within the context of lightweight, function-level programming tasks. It extends prior research by (1) utilizing objective (as opposed to the typically self-reported) measurements for programmers’ adoption of AI-generated code and (2) examining whether AI-generated comments added to code recommendations and development expertise drive AI-generated code adoption. We tested these potential drivers in an online experiment with 173 programmers. Participants were asked to answer some questions to demonstrate their level of development expertise. Then, they were asked to solve a LeetCode problem without AI support. After attempting to solve the problem on their own, they received an AI-generated solution to assist them in refining their solutions. The solutions provided were manipulated to include or exclude AI-generated comments (a between-subjects factor). Programmers’ adoption of AI-generated code was gauged by code similarity between AI-generated solutions and participants’ submitted solutions, providing a behavioral measurement of code adoption behaviors. Our findings revealed that, within the context of function-level programming tasks, the presence of comments significantly influences programmers’ adoption of AI-generated code regardless of the participants’ development expertise.

查看原文本刊更多论文

评论和专业知识还重要吗？关于程序员采用ai生成的JavaScript代码的实验

本文研究了在轻量级、函数级编程任务的背景下，影响程序员采用ai生成的JavaScript代码建议的因素。它通过(1)利用客观（而不是通常的自我报告）测量程序员对人工智能生成代码的采用来扩展先前的研究，(2)检查添加到代码建议和开发专业知识中的人工智能生成的注释是否推动了人工智能生成代码的采用。我们在173名程序员的在线实验中测试了这些潜在的驱动因素。与会者被要求回答一些问题，以展示他们的发展专业知识水平。然后，他们被要求在没有人工智能支持的情况下解决LeetCode问题。在尝试自己解决问题后，他们收到了一个人工智能生成的解决方案，以帮助他们完善解决方案。所提供的解决方案被操纵以包括或排除人工智能生成的评论（主题之间的因素）。通过人工智能生成的解决方案与参与者提交的解决方案之间的代码相似性来衡量程序员对人工智能生成代码的采用，从而提供了对代码采用行为的行为衡量。我们的研究结果表明，在功能级编程任务的背景下，无论参与者的开发专业知识如何，注释的存在都会显著影响程序员对人工智能生成代码的采用。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Journal of Systems and Software 工程技术-计算机：理论方法

CiteScore

8.60

自引率

5.70%

发文量

193

审稿时长

16 weeks

期刊介绍： The Journal of Systems and Software publishes papers covering all aspects of software engineering and related hardware-software-systems issues. All articles should include a validation of the idea presented, e.g. through case studies, experiments, or systematic comparisons with other approaches already in practice. Topics of interest include, but are not limited to: •Methods and tools for, and empirical studies on, software requirements, design, architecture, verification and validation, maintenance and evolution •Agile, model-driven, service-oriented, open source and global software development •Approaches for mobile, multiprocessing, real-time, distributed, cloud-based, dependable and virtualized systems •Human factors and management concerns of software development •Data management and big data issues of software systems •Metrics and evaluation, data mining of software development resources •Business and economic aspects of software development processes The journal welcomes state-of-the-art surveys and reports of practical experience for all of these topics.