GPT-O1 能杀死所有虫子吗？

arXiv - CS - Software Engineering Pub Date : 2024-09-16 DOI:arxiv-2409.10033

Haichuan Hu, Ye Shang, Guolin Xu, Congqing He, Quanjun Zhang

{"title":"GPT-O1 能杀死所有虫子吗？","authors":"Haichuan Hu, Ye Shang, Guolin Xu, Congqing He, Quanjun Zhang","doi":"arxiv-2409.10033","DOIUrl":null,"url":null,"abstract":"ChatGPT has long been proven to be effective in automatic program repair\n(APR). With the continuous iterations and upgrades of the ChatGPT version, its\nperformance in terms of fixes has already reached state-of-the-art levels.\nHowever, there are few works comparing the effectiveness and variations of\ndifferent versions of ChatGPT on APR. In this work, we evaluate the performance\nof the latest version of ChatGPT (O1-preview and O1-mini), ChatGPT-4o, and\nhistorical version of ChatGPT on APR. We study the improvements of the O1 model\nover traditional ChatGPT in terms of APR from multiple perspectives (repair\nsuccess rate, repair cost, behavior patterns), and find that O1's repair\ncapability exceeds that of traditional ChatGPT, successfully fixing all 40 bugs\nin the benchmark. Our work can serve as a reference for further in-depth\nexploration of the applications of ChatGPT in APR.","PeriodicalId":501278,"journal":{"name":"arXiv - CS - Software Engineering","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2024-09-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Can GPT-O1 Kill All Bugs?\",\"authors\":\"Haichuan Hu, Ye Shang, Guolin Xu, Congqing He, Quanjun Zhang\",\"doi\":\"arxiv-2409.10033\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"ChatGPT has long been proven to be effective in automatic program repair\\n(APR). With the continuous iterations and upgrades of the ChatGPT version, its\\nperformance in terms of fixes has already reached state-of-the-art levels.\\nHowever, there are few works comparing the effectiveness and variations of\\ndifferent versions of ChatGPT on APR. In this work, we evaluate the performance\\nof the latest version of ChatGPT (O1-preview and O1-mini), ChatGPT-4o, and\\nhistorical version of ChatGPT on APR. We study the improvements of the O1 model\\nover traditional ChatGPT in terms of APR from multiple perspectives (repair\\nsuccess rate, repair cost, behavior patterns), and find that O1's repair\\ncapability exceeds that of traditional ChatGPT, successfully fixing all 40 bugs\\nin the benchmark. Our work can serve as a reference for further in-depth\\nexploration of the applications of ChatGPT in APR.\",\"PeriodicalId\":501278,\"journal\":{\"name\":\"arXiv - CS - Software Engineering\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2024-09-16\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"arXiv - CS - Software Engineering\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/arxiv-2409.10033\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - CS - Software Engineering","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2409.10033","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}

引用次数: 0

摘要

ChatGPT 在自动程序修复（APR）方面的功效早已得到证实。随着 ChatGPT 版本的不断迭代和升级，它在修复方面的性能已经达到了最先进的水平。然而，比较不同版本的 ChatGPT 在 APR 上的效果和变化的工作却很少。在这项工作中，我们评估了最新版本的 ChatGPT（O1-preview 和 O1-mini）、ChatGPT-4o 和历史版本的 ChatGPT 在 APR 上的性能。我们从多个角度（修复成功率、修复成本、行为模式）研究了 O1 模型对传统 ChatGPT 在 APR 方面的改进，发现 O1 的修复能力超过了传统 ChatGPT，成功修复了基准测试中的所有 40 个错误。我们的工作可为进一步深入探索 ChatGPT 在 APR 中的应用提供参考。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

查看原文本刊更多论文

Can GPT-O1 Kill All Bugs?

ChatGPT has long been proven to be effective in automatic program repair (APR). With the continuous iterations and upgrades of the ChatGPT version, its performance in terms of fixes has already reached state-of-the-art levels. However, there are few works comparing the effectiveness and variations of different versions of ChatGPT on APR. In this work, we evaluate the performance of the latest version of ChatGPT (O1-preview and O1-mini), ChatGPT-4o, and historical version of ChatGPT on APR. We study the improvements of the O1 model over traditional ChatGPT in terms of APR from multiple perspectives (repair success rate, repair cost, behavior patterns), and find that O1's repair capability exceeds that of traditional ChatGPT, successfully fixing all 40 bugs in the benchmark. Our work can serve as a reference for further in-depth exploration of the applications of ChatGPT in APR.

求助全文

通过发布文献求助，成功后即可免费获取论文全文。去求助

来源期刊

arXiv - CS - Software Engineering

自引率

0.00%

发文量