Quality assessment of GPT-3.5 and Gemini 1.0 Pro for SQL syntax

IF 4.1 2区计算机科学 Q1 COMPUTER SCIENCE, HARDWARE & ARCHITECTURE

Computer Standards & Interfaces Pub Date : 2025-06-24 DOI:10.1016/j.csi.2025.104041

Cosmina-Mihaela Rosca , Adrian Stancu

{"title":"Quality assessment of GPT-3.5 and Gemini 1.0 Pro for SQL syntax","authors":"Cosmina-Mihaela Rosca , Adrian Stancu","doi":"10.1016/j.csi.2025.104041","DOIUrl":null,"url":null,"abstract":"<div><div>Nowadays, GPT-3.5 and Gemini 1.0 Pro are employed for various tasks, both for personal and professional use, in multiple domains like education, economy, computer science, etc. Given the increase in users, knowing the quality level of these artificial intelligence (AI) tools is important. Thus, the paper presents a comparative analysis of syntax accuracy generated for SQL databases utilizing the services of GPT-3.5 and Gemini 1.0 Pro. Firstly, the algorithms for testing GPT-3.5 and Gemini 1.0 Pro were developed. Secondly, five types of tests, which implied 700 queries, were conducted by considering requirements with low and high degrees of difficulty. The tests focus on syntax-generated accuracy using an experimental (NorthWind) database, syntax-generated accuracy study using a user-made database, syntax correction accuracy, different responses to the same question on the same account, and different responses to the same question on other accounts. The accuracy obtained for all tests revealed that the GPT-3.5 service has a value of 87 % for SQL syntax generation or correction, whereas the Gemini 1.0 Pro service has an accuracy of 80 %. These results underscore the effectiveness of GPT-3.5 and Gemini 1.0 Pro in assisting with SQL syntax tasks, albeit with differing levels of precision. The findings highlight the significance of human supervision and validation in ensuring the correctness of AI-generated responses, particularly in database-related tasks. The results affect developers and database administrators when selecting appropriate tools for query requirements. For now, replacing programmers with GPT-3.5 and Gemini 1.0 Pro is impossible.</div></div>","PeriodicalId":50635,"journal":{"name":"Computer Standards & Interfaces","volume":"95 ","pages":"Article 104041"},"PeriodicalIF":4.1000,"publicationDate":"2025-06-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Computer Standards & Interfaces","FirstCategoryId":"94","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0920548925000704","RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMPUTER SCIENCE, HARDWARE & ARCHITECTURE","Score":null,"Total":0}

引用次数: 0

Abstract

Nowadays, GPT-3.5 and Gemini 1.0 Pro are employed for various tasks, both for personal and professional use, in multiple domains like education, economy, computer science, etc. Given the increase in users, knowing the quality level of these artificial intelligence (AI) tools is important. Thus, the paper presents a comparative analysis of syntax accuracy generated for SQL databases utilizing the services of GPT-3.5 and Gemini 1.0 Pro. Firstly, the algorithms for testing GPT-3.5 and Gemini 1.0 Pro were developed. Secondly, five types of tests, which implied 700 queries, were conducted by considering requirements with low and high degrees of difficulty. The tests focus on syntax-generated accuracy using an experimental (NorthWind) database, syntax-generated accuracy study using a user-made database, syntax correction accuracy, different responses to the same question on the same account, and different responses to the same question on other accounts. The accuracy obtained for all tests revealed that the GPT-3.5 service has a value of 87 % for SQL syntax generation or correction, whereas the Gemini 1.0 Pro service has an accuracy of 80 %. These results underscore the effectiveness of GPT-3.5 and Gemini 1.0 Pro in assisting with SQL syntax tasks, albeit with differing levels of precision. The findings highlight the significance of human supervision and validation in ensuring the correctness of AI-generated responses, particularly in database-related tasks. The results affect developers and database administrators when selecting appropriate tools for query requirements. For now, replacing programmers with GPT-3.5 and Gemini 1.0 Pro is impossible.

查看原文本刊更多论文

GPT-3.5和Gemini 1.0 Pro SQL语法质量评估

如今，GPT-3.5和Gemini 1.0 Pro在教育、经济、计算机科学等多个领域被用于个人和专业用途的各种任务。鉴于用户的增加，了解这些人工智能（AI）工具的质量水平非常重要。因此，本文对使用GPT-3.5和Gemini 1.0 Pro服务生成的SQL数据库的语法准确性进行了比较分析。首先，开发了GPT-3.5和Gemini 1.0 Pro的测试算法。其次，通过考虑低难度和高难度的需求，进行了五种类型的测试，这意味着700个查询。测试的重点是使用实验性（NorthWind）数据库的语法生成准确性，使用用户自制数据库的语法生成准确性研究，语法更正准确性，同一帐户对同一问题的不同回答，以及其他帐户对同一问题的不同回答。所有测试获得的准确度显示，GPT-3.5服务在SQL语法生成或更正方面的准确度为87%，而Gemini 1.0 Pro服务的准确度为80%。这些结果强调了GPT-3.5和Gemini 1.0 Pro在协助SQL语法任务方面的有效性，尽管精度水平不同。研究结果强调了人类监督和验证在确保人工智能生成的响应的正确性方面的重要性，特别是在与数据库相关的任务中。在为查询需求选择合适的工具时，结果会影响开发人员和数据库管理员。目前，用GPT-3.5和Gemini 1.0 Pro取代程序员是不可能的。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Computer Standards & Interfaces 工程技术-计算机：软件工程

CiteScore

11.90

自引率

16.00%

发文量

审稿时长

6 months

期刊介绍： The quality of software, well-defined interfaces (hardware and software), the process of digitalisation, and accepted standards in these fields are essential for building and exploiting complex computing, communication, multimedia and measuring systems. Standards can simplify the design and construction of individual hardware and software components and help to ensure satisfactory interworking. Computer Standards & Interfaces is an international journal dealing specifically with these topics. The journal • Provides information about activities and progress on the definition of computer standards, software quality, interfaces and methods, at national, European and international levels • Publishes critical comments on standards and standards activities • Disseminates user''s experiences and case studies in the application and exploitation of established or emerging standards, interfaces and methods • Offers a forum for discussion on actual projects, standards, interfaces and methods by recognised experts • Stimulates relevant research by providing a specialised refereed medium.