ChatGPT对汉语理解能力的评价

IF 1.3 3区计算机科学 Q3 COMPUTER SCIENCE, INFORMATION SYSTEMS

Data Intelligence Pub Date : 2023-09-12 DOI:10.1162/dint_a_00232

Linhan Li, Huaping Zhang, Chunjin Li, Haowen You, Wenyao Cui

{"title":"ChatGPT对汉语理解能力的评价","authors":"Linhan Li, Huaping Zhang, Chunjin Li, Haowen You, Wenyao Cui","doi":"10.1162/dint_a_00232","DOIUrl":null,"url":null,"abstract":"Abstract ChatGPT has attracted extension attention of academia and industry. This paper aims to evaluate ChatGPT in Chinese language understanding capability on 6 tasks using 11 datasets. Experiments indicate that ChatGPT achieved competitive results in sentiment analysis, summary, and reading comprehension in Chinese, while it is prone to factual errors in closed-book QA. Further, on two more difficult Chinese understanding tasks, that is, idiom fill-in-the-blank and cants understanding, we found that a simple chain-of-thought prompt can improve the accuracy of ChatGPT in complex reasoning. This paper further analyses the possible risks of using ChatGPT based on the results. Finally, we briefly describe the research and development progress of our ChatBIT.","PeriodicalId":34023,"journal":{"name":"Data Intelligence","volume":"14 1","pages":"0"},"PeriodicalIF":1.3000,"publicationDate":"2023-09-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Evaluation on ChatGPT for Chinese Language Understanding\",\"authors\":\"Linhan Li, Huaping Zhang, Chunjin Li, Haowen You, Wenyao Cui\",\"doi\":\"10.1162/dint_a_00232\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"Abstract ChatGPT has attracted extension attention of academia and industry. This paper aims to evaluate ChatGPT in Chinese language understanding capability on 6 tasks using 11 datasets. Experiments indicate that ChatGPT achieved competitive results in sentiment analysis, summary, and reading comprehension in Chinese, while it is prone to factual errors in closed-book QA. Further, on two more difficult Chinese understanding tasks, that is, idiom fill-in-the-blank and cants understanding, we found that a simple chain-of-thought prompt can improve the accuracy of ChatGPT in complex reasoning. This paper further analyses the possible risks of using ChatGPT based on the results. Finally, we briefly describe the research and development progress of our ChatBIT.\",\"PeriodicalId\":34023,\"journal\":{\"name\":\"Data Intelligence\",\"volume\":\"14 1\",\"pages\":\"0\"},\"PeriodicalIF\":1.3000,\"publicationDate\":\"2023-09-12\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Data Intelligence\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.1162/dint_a_00232\",\"RegionNum\":3,\"RegionCategory\":\"计算机科学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q3\",\"JCRName\":\"COMPUTER SCIENCE, INFORMATION SYSTEMS\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Data Intelligence","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1162/dint_a_00232","RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"COMPUTER SCIENCE, INFORMATION SYSTEMS","Score":null,"Total":0}

引用次数: 0

摘要

ChatGPT已经引起了学术界和工业界的广泛关注。本文旨在使用11个数据集评估ChatGPT在6个任务上的中文理解能力。实验表明，ChatGPT在中文情感分析、摘要和阅读理解方面取得了较好的效果，但在闭卷问答中容易出现事实错误。此外，在两个难度更高的汉语理解任务，即习语填空和俚语理解上，我们发现一个简单的思维链提示可以提高ChatGPT在复杂推理中的准确性。本文在此基础上进一步分析了使用ChatGPT可能存在的风险。最后，简要介绍了ChatBIT的研究与开发进展。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

查看原文本刊更多论文

Evaluation on ChatGPT for Chinese Language Understanding

Abstract ChatGPT has attracted extension attention of academia and industry. This paper aims to evaluate ChatGPT in Chinese language understanding capability on 6 tasks using 11 datasets. Experiments indicate that ChatGPT achieved competitive results in sentiment analysis, summary, and reading comprehension in Chinese, while it is prone to factual errors in closed-book QA. Further, on two more difficult Chinese understanding tasks, that is, idiom fill-in-the-blank and cants understanding, we found that a simple chain-of-thought prompt can improve the accuracy of ChatGPT in complex reasoning. This paper further analyses the possible risks of using ChatGPT based on the results. Finally, we briefly describe the research and development progress of our ChatBIT.

求助全文

通过发布文献求助，成功后即可免费获取论文全文。去求助

来源期刊

Data Intelligence COMPUTER SCIENCE, INFORMATION SYSTEMS-

CiteScore

6.50

自引率

15.40%

发文量

审稿时长

8 weeks