An analysis of Watson vs. BARD vs. ChatGPT: The Jeopardy! Challenge

IF 2.5 4区计算机科学 Q3 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Ai Magazine Pub Date : 2023-08-30 DOI:10.1002/aaai.12118

Daniel E. O'Leary

{"title":"An analysis of Watson vs. BARD vs. ChatGPT: The Jeopardy! Challenge","authors":"Daniel E. O'Leary","doi":"10.1002/aaai.12118","DOIUrl":null,"url":null,"abstract":"<p>The recently released BARD and ChatGPT have generated substantial interest from a range of researchers and institutions concerned about the impact on education, medicine, law and more. This paper uses questions from the Watson Jeopardy! Challenge to compare BARD, ChatGPT, and Watson. Using those, Jeopardy! questions, we find that for high confidence Watson questions the three systems perform with similar accuracy as Watson. We also find that both BARD and ChatGPT perform with the accuracy of a human expert and that the sets of their correct answers are rated highly similar using a Tanimoto similarity score. However, in addition, we find that both systems can change their solutions to the same input information on subsequent uses. When given the same Jeopardy! category and question multiple times, both BARD and ChatGPT can generate different and conflicting answers. As a result, the paper examines the characteristics of some of those questions that generate different answers to the same inputs. Finally, the paper discusses some of the implications of finding the different answers and the impact of the lack of reproducibility on testing such systems.</p>","PeriodicalId":7854,"journal":{"name":"Ai Magazine","volume":"44 3","pages":"282-295"},"PeriodicalIF":2.5000,"publicationDate":"2023-08-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1002/aaai.12118","citationCount":"1","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Ai Magazine","FirstCategoryId":"94","ListUrlMain":"https://onlinelibrary.wiley.com/doi/10.1002/aaai.12118","RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE","Score":null,"Total":0}

引用次数: 1

Abstract

The recently released BARD and ChatGPT have generated substantial interest from a range of researchers and institutions concerned about the impact on education, medicine, law and more. This paper uses questions from the Watson Jeopardy! Challenge to compare BARD, ChatGPT, and Watson. Using those, Jeopardy! questions, we find that for high confidence Watson questions the three systems perform with similar accuracy as Watson. We also find that both BARD and ChatGPT perform with the accuracy of a human expert and that the sets of their correct answers are rated highly similar using a Tanimoto similarity score. However, in addition, we find that both systems can change their solutions to the same input information on subsequent uses. When given the same Jeopardy! category and question multiple times, both BARD and ChatGPT can generate different and conflicting answers. As a result, the paper examines the characteristics of some of those questions that generate different answers to the same inputs. Finally, the paper discusses some of the implications of finding the different answers and the impact of the lack of reproducibility on testing such systems.

查看原文本刊更多论文

Watson vs.BARD vs.ChatGPT:The Jeopardy的分析！挑战

最近发布的BARD和ChatGPT引起了一系列研究人员和机构的极大兴趣，他们担心这对教育、医学、法律等领域的影响。本文使用了《华生危险边缘》中的问题！挑战比较BARD、ChatGPT和Watson。使用这些，危险边缘！问题，我们发现对于高置信度Watson问题，三个系统的精度与Watson相似。我们还发现，BARD和ChatGPT的表现都具有人类专家的准确性，并且使用Tanimoto相似性得分对它们的正确答案集进行了高度相似的评级。然而，此外，我们发现，在后续使用中，两个系统都可以将其解决方案更改为相同的输入信息。当被给予同样的危险时！类别和问题多次，BARD和ChatGPT都可以生成不同且冲突的答案。因此，本文考察了其中一些问题的特征，这些问题对相同的输入产生了不同的答案。最后，本文讨论了找到不同答案的一些含义，以及缺乏再现性对测试此类系统的影响。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Ai Magazine 工程技术-计算机：人工智能

CiteScore

3.90

自引率

11.10%

发文量

审稿时长

>12 weeks

期刊介绍： AI Magazine publishes original articles that are reasonably self-contained and aimed at a broad spectrum of the AI community. Technical content should be kept to a minimum. In general, the magazine does not publish articles that have been published elsewhere in whole or in part. The magazine welcomes the contribution of articles on the theory and practice of AI as well as general survey articles, tutorial articles on timely topics, conference or symposia or workshop reports, and timely columns on topics of interest to AI scientists.