ChatGPT Needs a Chemistry Tutor Too

IF 2.5 3区教育学 Q2 CHEMISTRY, MULTIDISCIPLINARY

Journal of Chemical Education Pub Date : 2023-09-07 DOI:10.1021/acs.jchemed.3c00288

Alfredo J. Leon*, and , Dinesh Vidhani,

{"title":"ChatGPT Needs a Chemistry Tutor Too","authors":"Alfredo J. Leon*,  and , Dinesh Vidhani, ","doi":"10.1021/acs.jchemed.3c00288","DOIUrl":null,"url":null,"abstract":"<p >Artificial intelligence (AI) technology has the potential to revolutionize the education sector. This study sought to determine the efficacy of ChatGPT to correctly answer questions a learner would use and to elucidate how the AI was processing potential prompts. Our goal was to evaluate the role of prompt formats, response consistency, and reliability of ChatGPT responses. Analyzing prompt format, we see that the data do not demonstrate a statistically significant difference between multiple-choice and free-response questions. Neither format achieved scores higher than 37%, and testing at different locations did not improve scores. Interestingly, ChatGPT’s free version provides accurate responses to discipline-specific questions that contain information from unrelated topics as distractors, improving its accuracy over the free-response questions. It is important to consider, while ChatGPT can identify the correct answer within a given context, it may not be able to determine if the answer it selects is correct computationally or through analysis. The results of this study can guide future AI and ChatGPT training practices and implementations to ensure they are used to their fullest potential.</p>","PeriodicalId":43,"journal":{"name":"Journal of Chemical Education","volume":"100 10","pages":"3859–3865"},"PeriodicalIF":2.5000,"publicationDate":"2023-09-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"2","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Chemical Education","FirstCategoryId":"92","ListUrlMain":"https://pubs.acs.org/doi/10.1021/acs.jchemed.3c00288","RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"CHEMISTRY, MULTIDISCIPLINARY","Score":null,"Total":0}

引用次数: 2

Abstract

Artificial intelligence (AI) technology has the potential to revolutionize the education sector. This study sought to determine the efficacy of ChatGPT to correctly answer questions a learner would use and to elucidate how the AI was processing potential prompts. Our goal was to evaluate the role of prompt formats, response consistency, and reliability of ChatGPT responses. Analyzing prompt format, we see that the data do not demonstrate a statistically significant difference between multiple-choice and free-response questions. Neither format achieved scores higher than 37%, and testing at different locations did not improve scores. Interestingly, ChatGPT’s free version provides accurate responses to discipline-specific questions that contain information from unrelated topics as distractors, improving its accuracy over the free-response questions. It is important to consider, while ChatGPT can identify the correct answer within a given context, it may not be able to determine if the answer it selects is correct computationally or through analysis. The results of this study can guide future AI and ChatGPT training practices and implementations to ensure they are used to their fullest potential.

Abstract Image

查看原文本刊更多论文

ChatGPT也需要化学导师

人工智能（AI）技术有可能彻底改变教育部门。这项研究试图确定ChatGPT在正确回答学习者会使用的问题方面的功效，并阐明人工智能是如何处理潜在提示的。我们的目标是评估ChatGPT响应的提示格式、响应一致性和可靠性的作用。通过分析提示形式，我们发现数据并没有显示出多项选择题和自由回答题之间的统计学显著差异。两种形式的得分都不超过37%，不同地点的测试也没有提高分数。有趣的是，ChatGPT的免费版本提供了对学科特定问题的准确回答，这些问题包含来自无关主题的信息作为干扰因素，从而提高了其相对于免费回答问题的准确性。重要的是要考虑，虽然ChatGPT可以在给定的上下文中识别正确的答案，但它可能无法通过计算或分析来确定它选择的答案是否正确。这项研究的结果可以指导未来的人工智能和ChatGPT培训实践和实施，以确保它们充分发挥潜力。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Journal of Chemical Education 化学-化学综合

CiteScore

5.60

自引率

50.00%

发文量

465

审稿时长

6.5 months

期刊介绍： The Journal of Chemical Education is the official journal of the Division of Chemical Education of the American Chemical Society, co-published with the American Chemical Society Publications Division. Launched in 1924, the Journal of Chemical Education is the world’s premier chemical education journal. The Journal publishes peer-reviewed articles and related information as a resource to those in the field of chemical education and to those institutions that serve them. JCE typically addresses chemical content, activities, laboratory experiments, instructional methods, and pedagogies. The Journal serves as a means of communication among people across the world who are interested in the teaching and learning of chemistry. This includes instructors of chemistry from middle school through graduate school, professional staff who support these teaching activities, as well as some scientists in commerce, industry, and government.