{"title":"Can Generative AI and ChatGPT Outperform Humans on Cognitive-Demanding Problem-Solving Tasks in Science?","authors":"Xiaoming Zhai, Matthew Nyaaba, Wenchao Ma","doi":"10.1007/s11191-024-00496-1","DOIUrl":null,"url":null,"abstract":"<h3>Abstract</h3> <p>This study aimed to examine an assumption regarding whether generative artificial intelligence (GAI) tools can overcome the cognitive intensity that humans suffer when solving problems. We examine the performance of ChatGPT and GPT-4 on NAEP science assessments and compare their performance to students by cognitive demands of the items. Fifty-four 2019 NAEP science assessment tasks were coded by content experts using a two-dimensional cognitive load framework, including task cognitive complexity and dimensionality. ChatGPT and GPT-4 answered the questions individually and were scored using the scoring keys provided by NAEP. The analysis of the available data for this study was based on the average student ability scores for students who answered each item correctly and the percentage of students who responded to individual items. The results showed that both ChatGPT and GPT-4 consistently outperformed most students who answered each individual item in the NAEP science assessments. As the cognitive demand for NAEP science assessments increases, statistically higher average student ability scores are required to correctly address the questions. This pattern was observed for Grades 4, 8, and 12 students respectively. However, ChatGPT and GPT-4 were not statistically sensitive to the increase of cognitive demands of the tasks, except for Grade 4. As the first study focusing on comparing cutting-edge GAI and K-12 students in problem-solving in science, this finding implies the need for changes to educational objectives to prepare students with competence to work with GAI tools such as ChatGPT and GPT-4 in the future. Education ought to emphasize the cultivation of advanced cognitive skills rather than depending solely on tasks that demand cognitive intensity. This approach would foster critical thinking, analytical skills, and the application of knowledge in novel contexts among students. Furthermore, the findings suggest that researchers should innovate assessment practices by moving away from cognitive intensity tasks toward creativity and analytical skills to more efficiently avoid the negative effects of GAI on testing.</p>","PeriodicalId":771,"journal":{"name":"Science & Education","volume":"392 1 1","pages":""},"PeriodicalIF":3.1000,"publicationDate":"2024-01-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Science & Education","FirstCategoryId":"95","ListUrlMain":"https://doi.org/10.1007/s11191-024-00496-1","RegionNum":1,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}
引用次数: 0
Abstract
This study aimed to examine an assumption regarding whether generative artificial intelligence (GAI) tools can overcome the cognitive intensity that humans suffer when solving problems. We examine the performance of ChatGPT and GPT-4 on NAEP science assessments and compare their performance to students by cognitive demands of the items. Fifty-four 2019 NAEP science assessment tasks were coded by content experts using a two-dimensional cognitive load framework, including task cognitive complexity and dimensionality. ChatGPT and GPT-4 answered the questions individually and were scored using the scoring keys provided by NAEP. The analysis of the available data for this study was based on the average student ability scores for students who answered each item correctly and the percentage of students who responded to individual items. The results showed that both ChatGPT and GPT-4 consistently outperformed most students who answered each individual item in the NAEP science assessments. As the cognitive demand for NAEP science assessments increases, statistically higher average student ability scores are required to correctly address the questions. This pattern was observed for Grades 4, 8, and 12 students respectively. However, ChatGPT and GPT-4 were not statistically sensitive to the increase of cognitive demands of the tasks, except for Grade 4. As the first study focusing on comparing cutting-edge GAI and K-12 students in problem-solving in science, this finding implies the need for changes to educational objectives to prepare students with competence to work with GAI tools such as ChatGPT and GPT-4 in the future. Education ought to emphasize the cultivation of advanced cognitive skills rather than depending solely on tasks that demand cognitive intensity. This approach would foster critical thinking, analytical skills, and the application of knowledge in novel contexts among students. Furthermore, the findings suggest that researchers should innovate assessment practices by moving away from cognitive intensity tasks toward creativity and analytical skills to more efficiently avoid the negative effects of GAI on testing.
期刊介绍：
Science Education publishes original articles on the latest issues and trends occurring internationally in science curriculum, instruction, learning, policy and preparation of science teachers with the aim to advance our knowledge of science education theory and practice. In addition to original articles, the journal features the following special sections: -Learning : consisting of theoretical and empirical research studies on learning of science. We invite manuscripts that investigate learning and its change and growth from various lenses, including psychological, social, cognitive, sociohistorical, and affective. Studies examining the relationship of learning to teaching, the science knowledge and practices, the learners themselves, and the contexts (social, political, physical, ideological, institutional, epistemological, and cultural) are similarly welcome. -Issues and Trends : consisting primarily of analytical, interpretive, or persuasive essays on current educational, social, or philosophical issues and trends relevant to the teaching of science. This special section particularly seeks to promote informed dialogues about current issues in science education, and carefully reasoned papers representing disparate viewpoints are welcomed. Manuscripts submitted for this section may be in the form of a position paper, a polemical piece, or a creative commentary. -Science Learning in Everyday Life : consisting of analytical, interpretative, or philosophical papers regarding learning science outside of the formal classroom. Papers should investigate experiences in settings such as community, home, the Internet, after school settings, museums, and other opportunities that develop science interest, knowledge or practices across the life span. Attention to issues and factors relating to equity in science learning are especially encouraged.. -Science Teacher Education [...]
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号
