Marek Urban, Cyril Brom, Jiří Lukavský, Filip Děchtěrenko, Veronika Hein, Filip Svacha, Petra Kmoníčková, Kamila Urban
{"title":"“ChatGPT可能会犯错误。查看重要信息。”学生将ChatGPT内容整合到学术写作中的认知信念与元认知准确性","authors":"Marek Urban, Cyril Brom, Jiří Lukavský, Filip Děchtěrenko, Veronika Hein, Filip Svacha, Petra Kmoníčková, Kamila Urban","doi":"10.1111/bjet.13591","DOIUrl":null,"url":null,"abstract":"<div>\n \n <section>\n \n <p>Recent studies have conceptualized ChatGPT as an epistemic authority; however, no research has yet examined how epistemic beliefs and metacognitive accuracy affect students' actual use of ChatGPT-generated content, which often contains factual inaccuracies. Therefore, the present experimental study aimed to examine how university students integrate correct and incorrect information from expert-written and ChatGPT-generated articles when writing independently (<i>N</i> = 49) or with ChatGPT assistance (<i>N</i> = 49). Students working with ChatGPT-4o integrated more correct information from both expert-written (<i>d</i> = 0.64) and ChatGPT-generated articles (<i>d</i> = 0.95), but ChatGPT-assisted writing did not affect the amount of incorrect information sourced from the ChatGPT-generated article. Regardless of the condition, hierarchical regressions revealed that lower metacognitive bias was moderately associated with increased inclusion of correct information from the expert-written article (<i>R</i><sup>2</sup> = 12%). Conversely, a higher metacognitive bias (<i>R</i><sup>2</sup> = 10%) and epistemic beliefs (<i>R</i><sup>2</sup> = 12%) were moderately related to the inclusion of incorrect information from ChatGPT-generated articles. These findings suggest that while ChatGPT assistance enhances the integration of correct human- and AI-generated content, metacognitive skills remain essential to mitigate the risks of incorporating incorrect AI-generated information.</p>\n </section>\n \n <section>\n \n <div>\n \n <div>\n \n <h3>Practitioner notes</h3>\n <p>What is already known about this topic\n\n </p><ul>\n \n <li>Generative AI tools, such as ChatGPT, are increasingly regarded as epistemic authorities due to their authoritative tone and human-like interaction.</li>\n \n <li>ChatGPT has demonstrated utility in providing correct information and improving productivity in educational and professional contexts, but it is also prone to inaccuracies, hallucinations and misleading content.</li>\n \n <li>Students' epistemic beliefs and metacognitive skills predict their ability to critically evaluate and integrate conflicting information from multiple resources, particularly when searching for information on the Internet.</li>\n </ul>\n <p>What this paper adds\n\n </p><ul>\n \n <li>This study experimentally examines how students integrate correct and incorrect information from expert-written and ChatGPT-generated articles when writing independently or with ChatGPT's assistance.</li>\n \n <li>The findings show that ChatGPT assistance improves the inclusion of correct information but does not significantly reduce or increase the inclusion of incorrect ChatGPT-generated content.</li>\n \n <li>Metacognitive accuracy and epistemic beliefs are key factors in mitigating the inclusion of incorrect information, regardless of whether students work independently or with ChatGPT.</li>\n </ul>\n <p>Implications for practice and/or policy\n\n </p><ul>\n \n <li>Generative AI tools can outperform human experts in specific scenarios, requiring little to no evaluation. However, in situations where these tools generate misleading or incorrect content, the application of metacognitive skills and epistemic beliefs becomes essential to discern reliable information and avoid the integration of errors.</li>\n \n <li>Educational interventions should include activities requiring justification of knowledge, evaluation of resources and reflection upon human-generated and AI-generated texts to enhance students' ability to discern accurate from inaccurate information.</li>\n \n <li>Interventions focused on metacognitive accuracy and epistemic awareness can empower individuals to critically evaluate and differentiate between reliable and erroneous information, enhancing their recognition of misinformation.</li>\n </ul>\n </div>\n </div>\n </section>\n </div>","PeriodicalId":48315,"journal":{"name":"British Journal of Educational Technology","volume":"56 5","pages":"1897-1918"},"PeriodicalIF":8.1000,"publicationDate":"2025-04-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"“ChatGPT can make mistakes. Check important info.” Epistemic beliefs and metacognitive accuracy in students' integration of ChatGPT content into academic writing\",\"authors\":\"Marek Urban, Cyril Brom, Jiří Lukavský, Filip Děchtěrenko, Veronika Hein, Filip Svacha, Petra Kmoníčková, Kamila Urban\",\"doi\":\"10.1111/bjet.13591\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<div>\\n \\n <section>\\n \\n <p>Recent studies have conceptualized ChatGPT as an epistemic authority; however, no research has yet examined how epistemic beliefs and metacognitive accuracy affect students' actual use of ChatGPT-generated content, which often contains factual inaccuracies. Therefore, the present experimental study aimed to examine how university students integrate correct and incorrect information from expert-written and ChatGPT-generated articles when writing independently (<i>N</i> = 49) or with ChatGPT assistance (<i>N</i> = 49). Students working with ChatGPT-4o integrated more correct information from both expert-written (<i>d</i> = 0.64) and ChatGPT-generated articles (<i>d</i> = 0.95), but ChatGPT-assisted writing did not affect the amount of incorrect information sourced from the ChatGPT-generated article. Regardless of the condition, hierarchical regressions revealed that lower metacognitive bias was moderately associated with increased inclusion of correct information from the expert-written article (<i>R</i><sup>2</sup> = 12%). Conversely, a higher metacognitive bias (<i>R</i><sup>2</sup> = 10%) and epistemic beliefs (<i>R</i><sup>2</sup> = 12%) were moderately related to the inclusion of incorrect information from ChatGPT-generated articles. These findings suggest that while ChatGPT assistance enhances the integration of correct human- and AI-generated content, metacognitive skills remain essential to mitigate the risks of incorporating incorrect AI-generated information.</p>\\n </section>\\n \\n <section>\\n \\n <div>\\n \\n <div>\\n \\n <h3>Practitioner notes</h3>\\n <p>What is already known about this topic\\n\\n </p><ul>\\n \\n <li>Generative AI tools, such as ChatGPT, are increasingly regarded as epistemic authorities due to their authoritative tone and human-like interaction.</li>\\n \\n <li>ChatGPT has demonstrated utility in providing correct information and improving productivity in educational and professional contexts, but it is also prone to inaccuracies, hallucinations and misleading content.</li>\\n \\n <li>Students' epistemic beliefs and metacognitive skills predict their ability to critically evaluate and integrate conflicting information from multiple resources, particularly when searching for information on the Internet.</li>\\n </ul>\\n <p>What this paper adds\\n\\n </p><ul>\\n \\n <li>This study experimentally examines how students integrate correct and incorrect information from expert-written and ChatGPT-generated articles when writing independently or with ChatGPT's assistance.</li>\\n \\n <li>The findings show that ChatGPT assistance improves the inclusion of correct information but does not significantly reduce or increase the inclusion of incorrect ChatGPT-generated content.</li>\\n \\n <li>Metacognitive accuracy and epistemic beliefs are key factors in mitigating the inclusion of incorrect information, regardless of whether students work independently or with ChatGPT.</li>\\n </ul>\\n <p>Implications for practice and/or policy\\n\\n </p><ul>\\n \\n <li>Generative AI tools can outperform human experts in specific scenarios, requiring little to no evaluation. However, in situations where these tools generate misleading or incorrect content, the application of metacognitive skills and epistemic beliefs becomes essential to discern reliable information and avoid the integration of errors.</li>\\n \\n <li>Educational interventions should include activities requiring justification of knowledge, evaluation of resources and reflection upon human-generated and AI-generated texts to enhance students' ability to discern accurate from inaccurate information.</li>\\n \\n <li>Interventions focused on metacognitive accuracy and epistemic awareness can empower individuals to critically evaluate and differentiate between reliable and erroneous information, enhancing their recognition of misinformation.</li>\\n </ul>\\n </div>\\n </div>\\n </section>\\n </div>\",\"PeriodicalId\":48315,\"journal\":{\"name\":\"British Journal of Educational Technology\",\"volume\":\"56 5\",\"pages\":\"1897-1918\"},\"PeriodicalIF\":8.1000,\"publicationDate\":\"2025-04-09\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"British Journal of Educational Technology\",\"FirstCategoryId\":\"95\",\"ListUrlMain\":\"https://bera-journals.onlinelibrary.wiley.com/doi/10.1111/bjet.13591\",\"RegionNum\":1,\"RegionCategory\":\"教育学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"EDUCATION & EDUCATIONAL RESEARCH\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"British Journal of Educational Technology","FirstCategoryId":"95","ListUrlMain":"https://bera-journals.onlinelibrary.wiley.com/doi/10.1111/bjet.13591","RegionNum":1,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}
“ChatGPT can make mistakes. Check important info.” Epistemic beliefs and metacognitive accuracy in students' integration of ChatGPT content into academic writing
Recent studies have conceptualized ChatGPT as an epistemic authority; however, no research has yet examined how epistemic beliefs and metacognitive accuracy affect students' actual use of ChatGPT-generated content, which often contains factual inaccuracies. Therefore, the present experimental study aimed to examine how university students integrate correct and incorrect information from expert-written and ChatGPT-generated articles when writing independently (N = 49) or with ChatGPT assistance (N = 49). Students working with ChatGPT-4o integrated more correct information from both expert-written (d = 0.64) and ChatGPT-generated articles (d = 0.95), but ChatGPT-assisted writing did not affect the amount of incorrect information sourced from the ChatGPT-generated article. Regardless of the condition, hierarchical regressions revealed that lower metacognitive bias was moderately associated with increased inclusion of correct information from the expert-written article (R2 = 12%). Conversely, a higher metacognitive bias (R2 = 10%) and epistemic beliefs (R2 = 12%) were moderately related to the inclusion of incorrect information from ChatGPT-generated articles. These findings suggest that while ChatGPT assistance enhances the integration of correct human- and AI-generated content, metacognitive skills remain essential to mitigate the risks of incorporating incorrect AI-generated information.
Practitioner notes
What is already known about this topic
Generative AI tools, such as ChatGPT, are increasingly regarded as epistemic authorities due to their authoritative tone and human-like interaction.
ChatGPT has demonstrated utility in providing correct information and improving productivity in educational and professional contexts, but it is also prone to inaccuracies, hallucinations and misleading content.
Students' epistemic beliefs and metacognitive skills predict their ability to critically evaluate and integrate conflicting information from multiple resources, particularly when searching for information on the Internet.
What this paper adds
This study experimentally examines how students integrate correct and incorrect information from expert-written and ChatGPT-generated articles when writing independently or with ChatGPT's assistance.
The findings show that ChatGPT assistance improves the inclusion of correct information but does not significantly reduce or increase the inclusion of incorrect ChatGPT-generated content.
Metacognitive accuracy and epistemic beliefs are key factors in mitigating the inclusion of incorrect information, regardless of whether students work independently or with ChatGPT.
Implications for practice and/or policy
Generative AI tools can outperform human experts in specific scenarios, requiring little to no evaluation. However, in situations where these tools generate misleading or incorrect content, the application of metacognitive skills and epistemic beliefs becomes essential to discern reliable information and avoid the integration of errors.
Educational interventions should include activities requiring justification of knowledge, evaluation of resources and reflection upon human-generated and AI-generated texts to enhance students' ability to discern accurate from inaccurate information.
Interventions focused on metacognitive accuracy and epistemic awareness can empower individuals to critically evaluate and differentiate between reliable and erroneous information, enhancing their recognition of misinformation.
期刊介绍：
BJET is a primary source for academics and professionals in the fields of digital educational and training technology throughout the world. The Journal is published by Wiley on behalf of The British Educational Research Association (BERA). It publishes theoretical perspectives, methodological developments and high quality empirical research that demonstrate whether and how applications of instructional/educational technology systems, networks, tools and resources lead to improvements in formal and non-formal education at all levels, from early years through to higher, technical and vocational education, professional development and corporate training.
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号
