Prompting for creative problem-solving: A process-mining study

IF 4.7 1区教育学 Q1 EDUCATION & EDUCATIONAL RESEARCH

Learning and Instruction Pub Date : 2025-05-23 DOI:10.1016/j.learninstruc.2025.102156

Marek Urban , Jiří Lukavský , Cyril Brom , Veronika Hein , Filip Svacha , Filip Děchtěrenko , Kamila Urban

{"title":"Prompting for creative problem-solving: A process-mining study","authors":"Marek Urban , Jiří Lukavský , Cyril Brom , Veronika Hein , Filip Svacha , Filip Děchtěrenko , Kamila Urban","doi":"10.1016/j.learninstruc.2025.102156","DOIUrl":null,"url":null,"abstract":"<div><h3>Background</h3><div>Although generative-AI systems are increasingly used to solve non-routine problems, effective prompting strategies remain largely underexplored.</div></div><div><h3>Aims</h3><div>The present study investigates how university students prompt ChatGPT to solve complex ill-defined problems, specifically examining which prompts are associated with higher or lower problem-solving performance.</div></div><div><h3>Sample</h3><div>Seventy-seven university students (53 women; <em>M</em><sub>age</sub> = 22.4 years) participated in the study.</div></div><div><h3>Methods</h3><div>To identify various prompt types employed by students, the study utilized qualitative analysis of interactions with ChatGPT 3.5 during the resolution of the creative problem-solving task. Participants’ performance was measured by the quality, elaboration, and originality of their ideas. Subsequently, two-step clustering was employed to identify groups of low- and high-performing students. Finally, process-mining techniques (heuristics miner) were used to analyze the interactions of low- and high-performing students.</div></div><div><h3>Results</h3><div>The findings suggest that including clear evaluation criteria when prompting ChatGPT to generate ideas (<em>r</em><sub>s</sub> = .38), providing ChatGPT with an elaborated context for idea generation (<em>r</em><sub>s</sub> = .47), and offering specific feedback (<em>r</em><sub>s</sub> = .45), enhances the quality, elaboration, and originality of the solutions. Successful problem-solving involves iterative human-AI regulation, with high performers using an overall larger number of prompts (<em>d</em> = .82). High performers interacted with ChatGPT through dialogue, where they monitored and regulated the generation of ideas, while low performers used ChatGPT as an information resource.</div></div><div><h3>Conclusions</h3><div>These results emphasize the importance of active and iterative engagement for creative problem-solving and suggest that educational practices should foster metacognitive monitoring and regulation to maximize the benefits of human-AI collaboration.</div></div>","PeriodicalId":48357,"journal":{"name":"Learning and Instruction","volume":"99 ","pages":"Article 102156"},"PeriodicalIF":4.7000,"publicationDate":"2025-05-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Learning and Instruction","FirstCategoryId":"95","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0959475225000805","RegionNum":1,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}

引用次数: 0

Abstract

Background

Although generative-AI systems are increasingly used to solve non-routine problems, effective prompting strategies remain largely underexplored.

Aims

The present study investigates how university students prompt ChatGPT to solve complex ill-defined problems, specifically examining which prompts are associated with higher or lower problem-solving performance.

Sample

Seventy-seven university students (53 women; M_age = 22.4 years) participated in the study.

Methods

To identify various prompt types employed by students, the study utilized qualitative analysis of interactions with ChatGPT 3.5 during the resolution of the creative problem-solving task. Participants’ performance was measured by the quality, elaboration, and originality of their ideas. Subsequently, two-step clustering was employed to identify groups of low- and high-performing students. Finally, process-mining techniques (heuristics miner) were used to analyze the interactions of low- and high-performing students.

Results

The findings suggest that including clear evaluation criteria when prompting ChatGPT to generate ideas (r_s = .38), providing ChatGPT with an elaborated context for idea generation (r_s = .47), and offering specific feedback (r_s = .45), enhances the quality, elaboration, and originality of the solutions. Successful problem-solving involves iterative human-AI regulation, with high performers using an overall larger number of prompts (d = .82). High performers interacted with ChatGPT through dialogue, where they monitored and regulated the generation of ideas, while low performers used ChatGPT as an information resource.

Conclusions

These results emphasize the importance of active and iterative engagement for creative problem-solving and suggest that educational practices should foster metacognitive monitoring and regulation to maximize the benefits of human-AI collaboration.

查看原文本刊更多论文

创造性问题解决的提示：过程挖掘研究

尽管生成型人工智能系统越来越多地用于解决非常规问题，但有效的提示策略在很大程度上仍未得到充分探索。目的本研究调查了大学生如何提示ChatGPT解决复杂的不明确的问题，特别是检查哪些提示与更高或更低的解决问题的表现有关。77名大学生(女性53名；法师= 22.4岁)参加了这项研究。方法为了识别学生使用的各种提示类型，本研究利用ChatGPT 3.5在解决创造性问题任务过程中的交互作用进行定性分析。参与者的表现是通过他们想法的质量、阐述和独创性来衡量的。随后，采用两步聚类方法对表现较差的学生和表现较好的学生进行分组。最后，使用过程挖掘技术（启发式挖掘）来分析低绩效学生和高绩效学生之间的互动。研究结果表明，在促使ChatGPT产生想法时，包括明确的评估标准（rs = .38），为ChatGPT提供一个详细的想法产生环境（rs = .47），并提供具体的反馈（rs = .45），可以提高解决方案的质量，精细化和独创性。成功的问题解决涉及迭代的人类-人工智能调节，高绩效者使用大量的提示（d = .82）。高绩效人员通过对话与ChatGPT进行互动，在对话中他们监督和规范想法的产生，而低绩效人员则将ChatGPT作为信息资源使用。这些结果强调了积极和迭代参与对于创造性解决问题的重要性，并建议教育实践应促进元认知监测和监管，以最大限度地发挥人类与人工智能协作的好处。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Learning and Instruction Multiple-

CiteScore

11.30

自引率

4.80%

发文量

109

期刊介绍： As an international, multi-disciplinary, peer-refereed journal, Learning and Instruction provides a platform for the publication of the most advanced scientific research in the areas of learning, development, instruction and teaching. The journal welcomes original empirical investigations. The papers may represent a variety of theoretical perspectives and different methodological approaches. They may refer to any age level, from infants to adults and to a diversity of learning and instructional settings, from laboratory experiments to field studies. The major criteria in the review and the selection process concern the significance of the contribution to the area of learning and instruction, and the rigor of the study.