Investigating the Effect of Chatbot-to-User Questions and Directives on Student Participation

Abstract

Pedagogical chatbots are used to elicit information from students. Yet, how the amount of student-generated content differs according to the type of chatbot-delivered questions is unknown. To this end, 19 South Korean English majors completed six chatbot assignments through an in-house developed Facebook Messenger chatbot. The chatbot activity entailed creating original stories for class presentations. In addition to directives requesting plot details, the chatbot used closed-ended button reply questions, open-ended questions, and fill-in-the-blank template statements to help students create stories. Results indicated that button reply questions allowed for pacing, recall and content assessment and required low levels of critical thinking. Next, open-ended questions and fill-in-the-blank template statements resulted in similar word count production but different levels of creativity and critical thinking. Lastly, directives requesting user input resulted in 35% more output, indicating students took more action when told to do something than when asked. Regarding the novelty effect, fewer students volunteered to do the sixth and final chatbot activity, but those who did produced word count on par with their initial chatbot activity.