Sketch-Based Interaction for Planning-Based Interactive Storytelling

Drawings have been used for thousands of years as a visual complement to oral and written storytelling. The evolution of technology and the advent of interactive narratives brings the possibility of exploring drawings and storytelling in new ways. This paper presents a new sketch-based interaction method for planning-based interactive storytelling systems, which uses a deep learning model based on a Convolutional Neural Network to recognize digital hand-drawn sketches. By combining real time sketch recognition with a planning-based plot generation algorithm, the proposed system allows users to interact with narratives by sketching objects on smartphones or tablet computers, which are then recognized by the system and converted into virtual objects in the story world, thereby affecting the plot of the narrative. Preliminary results show that the sketch recognition model has a remarkable accuracy for small sets of sketch classes (accuracy of 95.1 % for 14 classes), which are sufficient to provide a good variety of interaction options. In addition, it can also be extended to more complex scenarios while maintaining a considerable accuracy (87.4% for 172 classes and 71.6% for 345 classes).