Evaluating Reader Comprehension of Plan-Based Stories Containing Failed Actions

A growing number of algorithms for story planning include the ability to create stories with failed actions -- in particular failed actions that occur because of the mistaken beliefs of the characters attempting them. To date, most of these systems have been evaluated analytically, primarily by comparing their expressive range to prior story generation systems. Empirical evaluation of these systems has been preliminary. In this paper, we outline a general comprehension-based approach to the evaluation of plan-based story generation. We describe how we specialize it for use evaluating story plans containing failed actions, and we describe the design and results of an experiment using this approach to evaluate plot lines produced by HeadSpace, a system that models the beliefs of characters and uses that model to generate plot lines containing actions that are attempted but that fail.