Online prediction of activities with structure: Exploiting contextual associations and sequences

Abstract

Many human activities, given their intrinsic modularity, present structural information which can be exploited by classification algorithms: this enhances the capability of robots to predict activities. We introduce a semantic reasoning paradigm in which, via logical and statistical learning, we discriminate between actions on the basis of contextual associations. An example of this is considering the co-occurrence of scenario objects when predicting an action. We also combine such probabilistic reasoning with traditional sequence likelihood modeling. The system, given partial execution evidence of a task (e.g. assembling a car), first reasons in logical terms over qualitative primitives to constrain the space of possibilities, and then predicts the most sequentially likely action (e.g. `PickAnd-PutScrew'). A further claim is also the representation of actions in tractable logic, enabling online-capable recognition. Our evaluation, adopting annotated primitives of motion and tool usage, proves that simple sequence-only prediction methods (i.e. bigram sequence information, 59.80%) are outperformed by the proposed polynomial-time context- and sequence-aware inference (i.e. with 8 primitives, various degrees of partial evidence and bigram sequence information, 78.43%), proving the effectiveness of the combined approach.