Controlling Swarms by Visual Demonstration

Agent-based modeling is a paradigm of modeling dynamic systems of interacting agents that are individually governed by specified behavioral rules. Training a model of such agents to produce an emergent behavior by specification of the emergent (as opposed to agent) behavior is easier from a demonstration perspective. While many approaches involve manual behavior specification via code or reliance on a defined taxonomy of possible behaviors, the meta-modeling framework in existing work generates mapping functions between agent-level parameters and swarm-level parameters which are re-usable once generated. This work builds on that framework by integrating demonstration by image or video. The demonstrator specifies spatial motion of the agents over time, and retrieves agent-level parameters required to execute that motion. The framework, at its core, uses computationally cheap image processing algorithms. Our work is tested with a combination of primitive visual feature extraction methods (contour area and shape) and features generated using a pre-trained deep neural network in different stages of image featurization. Experimental results show significant coherence between demonstrated behavior and predicted behavior based on estimated agent-level parameters specific to a class of behaviors (spatial arrangement of agents).