Agent orientated annotation in model based visual surveillance

The paper presents an agent based surveillance system for use in monitoring scenes involving both pedestrians and vehicles. The system supplies textual descriptions for the dynamic activity occurring in the 3D world. These are derived by means of dynamic and probabilistic inference based on geometric information provided by a vision system that tracks vehicles and pedestrians. The symbolic scene annotation is given at two major levels of description: the object level and the inter-object level. At object level, each tracked pedestrian or vehicle is assigned a behaviour agent which uses a Bayesian network to infer the fundamental features of the objects' trajectory, and continuously updates its textual description. The inter-object interaction level is interpreted by a situation agent which is created dynamically when two objects are in close proximity. In the work included here the situation agent can describe a two-object interaction in terms of basic textual annotations, to summarise the dynamics of the local action.