Agents' Monitoring Approach for Big Data

Abstract

Data sources are becoming more prevalent as digital devices in the form of sensor networks, capture and record vast amounts of individual and environmental data. Much of this data is to some extent redundant, as it is captured or held at many locations, or is of a low priority level, so can safely be ignored. The distributed nature of such data, however, means that it is impossible to identify redundant or useless information without full scale analysis. The velocity of data arrival, the volume of data and the heterogeneous nature (variety) of the data makes this task unfeasible for any real time analysis, which is becoming more desirable in real world situations. Thus this paper is looking to utilize a multi-agent system or federation of agents to analyse data in a distributed manner. The method of setting up such an agent team is proposed so as to engender a cohesive team ethos endowing the agent federation with the power of a single agent's goal. It is then shown that this leads to a specific network topology to emerge within the agent team. Furthermore such a topology allows an acquaintance monitoring algorithm to be applied. This is shown to actively attenuate and prioritize data by suggesting only those sensor nodes that are likely to be in possession of useful and non-redundant data, using only data local to each agent team member. The results are gained, in this first instance, by a simulation.