Towards a Generic Methodology for Evaluating MAS Performance

Abstract

As agent technology (AT) becomes a well-established engineering field of computing, the need for generalized, standardized methodologies for agent evaluation is imperative. Despite the plethora of available development tools and theories that researchers in agent computing have access to, there is a remarkable lack of general metrics, tools, benchmarks and experimental methods for formal validation and comparison of existing or newly developed systems. It is argued that AT has reached a certain degree of maturity, and it is therefore feasible to move from ad-hoc, domain-specific evaluation methods to standardized, repeatable and easily verifiable procedures. In this paper, we outline a first attempt towards a generic evaluation methodology for MAS performance. Instead of following the research path towards defining more powerful mathematical description tools for determining intelligence and performance metrics, we adopt an engineering point of view to the problem of deploying a methodology that is both implementation and domain independent. The proposed methodology consists of a concise set of steps, novel theoretical representation tools and appropriate software tools that assist evaluators in selecting the appropriate metrics, undertaking measurement and aggregation techniques for the system at hand