Preferred Benchmarking Criteria for Systematic Taxonomy of Embedded Platforms (STEP) in Human System Interaction Systems

Abstract

The rate of progress in the field of Artificial Intelligence (AI) and Machine Learning (ML) has significantly increased over the past ten years and continues to accelerate. Since then, AI has made the leap from research case studies to real production ready applications. The significance of this growth cannot be undermined as it catalyzed the very nature of computing. Conventional platforms struggle to achieve greater performance and efficiency, what causes a surging demand for innovative AI accelerators, specialized platforms and purpose-built computes. At the same time, it is required to provide solutions for assessment of ML platform performance in a reproducible and unbiased manner to be able to provide a fair comparison of different products. This is especially valid for Human System Interaction (HSI) systems that require specific data handling for low latency responses in emergency situations or to improve user experience, as well as for preserving data privacy and security by processing it locally. Taking it into account, this work presents a comprehensive guideline on preferred benchmarking criteria for evaluation of ML platforms that include both lower level analysis of ML models and system-level evaluation of the entire pipeline. In addition, we propose a Systematic Taxonomy of Embedded Platforms (STEP) that can be used by the community and customers for better selection of specific ML hardware consistent with their needs for better design of ML-based HSI solutions.