SpeechColab leaderboard: An open-source platform for automatic speech recognition evaluation

Abstract

In the wake of the surging tide of deep learning over the past decade, Automatic Speech Recognition (ASR) has garnered substantial attention, leading to the emergence of numerous publicly accessible ASR systems that are actively being integrated into our daily lives. Nonetheless, impartial and replicable evaluations of these ASR systems encounter challenges due to various subtleties. In this paper we introduce the SpeechColab Leaderboard, a general-purpose, open-source platform designed for ASR evaluation. With this platform: (i) We report a comprehensive benchmark, unveiling the current state-of-the-art panorama for ASR systems, covering both open-source models and industrial commercial services. (ii) We quantize how distinct nuances in the scoring pipeline influence the final benchmark outcomes, including capitalization, punctuation, interjection, contraction, synonym usage, compound words, etc. These issues have gained prominence in the context of the transition towards End-to-End ASR systems. (iii) We propose and discuss a modification to the conventional Token-Error-Rate (TER) metric, called modified-TER (mTER), inspired from Kolmogorov Complexity and Normalized Information Distance (NID). The proposed metric becomes normalized and symmetrical (with regard to reference and hypothesis). A large-scale empirical study is then presented comparing TER and mTER. The SpeechColab Leaderboard is accessible at https://github.com/SpeechColab/Leaderboard.