Evaluation and Development of Onomatopoeia CAPTCHAs

The Completely Automated Public Turing Test to Tell Computers and Humans Apart (CAPTCHA) is a type of challenge–response test used for avoiding malicious software during automated registration. It plays an important role in security because fraud with computer agents is serious. The requirements of CAPTCHAs are: 1) a human can easily solve a CAPTCHA, 2) a computer cannot solve a CAPTCHA, and 3) CAPTCHAs can be generated automatically. As deep learning technology has been developed, many of the existing CAPTCHAs were compromised and fail to satisfy condition (2). In this study, we propose a new “onomatopoeia CAPTCHA” that applies onomatopoeia; i.e., words containing sounds similar to the noises they describe. Humans usually understand onomatopoeia unconsciously and use it in daily conversation; thus, it is clearly easy for humans to solve. However, it is difficult for computers because the mechanisms to recognize onomatopoeia are not very clear even now [1]. One of the difficulties of CAPTCHA schemes is the lack of reliable accuracy metrics. Some of the existing works deal with successful rate defined as a fraction of correctly answered tests. However, if we modify schemes as more complicated so that condition 2) is satisfied, then it may be hard to be solved by human resulting failure of condition 1) . So, we need to balance the tradeoff of two conditions. To address the above issues of CAPTCHA scheme, we introduce two evaluation metrics, Human Acceptance Rate (HAR) and Machine Acceptance Rate (MAR), measured through comprehensive experiments. To balance both acceptance rates, we try to improve HAR with the five proposed schemes looking for the best scheme that allows humans solve CAPTCHA easily. Similarly, we attempt to reduce MAR as smaller as possible, that is, to make CAPTCHA unbreakable against attackers. Our experiment is evaluated by 63 Japanese and 63 foreigners participating from 16 countries. One of the proposed style of CAPTCHA is based on the Manga comics with onomatopoeia that may be recognized wide range of subject without suffering form language barrier and hence it helps to extend the coverage of users. Our contribution of this work is as follows. • A new CAPTCHA scheme using Onomatopoeia that is able to be synthesized from system. • A comprehensive evaluation of accuracy metrics with respects to both human and machine (HAR and MAR), with five styles of queries and five smart attackers. • An evaluation made be by a broad domain of subjects with distinct background knowledge in the world including 63 Japanese and 63 non-Japanese.