Contrasting results and effectiveness of controlled experiments with crowdsourced data in the evaluation of auditory reaction times

Abstract

We developed an application for the Android platform to test reaction times to auditory stimuli on mobile devices. During tests, users were asked to respond as quickly as possible to auditory events provided through headphone playback, and to also identify the directions of those events based on stereo panning. This paper presents a comparative evaluation of data (i.e. response times in seconds and error rates) collected through controlled supervised experiments as opposed to a crowdsourcing based solution. It is demonstrated that in some respects, crowdsourced data and laboratory data show similar results within their own category, but that at the same time a statistical comparison between the two measurement configurations is difficult due to the significant amount of noisy (outlier) data in the crowdsourced case.