Crowdsourced mobile data collection: lessons learned from a new study methodology

In this paper we explore a scalable data collection methodology that simultaneously achieves low cost and a high degree of control. We use popular online crowdsourcing platforms to recruit 63 subjects for a 90-day data collection that resulted in over 75K hours of data. The total cost of data collection was dramatically lower than for alternative methodologies, with total subject compensation under $3.5K US, and a total of less than 10 hours/week spent by researchers managing the study. At the same time, our methodology enhances control and enables richer study protocols by allowing direct contact with subjects. We were able to conduct surveys, exchange messages, and debug remotely with feedback from subjects. In addition to reporting on study details, we also discuss interesting findings and offer lessons learned.