Managing Sensor Data Streams: Lessons Learned from the WeBike Project

Abstract

We present insights on data management resulting from a field deployment of approximately 30 sensor-equipped electric bicycles (e-bikes) at the University of Waterloo. The trial has been in operation for the last two-and-a-half years, and we have collected and analyzed more than 150 gigabytes of data. We discuss best practices for the entire data management process, spanning data collection, extract-transform-load, data cleaning, and choosing a suitable data management ecosystem. We also comment on how our experiences will inform the design of a future large-scale field trial involving several thousand fully-instrumented e-bikes.