Improving activity data collection with on-device personalization using fine-tuning

Abstract

One of the biggest challenges of activity data collection is the unavoidability of relying on users and keep them engaged to provide labels consistently. Recent breakthroughs in mobile platforms have proven effective in bringing deep neural networks powered intelligence into mobile devices. In this study, we propose on-device personalization using fine-tuning convolutional neural networks as a mechanism in optimizing human effort in data labeling. First, we transfer the knowledge gained by on-cloud pre-training based on crowdsourced data to mobile devices. Second, we incrementally fine-tune a personalized model on every individual device using its locally accumulated input. Then, we utilize estimated activities customized according to the on-device model inference as feedback to motivate participants to improve data labeling. We conducted a verification study and gathered activity labels with smartphone sensors. Our preliminary evaluation results indicate that the proposed method outperformed the baseline method by approximately 8% regarding accuracy recognition.