Limits of speech in connected homes: Experimental comparison of self-reporting tools for human activity recognition

Data annotation for human activity recognition is a well-known challenge for researchers. In particular, annotation in daily life settings relies on self-reporting tools with unknown accuracy. Speech is a promising interface for activity labeling. In this work, we compare the accuracy of two commercially available tools for annotation: voice diaries and connected buttons. We retrofit the water meters of thirty homes in the USA for infrastructure-mediated sensing. Participants are split into equal groups and receive one of the self-reporting tools. The balanced accuracy metric is transferred from the field of machine learning to the evaluation of the annotation performance. Our results show that connected buttons perform significantly better than the voice diary, with 92% median accuracy and 65% median reporting rate. Using questionnaire answers, we highlight that annotation performance is impacted by habit formation and sentiments toward the annotation tool. The use case for data annotation is to disaggregate water meter data into human activities beyond the point of use. We show that it is feasible with a machine-learning model and the corrected annotations. Finally, we formulate recommendations for the design of studies and intelligent environments around the key ideas of proportionality and immediacy.