Tools for Collecting, Synchronizing, and Annotating Ecologically Valid Social Behavior In-the-Wild

So many Multimedia Systems are designed for the online world. We may be led to believe that the real world we live in now already exists online. When I talk about the online world, I am referring to the countless images and videos, which are commonly scraped from the internet for the generation of many benchmark datasets. This 'finders keepers' attitude towards online found data has hailed a revolution in the development of multimodal computing systems but could be leading towards a dead end in the development of socially intelligent systems. The reality is that data containing social behavior is private; this growing realization has led to the withdrawal of major benchmarks. To develop truly socially intelligent systems, we desperately need ethically sourced ecologically valid data. I argue that we need to return to the drawing broad and reconsider our data gathering procedures to make meaningful progress. In this talk, I offer a different perspective on the first and perhaps fundamental step in developing socially intelligent multimodal systems. Using the ConfLab dataset and data collection concept as a case study, I will discuss the challenges of collecting such data. I will also present solutions for wireless multi-sensor synchronization, sensing (https://github.com/TUDelft-SPC-Lab/spcl_midge_hardware), and continuous annotation.