The development of a Kinect-based online socio-meter for users with social and communication skill impairments: A computational sensing approach

Some people have impairments in social skills due to autism, phobia, or physical disabilities such as blindness. The proposed web-based system is intended to measure social interaction and communication among people situated around the target user based on their gestures and behaviors. Such multimodal portraits of behavioral data will be captured using the Kinect sensor (version 2) and provide rich sources to predict their social relationship, which will then be delivered to the target user in both qualitative and quantitative units, either in mobile or desktop applications. Currently, our system is capable of capturing and analyzing the raw data, sending the processed one to Microsoft Azure for data visualization and further behavioral analysis. The visualized data can then be ported to the therapists, doctors, parents, or caregivers for further actions. Unlike previous works which aim to accurately classify the dyadic social interactions between an adult-child pair, and thus characterizing the physiological synchrony between the pair, our study thereby adds richness to the understanding of computational sensing for the diagnosis, assessment and intervention of children with autism.