MultiCoCoA: Multimodal data collector from collocated collaborative activities

Abstract

Collaborative work requires developing and applying soft skills that can influence the formation of social, emotional, and professional skills. Nevertheless, assessing the effectiveness of teamwork, collaboration, and communication is challenging and is commonly addressed by qualitative research approaches. Although through isolated initiatives, Multimodal Learning Analytics (MMLA) techniques have successfully addressed the challenge of measuring different communication features. This work presents MultiCoCoA, a multimodal analytics framework to facilitate data collection in collaborative activities. MultiCoCoA integrates state-of-the-art MMLA techniques and machine learning techniques to analyze audio and video data, and it can help identify areas to improve communication skills. MultiCoCoA allows data to be uploaded and analyzed intuitively, presenting the results through data visualization features and downloadable CSV files for its use with data analysis tools. To evaluate MultiCoCoA’s performance, we conducted both technical validation and user feedback analysis. In terms of accuracy, the system was tested using over 5700 manually labeled video frames from two sessions of collaborative software planning, achieving 92.85% precision in detecting spoken interventions, 85.59% in direction-of-arrival estimation, and 74.88% for identifying observer–observed gaze pairs. To assess usability, we applied the System Usability Scale with five professionals in software development roles, obtaining a favorable usability perception and highlighting ease of use and functional integration, alongside contextual suggestions for deployment in dynamic work environments. The expected outcome of MultiCoCoA is to support research in communication and collaboration, providing quantitative insights to complement existing research methods.