Awareness support in collaborative programming tools: An evaluation based on programmer's perception and eye tracking

Groupware technology is an essential asset for organizations and a successful medium to support social meetings and teacher-learning processes when people are geographically distributed. Computer programming is a domain that can take advantage of the Collaborative Work paradigm and of this technology. However, to be truly effective, groupware systems must provide suitable awareness support, i.e., be capable of informing users of the activity taking place and of the team outcome, so that they can build a setting for their work. The study of awareness mechanisms and their influence on collaborative programming processes is an open research question. In this context, this article contributes an experimental study to evaluate the mechanisms to support coordination, communication, and awareness of COLLECE, a distributed group programming system rich in awareness support. This study brings as a novel approach the combination of subjective and objective information sources, specifically eye tracking techniques, and incorporates a heuristic evaluation of awareness support based on a well-known framework. Thanks to this experiment, we have been able to verify that this blended approach, in which users participate intensively, provides a more comprehensive and deeper assessment of awareness and other coordination and communication mechanisms, and thus a better understanding of how their use can influence the collaborative programming process and outcome (program or piece of software). The study has revealed strengths, weaknesses and opportunities for improvement of the evaluated system, and has yielded concrete results as to which are the most useful awareness components of the user interface not related to the activity supported by the system (programming), these being the traffic lights; the easiest to use, such as the session panel; or the components that impose a greater cognitive load, such as the multi-scrollbar. Regarding the relationship between the supported awareness dimensions and the quality of the collaboration outcome (computer program), the results point to the need to adequately support the What-Artifact dimension, i.e., highlighting in which part of the shared workspace (the source code) the user with floor control is working.