Analysis of images to identify specific user actions

Abstract

The study of any writing process, such as translation or programming, typically involves collecting data about user actions including keystrokes and mouse clicks. A tool designed for this type of data collection is ResearchLogger and, in addition to keystrokes and clicks, it collects screenshots at predefined intervals and images around the mouse pointer. The main roadblock for researchers using this tool is analyzing the images it generates, which are very diverse and usually need to be processed manually. In particular, one action that is very hard to recognize without using the images is text selection; this action is very important since it might indicate copying, replacing or formatting of a text passage. In this paper, we present two solutions that support researchers in the review of such images and we test them using data from a previous pilot study. Results show that they can be useful to detect images with the specific user action of text selection.