Assessment of captions’ quality by students of a MOOC on accessibility of digital materials

Captions constitute one of the fundamental components of audiovisual accessibility. Ensuring that captions comply with applicable accessibility standards is an important and complex task that requires dedication and specific knowledge. It goes beyond checking whether or not the video has captions. This work aims to evaluate whether students can adequately assess the quality of captions for the deaf or hard of hearing. It analyses data from an activity of a MOOC on accessibility of digital materials, in which they assessed the quality of captions. It was a voluntary task within the course delivered through an Open edX platform. At the time of enrollment, the students were randomly divided into two groups (AB and BA) so that the cohort AB watched the well-captioned video first and then the poorly-captioned video (with identified errors only in relation to 11 requirements of the UNE 153010:2012 standard), and the cohort BA watched them the other way round. After watching each of the videos, 87 students responded to a survey of 18 items with a 5-point Likert scale. The experimental design was a randomized, triple-blind, crossover AB/BA study. The data were analyzed using a multilevel multiple ordinal regression model. The main conclusions are: (1) basic training on accessibility of multimedia learning materials enables students to perform a preliminary assessment of the accessibility of captions; (2) qualitative guidelines present fewer problems than quantitative ones when identifying accessibility problems; (3) novice evaluators tend to underscore some of the items in non-accessible captions, compared to accessible captions.