Helping students keep up with real-time captions by pausing and highlighting

We explore methods for improving the readability of real- time captions by allowing users to more easily switch their gaze between multiple visual information sources. Real-time captioning provides deaf and hard of hearing (DHH) users with access to spoken content during live events, and the web has allowed these services to be provided via remotely- located captioning services, and for web content itself. However, despite caption benefits, spoken language reading rates often result in DHH users falling behind spoken content, especially when the audio is paired with visual references. This is particularly true in classroom settings, where multi-modal content is the norm, and captions are often poorly positioned in the room, relative to speakers. Additionally, this accommodation can benefit other students who face temporary or "situational" disabilities such as listening to unfamiliar speech accents, or if a student is in a location with poor acoustics. In this paper, we explore pausing and highlighting as a means of helping DHH students keep up with live classroom content by helping them track their place when reading text involving visual references. Our experiments show that by providing users with a tool to more easily track their place in a transcript while viewing live video, it is possible for them to follow visual content that might otherwise have been missed. Both pausing and highlighting have a positive impact on students' scores on comprehension tests, but highlighting is preferred to pausing, and yields nearly twice as large of an improvement. We then discuss several issues with captioning that we observed during our design process and user study, and then suggest future work that builds on these insights.