An Application to Provide Translated Subtitles and Pictures for Youth English Learners using Speech-to-Text and Nlp Techniques

Abstract

Currently, thousands of free K-12 educational videos exist online with the aim of trying to help young students learn outside of the typical scholastic environment. However, most of these videos are in English, so without subtitles it may be difficult for non-native English-speaking students to fully understand them. These students may need to spend time searching for translations and understanding content, which can distract them from grasping the important concepts within the videos. The state-ofthe- art of speech-to-text and NLP techniques might help this group digest the content of instructional videos more effectively. This paper proposes an application that uses speech-to-text, machine translation, and NLP techniques to generate translated subtitles and visual learning aids for viewers of instructional videos. This video application supports more than 20 languages. We applied our application to some popular online educational videos and conducted a qualitative evaluation of its approach and effectiveness. The results demonstrated that the application could successfully translate the English of the videos into the viewers’ native language(s), detect keywords, and display relevant images to further facilitate contextual understanding.