Personal Diary Generation from Wearable Cameras with Concept Augmented Image Captioning and Wide Trail Strategy

Abstract

Writing diary is not only a hobby but also provides a personal lifelog for better analysis and understanding of a user's daily activities and events. However, in a busy society, people may not have enough time to write in diary all their social interaction. This motivates our proposal to develop a ubiquitous system to automatically generate daily text diary using our novel method for image captioning from photos taken periodically from wearable cameras. We propose to incorporate common visual concepts extracted from a photo to enhance the details of the image description. We also propose a wide trail beam search strategy to enhance the naturalness of text caption. Our captioning method improves the results on MSCOCO dataset on four metrics: BLEU, METEOR, ROUGE-L, CIDEr. As compared to the method proposed by Xu et.al and Neuraltalk of Karpathy, our model has better performance on all four metrics. We also develop smart glasses and a prototype smart workplace in which people can have their personal diary generated from photos taken by smart glasses. Furthermore, we also apply a transformer machine translation model in order to translate captions into Vietnamese language. The results are promising and can be used for Vietnamese people.