UIT-OpenViIC: An open-domain benchmark for evaluating image captioning in Vietnamese

Abstract

Image captioning is one of the vision-language tasks that continues to attract interest from the research community worldwide in the 2020s. The MS-COCO Caption benchmark is commonly used to evaluate the performance of advanced captioning models, even though it was introduced in 2015. However, recent captioning models trained on the MS-COCO Caption dataset perform well only in English language patterns; they do not perform as effectively in describing contexts specific to Vietnam or in generating fluent Vietnamese captions. To contribute to the low-resources research community as in Vietnam, we introduce a novel image captioning dataset in Vietnamese, the Open-domain Vietnamese Image Captioning dataset (UIT-OpenViIC). The introduced dataset includes complex scenes captured in Vietnam and manually annotated by Vietnamese under strict rules and supervision. In this paper, we present in more detail the dataset creation process. From preliminary analysis, we show that our dataset is challenging to recent state-of-the-art (SOTA) Transformer-based baselines, which performed well on the MS COCO dataset. Then, the modest results prove that UIT-OpenViIC has room to grow, which can be one of the standard benchmarks in Vietnamese for the research community to evaluate their captioning models. Furthermore, we present a CAMO approach that effectively enhances the image representation ability by a multi-level encoder output fusion mechanism, which helps improve the quality of generated captions compared to previous captioning models. In our experiments, we show that our dataset is more diverse and challenging than the MS-COCO caption dataset, as indicated by the significantly lower CIDEr scores on our testing set, ranging from 59.52 to 62.47 compared to MS-COCO. For the CAMO approach, experiments on UIT-OpenViIC show that when equipped with a captioning baseline model, it can improve performance by 0.8970 to 4.9167 CIDEr.