Improving the Quality of Image Captioning using CNN and LSTM Method

Abstract

In image captioning improving the content of the image by describing the meaning of the picture is a challenge as it should not only be understandable to the user but also described in a short and clear sentence. The proposed solution uses CNN and LSTM as captioning models in an Encoder and the Decoder methodology is used to translate an image into a sentence. The Xception architecture is modified by adding a depth wise convolution layer. A custom activation function is created based on swish and mish. CNN is used for feature extraction, RNN is used for sequence prediction, and LSTM for framing the words into a sentence. The proposed work is validated on two-dimensional image datasets such as dog category data extracted from the flicker8k dataset and real-time images captured through a webcam. The training/testing shows improved loss value, caption prediction time, and an increase in the quality of caption in terms of the BLEU@I parameter.