CNN-LSTM based Social Media Post Caption Generator

Abstract

In artificial intelligence automatically captioning an image is a challenging problem that brings computer vision and natural language processing together, but the problem becomes more challenging when artificial intelligence has to caption an Instagram post. This is challenging because the caption of an Instagram post is not a simple description of the image but it contains some abstract features like jokes, sarcasm, references, etc. There has been a lot of work on image captioning but there is hard work on captioning social media posts. So, I propose a deep learning model that uses an amalgamation of Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) Recurrent Neural Network (RNN). The first major task was to get a dataset but there is no such popular dataset available. The option to scrap Instagram was explored but turns out scraping Instagram is prohibited, luckily, I was able to get a dataset available on Kaggle containing 35,000 records but due to computation limitations, I used 5000 images and captions for training. Eventually, the trained model was not very accurate due to fewer numbers of epochs i.e 20, small training dataset and majorly because of the low-quality dataset. Even though the model did not show better results but the methodology to generate captions for Instagram posts was established.