An Enhanced Deep Learning Model for Duplicate Question Detection on Quora Question pairs using Siamese LSTM

Abstract

The question answering platform Quora has millions of users which increases the probability of questions asked with similar intent. One question may be structured in two different ways by two users, and answering similar questions repeatedly impacts user experience. Manual filtration of such questions is a tedious task, so Quora attempts to detect and remove these duplicate questions by using the Random Forest Model, which is not completely effective. As Quora contains question answers in the form of text data, different Natural Language Processing techniques are used to transform the text data into numerical vectors. In this research, the log loss metric acts as the primary metric to evaluate different models. The primary contribution is that the Siamese network is used to process two questions parallelly and find vectors representation of each question. The vectors computed by this method enables similarity detection which is more effective than existing models. GloVe word embedding is used to understand the semantic similarity between two questions. The random classifier is built as the base model and logistic regression, linear SVM and XGBoost model are used to reduce the log loss. Finally, a Siamese LSTM is proposed which reduces the loss dramatically.