Improving Similar Question Retrieval using a Novel Tripartite Neural Network based Approach

Abstract

Collective intelligence of the crowds is distilled together in various Community Question Answering (CQA) Services such as Quora, Yahoo Answers, Stack Overflow forums, wherein users share their knowledge, providing both informational and experiential support to other users. As users often search for similar information, probabilities are high that for a new incoming question, there is a related question-answer pair existing in the CQA dataset. Therefore, an efficient technique for similar question identification is need of the hour. While data is not a bottleneck in this scenario, addressing the vocabulary diversity generated by a variety pool of users certainly is. This paper proposes a novel tripartite neural network based approach towards the similar question retrieval problem. The network takes inputs in the form of question-answer and new question triplet and learns internal representations from similarities among them. Our approach achieves classification performances upto 77% on a real world CQA dataset.We have also compared our method with two other baselines and found that it performs significantly better in handling the problem of vocabulary diversity and 'zero-lexical overlap' among questions.