Peer review in online forums: Classifying feedback-sentiment

Abstract

Replies posted in technical online forums often contain feedback to the author of the parent comment in the form of agreement, doubt, gratitude, contradiction, etc. We call this feedback-sentiment. Inference of feedback-sentiment has application in expert finding, fact validation, and answer validation. To study feedback-sentiment, we use nearly 25 million comments from a popular discussion forum (Slash-dot, org), spanning over 10 years. We propose and test a heuristic that feedback-sentiment most commonly appears in the first sentence of a forum reply. We introduce a novel interactive decision tree system that allows us to train a classifier using principles from active learning. We classify individual reply sentences as positive, negative, or neutral, and then test the accuracy of our classifier against labels provided by human annotators (using Amazon's Mechanical Turk). We show how our classifier outperforms three general-purpose sentiment classifiers for the task of finding feedback-sentiment.