Thomson Reuters' Solution for Triple Ranking in the FEIII 2017 Challenge

In this paper we describe our approach to the triple ranking task of the FEIII 2017 challenge. Our method leveraged different machine learning classifiers in an ensemble as well as Thomson Reuters knowledge bases and information services to bring in external world knowledge of mentioned entities and extract information from the contextual sentences. Internal evaluation of our method was done by computing the Normalized Discounted Cumulative Gain (NDCG) as tracked by the challenge and classification accuracy. The official FEIII Challenge evaluation showed our system performed highly in single ranking of all triples, placing in 2nd or 3rd place out of 17 participants for 4 of 6 scoring variants; the system also performed above average in per role ranking for 4 of 6 average role NDCG scoring variants.