A crowdsourced system for user studies in information extraction

Abstract

In this paper, from an entity linking (EL) system, we take a set of tweets, where some subsequence of words is annotated with possible meaning/entities and these entities are linked with several Wikipedia pages. We propose a model using crowdsourcing to disambiguate and decide about the accurate Wikipedia page that must be linked with a definite word/spot. We discuss about importance of crowdsourcing and compare different crowdsourcing systems and at the end, introduce crowdflower. We discuss about the crowdflower features in particular. Finally, we analyse output reports of the crowdflower and present a novel approach to select the reliable results. In summary, our observations show that reliable results have a confidence rate over 0.5.