Credibility as Signal: Predicting Evaluations of Credibility by a Signal-Based Model

Abstract

In this paper we propose the model of signal for objects that are subject to evaluation by crowdsourcing. Such signal, constructed as probability of distribution using Normal Random Utility Model (NRUM), can be used to measure object's performance, create rankings or predict next evaluations. Our model is designed for monadic scale evaluations where evaluators can have different expertise or bias for using scale. Moreover, our model is constructed for situations where we can have a lot of missing evaluations or varying numbers of evaluations for each object and from each evaluator, typical for crowdsourcing data. We have built a model for medical Web pages credibility from real crowdsourcing data and have evaluated the model's predictive ability, proving its superiority to alternative prediction methods.