The Square-Root Agreement Rule for Incentivizing Objective Feedback in Online Platforms

A major challenge in obtaining evaluations of products or services on e-commerce platforms is that of eliciting informative feedback in the absence of verifiability. We propose a simple incentive mechanism for obtaining objective feedback on such platforms. In this mechanism, an agent gets a reward only if her answer for an evaluation matches that of her peer, where this reward is inversely proportional to a popularity index of each answer. This index is defined to be the square-root of the empirical frequency at which any two agents performing the same evaluation agree on the particular answer. Rarely agreed-upon answers thus earn a higher reward than answers for which agreements are relatively more common. We call this mechanism the Square-Root Agreement Rule (SRA). A key feature of platforms that SRA leverages is the existence of a large number of similar entities to be evaluated (e.g., restaurants, sellers, services, etc.); in this regime, we show that truthful behavior is a strict Bayes-Nash equilibrium of the game induced by SRA. Further, as the number of evaluation tasks across the platform grows, this equilibrium is asymptotically optimal for the agents across all symmetric equilibria. Moreover, under a mild condition, we show that any symmetric equilibrium that gives a higher expected payoff to the agents than the truthful equilibrium must be close to being fully informative when the number of evaluations is large. SRA can thus be an effective approach for administering reward-based incentive schemes (e.g., rebates, reputation score, etc.) on these platforms.