Problem with Cross-Cultural Comparison of User-Generated Ratings on Mechanical Turk

Abstract

Many online services serve diverse populations spanning many countries and cultures. Some of these services rely on user-generated ratings to curate and filter information, or to inform other users. However, little is known about how various cultural biases and cross-cultural differences affect such ratings. We studied how Indian and American workers on Mechanical Turk differ in their response styles by asking them to rate three products. We also explored several dimensions of cultural differences including social orientation (individualism vs. collectivism), social desirability, and thinking style (holistic vs. analytic). We found that Indian workers tended to use higher ratings on all items, including both product ratings and the different survey instruments. We discussed the implications for collecting ratings from culturally diverse populations, and for cross-cultural studies on Mechanical Turk.