五是最亮的星。但有多亮？测试在线评论中星级评定的等距性

IF 8.9 2区管理学 Q1 MANAGEMENT

Organizational Research Methods Pub Date : 2024-01-08 DOI:10.1177/10944281231223412

Balázs Kovács

{"title":"五是最亮的星。但有多亮？测试在线评论中星级评定的等距性","authors":"Balázs Kovács","doi":"10.1177/10944281231223412","DOIUrl":null,"url":null,"abstract":"Organizational research increasingly relies on online review data to gauge perceived valuation and reputation of organizations and products. Online review platforms typically collect ordinal ratings (e.g., 1 to 5 stars); however, researchers often treat them as a cardinal data, calculating aggregate statistics such as the average, the median, or the variance of ratings. In calculating these statistics, ratings are implicitly assumed to be equidistant. We test whether star ratings are equidistant using reviews from two large-scale online review platforms: Amazon.com and Yelp.com. We develop a deep learning framework to analyze the text of the reviews in order to assess their overall valuation. We find that 4 and 5-star ratings, as well as 1 and 2-star ratings, are closer to each other than 3-star ratings are to 2 and 4-star ratings. An additional online experiment corroborates this pattern. Using simulations, we show that the distortion by non-equidistant ratings is especially harmful in cases when organizations receive only a few reviews and when researchers are interested in estimating variance effects. We discuss potential solutions to solve the issue with rating non-equidistance.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"22 12","pages":""},"PeriodicalIF":8.9000,"publicationDate":"2024-01-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Five Is the Brightest Star. But by how Much? Testing the Equidistance of Star Ratings in Online Reviews\",\"authors\":\"Balázs Kovács\",\"doi\":\"10.1177/10944281231223412\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"Organizational research increasingly relies on online review data to gauge perceived valuation and reputation of organizations and products. Online review platforms typically collect ordinal ratings (e.g., 1 to 5 stars); however, researchers often treat them as a cardinal data, calculating aggregate statistics such as the average, the median, or the variance of ratings. In calculating these statistics, ratings are implicitly assumed to be equidistant. We test whether star ratings are equidistant using reviews from two large-scale online review platforms: Amazon.com and Yelp.com. We develop a deep learning framework to analyze the text of the reviews in order to assess their overall valuation. We find that 4 and 5-star ratings, as well as 1 and 2-star ratings, are closer to each other than 3-star ratings are to 2 and 4-star ratings. An additional online experiment corroborates this pattern. Using simulations, we show that the distortion by non-equidistant ratings is especially harmful in cases when organizations receive only a few reviews and when researchers are interested in estimating variance effects. We discuss potential solutions to solve the issue with rating non-equidistance.\",\"PeriodicalId\":19689,\"journal\":{\"name\":\"Organizational Research Methods\",\"volume\":\"22 12\",\"pages\":\"\"},\"PeriodicalIF\":8.9000,\"publicationDate\":\"2024-01-08\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Organizational Research Methods\",\"FirstCategoryId\":\"91\",\"ListUrlMain\":\"https://doi.org/10.1177/10944281231223412\",\"RegionNum\":2,\"RegionCategory\":\"管理学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"MANAGEMENT\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Organizational Research Methods","FirstCategoryId":"91","ListUrlMain":"https://doi.org/10.1177/10944281231223412","RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"MANAGEMENT","Score":null,"Total":0}

引用次数: 0

摘要

组织研究越来越依赖于在线评论数据来衡量组织和产品的认知价值和声誉。在线评论平台通常收集的是序数评分（如 1 到 5 星）；然而，研究人员通常将其视为心数数据，计算平均值、中位数或评分方差等综合统计数据。在计算这些统计数据时，评级被隐含地假定为等距的。我们使用两个大型在线评论平台的评论来检验星级评分是否等距：亚马逊和 Yelp.com。我们开发了一个深度学习框架来分析评论文本，以评估其整体价值。我们发现，4 星和 5 星评价以及 1 星和 2 星评价之间的距离比 3 星和 2 星以及 4 星评价之间的距离更近。另外一项在线实验也证实了这一模式。通过模拟实验，我们发现当组织只收到几条评论时，当研究人员对估计方差效应感兴趣时，非等距评分的失真尤其有害。我们讨论了解决非等距评分问题的潜在方案。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

查看原文本刊更多论文

Five Is the Brightest Star. But by how Much? Testing the Equidistance of Star Ratings in Online Reviews

Organizational research increasingly relies on online review data to gauge perceived valuation and reputation of organizations and products. Online review platforms typically collect ordinal ratings (e.g., 1 to 5 stars); however, researchers often treat them as a cardinal data, calculating aggregate statistics such as the average, the median, or the variance of ratings. In calculating these statistics, ratings are implicitly assumed to be equidistant. We test whether star ratings are equidistant using reviews from two large-scale online review platforms: Amazon.com and Yelp.com. We develop a deep learning framework to analyze the text of the reviews in order to assess their overall valuation. We find that 4 and 5-star ratings, as well as 1 and 2-star ratings, are closer to each other than 3-star ratings are to 2 and 4-star ratings. An additional online experiment corroborates this pattern. Using simulations, we show that the distortion by non-equidistant ratings is especially harmful in cases when organizations receive only a few reviews and when researchers are interested in estimating variance effects. We discuss potential solutions to solve the issue with rating non-equidistance.

求助全文

通过发布文献求助，成功后即可免费获取论文全文。去求助

来源期刊

Organizational Research Methods Multiple-

CiteScore

23.20

自引率

3.20%

发文量

期刊介绍： Organizational Research Methods (ORM) was founded with the aim of introducing pertinent methodological advancements to researchers in organizational sciences. The objective of ORM is to promote the application of current and emerging methodologies to advance both theory and research practices. Articles are expected to be comprehensible to readers with a background consistent with the methodological and statistical training provided in contemporary organizational sciences doctoral programs. The text should be presented in a manner that facilitates accessibility. For instance, highly technical content should be placed in appendices, and authors are encouraged to include example data and computer code when relevant. Additionally, authors should explicitly outline how their contribution has the potential to advance organizational theory and research practice.