{"title":"HAEM: Obtaining Higher-Quality Classification Task Results with AI Workers","authors":"Yuki Yamashita, Hiroyoshi Ito, Kei Wakabayashi, Masaki Kobayashi, Atsuyuki Morishima","doi":"10.1145/3501247.3531580","DOIUrl":"https://doi.org/10.1145/3501247.3531580","url":null,"abstract":"Obtaining high-quality results for a fixed set of classification tasks with a limited budget is a critical issue in crowdsourcing. The introduction of AI models to complement the process should be explored. However, there are few existing approaches to directly address the problem, which have been proposed in the context of how to train AI models using noisy crowdsourced data. This paper presents a more direct approach for solving the problem of introducing AI to improve the task results of human workers for a fixed number of tasks with a limited budget; we deal with an AI model as a worker and aggregates the results of both human and AI workers in a symmetric manner. The proposed “Human-AI EM” (HAEM) algorithm, which extends the Dawid Skene model, treats the AI models as workers and explicitly computes their confusion matrices to derive higher-quality aggregation results. We conducted an extensive set of experiments and compared HAEM with two other methods (MBEM and Dawid Skene model). We found that AI-powered HAEM showed better performance than the two methods in most cases and that the HAEM often performed better than the Dawid Skene model with additional human workers. We also found that AI workers work well when they are good at identifying particular classes even if they do not have very good overall accuracy.","PeriodicalId":136381,"journal":{"name":"Proceedings of the 14th ACM Web Science Conference 2022","volume":"30 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-06-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123856741","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Enhancing Creativity as Innovation via Asynchronous Crowdwork","authors":"Pradeep K. Murukannaiah","doi":"10.1145/3501247.3531555","DOIUrl":"https://doi.org/10.1145/3501247.3531555","url":null,"abstract":"Synchronous, face-to-face interactions such as brainstorming are considered essential for creative tasks (the old normal). However, face-to-face interactions are difficult to arrange because of the diverse locations and conflicting availability of people—a challenge made more prominent by work-from-home practices during the COVID-19 pandemic (the new normal). In addition, face-to-face interactions are susceptible to cognitive interference. We employ crowdsourcing as an avenue to investigate creativity in asynchronous, online interactions. We choose product ideation, a natural task for the crowd since it requires human insight and creativity into what product features would be novel and useful. We compare the performance of solo crowd workers with asynchronous teams of crowd workers formed without prior coordination. Our findings suggest that, first, crowd teamwork yields fewer but more creative ideas than solo crowdwork. The enhanced team creativity results when Second, cognitive interference, known to inhibit creativity in face-to-face teams, may not be significant in crowd teams. Third, teamwork promotes better achievement emotions for crowd workers. These findings provide a basis for trading off creativity, quantity, and worker happiness in setting up crowdsourcing workflows for product ideation.","PeriodicalId":136381,"journal":{"name":"Proceedings of the 14th ACM Web Science Conference 2022","volume":"44 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-06-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127167442","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Radin Hamidi Rad, E. Bagheri, M. Kargar, D. Srivastava, Jaroslaw Szlichta
{"title":"Subgraph Representation Learning for Team Mining","authors":"Radin Hamidi Rad, E. Bagheri, M. Kargar, D. Srivastava, Jaroslaw Szlichta","doi":"10.1145/3501247.3531578","DOIUrl":"https://doi.org/10.1145/3501247.3531578","url":null,"abstract":"Team mining is concerned with the identification of a group of experts that are able to collaborate with each other in order to collectively cover a set of required skills. This problem has mainly been addressed either through graph search, which looks for subgraphs that satisfy the skill requirements or through neural architectures that learn a mapping from the skill space to the expert space. An exact graph-based solution to this problem is intractable and its heuristic variants are only able to identify sub-optimal solutions. On the other hand, neural architecture-based solutions are prone to overfitting and simplistically reduce the problem of team formation to one of expert ranking. Our work in this paper proposes an unsupervised heterogeneous skip-gram-based subgraph mining approach that can learn representations for subgraphs in a collaboration network. Unlike previous work, the subgraph representations allow our method to mine teams that have past collaborative history and collectively cover the requested desirable skills. Through our experiments, we demonstrate that our proposed approach is able to outperform a host of state-of-the-art team mining techniques from both quantitative and qualitative perspectives.","PeriodicalId":136381,"journal":{"name":"Proceedings of the 14th ACM Web Science Conference 2022","volume":"16 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-06-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123082249","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Topic modeling approaches to counter online grooming","authors":"Nikolaos Lykousas, C. Patsakis","doi":"10.1145/3501247.3539507","DOIUrl":"https://doi.org/10.1145/3501247.3539507","url":null,"abstract":"This paper describes a topic model-based approach for analyzing a large-scale dataset of chat messages exchanged between the users of LiveMe, a major social live streaming platform, in the context of broadcasts involving sexually explicit content. The analysis reveals the characteristics of predatory behavior targeting minors and its criminal dimension in an accessible and explainable manner.","PeriodicalId":136381,"journal":{"name":"Proceedings of the 14th ACM Web Science Conference 2022","volume":"24 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-06-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132244076","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Evaluating Digital Polarization in Multi-Party Systems: Evidence from the German Bundestag","authors":"Amber Chin, C. C. Vieira, Jisu Kim","doi":"10.1145/3501247.3531547","DOIUrl":"https://doi.org/10.1145/3501247.3531547","url":null,"abstract":"Social networking sites like Twitter and Facebook have become dominant sources of political activity with many politicians choosing to leverage these platforms. This rise in popularity has led many researchers to investigate the impact of these platforms on digital communication, specifically with regard to digital polarization. Most of the current polarization literature is centered on two-party systems with little attention given to multi-party parliamentary networks. As such, we leverage data from members of the German Bundestag (MdBs) to better understand digital polarization in MdB Twitter interactions. This paper expands the current literature to include a multi-level network evaluation of a parliamentary body by evaluating partisan polarization on the scale of retweets, mentions, and following-follower relationships. We employ social network analysis and text-based sentiment analysis to understand both the structure and sentiment polarity of the networks and find that polarization varies based on the level of the network that is being examined. Results indicate that the highest polarization occurs in the retweets network and the lowest occurs in the mentions network, while the sentiment analysis suggests same-party mentions are significantly more positive than cross-party mentions.","PeriodicalId":136381,"journal":{"name":"Proceedings of the 14th ACM Web Science Conference 2022","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-06-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130025239","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Sara De Candia, G. D. F. Morales, Corrado Monti, F. Bonchi
{"title":"Social Norms on Reddit: A Demographic Analysis","authors":"Sara De Candia, G. D. F. Morales, Corrado Monti, F. Bonchi","doi":"10.1145/3501247.3531549","DOIUrl":"https://doi.org/10.1145/3501247.3531549","url":null,"abstract":"Social norms, the shared informal rules of acceptable behavior, drive and reflect the evolution of societies. As an increasingly large part of social interactions happens online, social media data offers an unprecedented opportunity to assess the perception of social norm boundaries in-the-wild. In this regard, Reddit’s r/AITA represents an invaluable source of codified social norms. This subreddit is an online forum where individuals describe how they acted in a specific situation in the past, and ask for the feedback of the community about whether their behavior was deviant or socially acceptable. Other users in the community share their views and express a judgment codified by a tag. This study focuses on assessing which factors are associated with judgements expressed by the community. Specifically, we investigate two main factors: the demographics of the author of the submission and the topic of the submission. Our analysis shows a clear gender imbalance in the judgements, with submissions by male authors receiving negative judgements with a 62% higher likelihood. Older authors (≥ 28) also have a higher chance of receiving negative judgements (+21%). Regarding topics, submissions about romantic relationships and work tend to be judged more positively (+69% and +70%, respectively), thus hinting towards a role of the community as a support group, especially for female participants. We then focus on controversial submissions which garner heterogeneous judgements. We find that these submissions are clearly separable from those ones that are unanimously judged, and that male and older (≥ 28) authors are more likely to describe controversial situations that split the community (+26% and +22%, respectively). Finally, we focus on the characteristics of the evaluators. We find that their judgements are associated with the other communities they belong to (signifying other interests and experiences), with an effect size comparable to the demographic group of the author. By combining all these variables—demographics of the author and communities of the evaluator—we are able to build a classifier that predicts a deviance judgement on a submission with AUC = 0.85.","PeriodicalId":136381,"journal":{"name":"Proceedings of the 14th ACM Web Science Conference 2022","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-06-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"117090344","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"There is a fine Line between Personalization and Surveillance: Semantic User Interest Tracing via Entity-level Analytics","authors":"Amit Kumar, M. Spaniol","doi":"10.1145/3501247.3531592","DOIUrl":"https://doi.org/10.1145/3501247.3531592","url":null,"abstract":"User interest tracing is a common practice in many Web use-cases including, but not limited to, search, recommendation or intelligent assistants. The overall aim is to provide the user a personalized “Web experience” by aggregating and exploiting a plenitude of user data derived from collected logs, accessed contents, and/or mined community context. As such, fairly basic features such as terms and graph structures can be utilized in order to model a user’s interest. While there are clearly positive aspects in the before mentioned application scenarios, the user’s privacy is highly at risk. In order to highlight inherent privacy risks, this paper studies Semantic User Interest Tracing (SUIT in short) by investigating a user’s publishing/editing behavior of Web contents. In contrast to existing approaches, SUIT solely exploits the (semantic) concepts [categories] inherent in documents derived via entity-level analytics. By doing so, we raise Web contents to the entity-level. Thus, we are able to abstract the user interest from plain text strings to “things”. In particular, we utilize the inherited structural relationships present among the concepts derived from a knowledge graph in order to identify the user associated with a specific Web content. Our extensive experiments on Wikipedia show that our approach outperforms state of the art approaches in tracing and predicting user behavior in a single language. In addition, we also demonstrate the viability of our semantic (language-agnostic) approach in multi-lingual experiments. As such, SUIT is capable of revealing the user’s identity, which demonstrates the fine line between personalization and surveillance, raising questions regarding ethical considerations at the same time.","PeriodicalId":136381,"journal":{"name":"Proceedings of the 14th ACM Web Science Conference 2022","volume":"4 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-06-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126842894","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"The Effect of Hiding Dislikes on the Use of YouTube's Like and Dislike Features","authors":"Soon-Gyo Jung, Joni O. Salminen, B. Jansen","doi":"10.1145/3501247.3531546","DOIUrl":"https://doi.org/10.1145/3501247.3531546","url":null,"abstract":"Using data from a major international news organization, we investigate the effect of hiding the count of dislikes from YouTube viewers on the propensity to use the video like/dislike features. We compare one entire month of videos before (n = 478) and after (n = 394) YouTube began hiding the dislikes counts. Collectively, these videos had received 450,200 likes and 41,892 dislikes. To account for content variability, we analyze the likes/dislikes by sentiment class (positive, neutral, negative). Results of chi-square testing show that while both likes and dislikes decreased after the hiding, dislikes decreased substantially more. We repeat the analysis with four other YouTube news channels in various languages (Arabic, English, French, Spanish) and one non-news organization, with similar results in all but one case. Findings from these multiple organizations suggest that YouTube hiding the number of dislikes from viewers has altered the user-platform interactions for the like/dislike features. Therefore, comparing the like/dislike metrics before and after the removal would give invalid insights into users’ reactions to content on YouTube.","PeriodicalId":136381,"journal":{"name":"Proceedings of the 14th ACM Web Science Conference 2022","volume":"19 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-06-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125637391","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Measuring Web Cookies in Governmental Websites","authors":"Matthias Götze, Costas Iordanou, Georgios Smaragdakis","doi":"10.1145/3501247.3531545","DOIUrl":"https://doi.org/10.1145/3501247.3531545","url":null,"abstract":"In recent years, governments worldwide have moved their services online to better serve their citizens. Benefits aside, this choice increases the danger of tracking via such sites. This is of great concern as governmental websites increasingly become the only interaction point with the government. In this paper, we investigate popular governmental websites across different countries and assess to what extent the visits to these sites are tracked by third-parties. Our results show that, unfortunately, tracking is a serious concern, as in some countries up to 90% of these websites create cookies of third-party trackers without any consent from users. Non-session cookies, that are created by trackers and can last for days or months, are widely present even in countries with strict user privacy laws. We also show that the above is a problem for official websites of international organizations and popular websites that inform the public about the COVID-19 pandemic.","PeriodicalId":136381,"journal":{"name":"Proceedings of the 14th ACM Web Science Conference 2022","volume":"92 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-06-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132176569","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
A. Klasnja, Negar Arabzadeh, Mahbod Mehrvarz, E. Bagheri
{"title":"On the Characteristics of Ranking-based Gender Bias Measures","authors":"A. Klasnja, Negar Arabzadeh, Mahbod Mehrvarz, E. Bagheri","doi":"10.1145/3501247.3531540","DOIUrl":"https://doi.org/10.1145/3501247.3531540","url":null,"abstract":"With increased recent awareness on the possible impact of retrieval techniques on intensifying gender biases, researchers have embarked on defining quantifiable gender bias metrics that can provide the means to concretely measure such biases in practice. While successful in allowing for identifying possible sources of gender bias, there has been little work that systematically explores the characteristics of these metrics. This paper argues that effective future works on gender biases in information retrieval require a careful understanding of the bias metrics in terms of their consistency, robustness, sensitivity and also their relation with psychological characteristics and what they actually measure. Through our experiments, we show that more rigorous work on gender bias metrics need to be pursued as existing metrics may not necessarily be consistent and robust and often capture differing psychological characteristics.","PeriodicalId":136381,"journal":{"name":"Proceedings of the 14th ACM Web Science Conference 2022","volume":"109 ","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-06-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134161508","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}