Guideline for Novel Fine-Grained Sentiment Annotation and Data Curation: A Case Study

Abstract

Driven by the rise of the internet, recent years have witnessed the gradual manifestation of commercial values of online reviews. In movie industry, sentiment analysis serves as the foundation for mining user preferences among diverse and multi-layered audiences, providing insight into the market value of movies. As a representative task, aspect-based sentiment analysis (ABSA) aims to analyse and extract fine-grained sentiment elements and their relations in terms of discussed aspects. Relevant studies, particularly in the realm of deep learning research, face challenges due to insufficient annotated data. To alleviate this problem, we propose a guideline for fine-grained sentiment annotations that defines aspect categories, describes the method for annotating aspect sentiment triplets, either simple or complex and designs a scheme to represent hierarchical labels. Based on this, an ABSA dataset tailored for the movie domain is curated by annotating on 1100 Chinese short reviews acquired from Douban. Applicability of both the annotation guideline and curated data is evaluated through inter-annotator consistency and self-consistency checks, and domain adaptation assessment of e-commerce and healthcare cases. Predictive performance of machine learning models on this dataset shed light on possible applications in more fine-grained sentiment analysis in the movie domain, for example, figuring out the aspects from which to stimulate viewership and influence public opinions, thereby providing substantial support for the movie's box office performance. Finally, we extended our fine-grained sentiment annotation guideline to the e-commerce and healthcare. Through empirical experimentation, we demonstrated the universality of these guideline across diverse domains.