Assessment of an annotation method for the detection of Spanish argumentative, non-argumentative, and their components

Abstract

There are many annotation methods for the English language based on adapting an argumentation model according to the study domain. However, as far as research has been done, there are no annotation methods for detecting argumentative content in Spanish, not only due to the complexity of identifying the evidence but also because of the lack of data available for this task. The research aims to present and evaluate an annotation method consisting of an adapted argumentation model, an annotation guide, and an annotation process based on Twitter data analysis. The Inter Annotator Agreement (IAA) study achieves 0.63 Fleiss Kappa for Argument/Non-Argument tagging, 0.35 Fleiss Kappa for Argument Component tagging, and 0.53 Fleiss Kappa for Non-Argument Component tagging, while the best Cohen's kappa (k) index achieved, was 0.73, 0.52 and 0.75 respectively. The results' assessment highlights the need to include linguistic segmentation rules for the second annotation task. It is crucial to use discourse markers for the claim and evidence detection. For the first annotation task, it determined that if the prevalence index and the bias index are very low, the prevalence index predominates over the bias index because k increases (0.52<=k<=0.72); likewise, for the third annotation task, when the observed agreement index is almost perfect (0.92) the value of k increases (k=0.75) despite a high prevalence index and a low bias index. The annotated corpus with a Fleiss Kappa >= 0.60, agreement and disagreement tables, and confusion matrices code are available on Mendeley Data Repository.