Applying Formal Logic Validation to Enhance Natural Language Understanding

Inconsistencies and ambiguities of annotation can cause vagueness in the results obtained by natural language understanding (NLU). The quality of the type systems used for annotation affects the quality of annotation. To achieve highly accepted sets of annotated documents, the Fleiss' kappa score has been widely used to observe the level of agreement from annotated results, submitted by different human annotators. The challenge is that the kappa score cannot be used to validate the type systems nor to identify any incorrect annotations. Thus, we proposed an application of formal logic for validating type systems and annotations against expert rules. Experiments have been done by using four different type systems and annotation sets created by an expert and three novices. Our proposed formal logic model was used to validate the novice type systems and annotations against the expert rules. The results show that the technique could help identifying inconsistencies between expert and novice annotations, by using a model checker. The number of detected inconsistencies impacts the level of achieved F1 score. Thus, the proposed formal logic technique could be used to guide novice annotators to develop accepted type systems. This will help to enhance the performance of the generated machine learning models used by the NLU.