Combined WSD algorithms with LSA to identify semantic similarity in unstructured textual data

Abstract

Semantically related sentence may not have any word in common. However, identifying the semantic similarity between words at sentence level possess difficult challenges such as polysemy, synonyms, heterogeneity and sparsity of unstructured textual datasets. It is assumed that sentences with similar text or words in common are semantically related. It means that the standard Information Retrieval (IR) measure based on word co-occurrence are not appropriate to tackle the aforementioned challenges of identifying semantics in unstructured text documents. Many semantic similarity measures have been proposed to resolve this non-trivial issues, but many existing studies did not properly utilize the combination of Corpus and Knowledge-based approach to solve the syntactic construct and the roles of Part Of Speech in identifying semantic similarities in sentences. In this research, we aim at proposing a method for measuring sentence semantic similarity identification that combines two algorithms from the knowledge-based Word Sense Disambiguation algorithms with Latent Semantic Analysis to identify the semantic similarity of sentences and to compare results with human evaluation.