Detecting Text Similarity on a Scalable No-SQL Database Platform

Abstract

The paper looks at the platform scalability problem for near-to-similar document detection tasks. The application areas for the proposed approach include plagiarism detection and text filtering in data leak prevention systems. The paper reviews limitations of the current solutions based on the relational DBMS and suggests data structure suitable for implementation in no-SQL databases on the highly scalable clustered platforms. The proposed data structure is based on "key-value" model and it does not depend on the shingling method used to encode the text. The proposed model was implemented on the clustered MongoDB platform and tested with the large dataset on the platform that was scaled up horizontally during the experiment. The experiments indicated the applicability of the proposed approach to near-to-similar document detection.