Distributed LSI: Parallel preprocessing and vector sharing

Abstract

The technique of latent semantic indexing (LSI) has a wide variety of uses in intelligence and security informatics applications. LSI processing generates high-dimensional vectors that are used to represent individual items of interest and the features of which those items are composed. Historically, LSI representation vectors have been generated in a single computing environment (workstation, server, or VM instance). However, this is not a requirement. This paper describes two approaches to distributing elements of LSI processing. The first, parallelization of the preprocessing stage, can significantly decrease the time required for creation of LSI indexes. The second, vector sharing, can dramatically improve security in distributed LSI environments.