Removal of redundancy in documents retrieved from different resources

Abstract

This paper describes a methodology for removing (partially or totally) redundant information received from different documents in an effort to synthesize new documents. In particular, information retrieved from different databases may have various forms, such as images, natural language text, data, etc. These pieces of information may be parts of one or more documents related with a specific subject. This means that a number of text-paragraphs and images may occur (or retrieved) more than once, by creating redundancy in the storage space. Thus, in order to create a new redundant-less document the duplicated parts of information have to be removed. The methodology presented analyzes text-paragraphs and images received from different DBs by using a set of similarity criteria in order to make a decision for the removal of the duplicated ones. Illustrative examples are provided.