Research and implementation of an efficient incremental synchronization method based on Timestamp

Abstract

With the application of information management system, enterprises build many independent databases to meet different business needs. In order to mine information from multi-source data, data needs to be extracted to a specified data warehouse. This can be accomplished through ETL(extract-transformation-loading), Among them, data extraction is a key stage in ETL. After comparing the advantages and disadvantages of various incremental data extraction mechanisms, this paper proposes a time-stamp-based incremental synchronization scheme, which can not only improve the extraction efficiency, but also solve the shortcoming that the time-stamp extraction method cannot capture deleted data.