{"title":"动态ETL增量负载分析与数据集成数据仓库","authors":"Zulkifli Arsyad","doi":"10.32627/internal.v4i2.260","DOIUrl":null,"url":null,"abstract":"Data integration is a combination of techniques and businesses that are used to collect data from different sources into useful and valuable information ETL process that includes extracting data from various data sources, transforming data to form and calculate data and load data on target storage, to support data warehouse need. Based on organizations and industries that have implemented data warehouse, the problem that generally arises regarding data load is the difficulty in integrating different data sources, how to form data from various data formats into uniform data, how to integrate data delta between data sources and target storage in an incremental load process so that this data synchronization process can be carried out continuously and relatively faster. ETL process requires a platform that can facilitate data integration needs, in order to run this process. SSIS (SQL Server Integration Service) is a Data Integration platform to build an enterprise-level data integration and solutions for data transformation. Integration Service can extract and change data (transform) from various sources such as XML data files, flat files, APIs, and relational data sources, and then load into one or several destination data. According to the problem related to data load, we will examine how the solution model uses SSIS for the ETL process. This paper proposed an ETL Architecture model by completing the ETL process for full & incremental load extraction and the original data layer.","PeriodicalId":421147,"journal":{"name":"INTERNAL (Information System Journal)","volume":"35 2 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2021-12-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"1","resultStr":"{\"title\":\"Analisis Dynamic ETL Incremental Load untuk Data Integration Datawarehouse\",\"authors\":\"Zulkifli Arsyad\",\"doi\":\"10.32627/internal.v4i2.260\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"Data integration is a combination of techniques and businesses that are used to collect data from different sources into useful and valuable information ETL process that includes extracting data from various data sources, transforming data to form and calculate data and load data on target storage, to support data warehouse need. Based on organizations and industries that have implemented data warehouse, the problem that generally arises regarding data load is the difficulty in integrating different data sources, how to form data from various data formats into uniform data, how to integrate data delta between data sources and target storage in an incremental load process so that this data synchronization process can be carried out continuously and relatively faster. ETL process requires a platform that can facilitate data integration needs, in order to run this process. SSIS (SQL Server Integration Service) is a Data Integration platform to build an enterprise-level data integration and solutions for data transformation. Integration Service can extract and change data (transform) from various sources such as XML data files, flat files, APIs, and relational data sources, and then load into one or several destination data. According to the problem related to data load, we will examine how the solution model uses SSIS for the ETL process. This paper proposed an ETL Architecture model by completing the ETL process for full & incremental load extraction and the original data layer.\",\"PeriodicalId\":421147,\"journal\":{\"name\":\"INTERNAL (Information System Journal)\",\"volume\":\"35 2 1\",\"pages\":\"0\"},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2021-12-30\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"1\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"INTERNAL (Information System Journal)\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.32627/internal.v4i2.260\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"INTERNAL (Information System Journal)","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.32627/internal.v4i2.260","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 1
摘要
数据集成是技术和业务的结合,用于将来自不同数据源的数据收集到有用和有价值的信息ETL过程,包括从各种数据源提取数据,将数据转换为形成和计算数据,并将数据加载到目标存储中,以支持数据仓库需求。基于已经实施数据仓库的组织和行业,在数据负载方面普遍存在的问题是难以集成不同的数据源,如何将不同数据格式的数据形成统一的数据,如何在增量加载过程中集成数据源与目标存储之间的数据增量,从而使这个数据同步过程能够持续且相对较快地进行。ETL流程需要一个能够方便数据集成需求的平台,才能运行此流程。SSIS (SQL Server Integration Service)是一个数据集成平台,用于构建企业级的数据集成和数据转换解决方案。Integration Service可以从各种来源(如XML数据文件、平面文件、api和关系数据源)提取和更改数据(转换),然后加载到一个或多个目标数据中。根据与数据负载相关的问题,我们将研究解决方案模型如何将SSIS用于ETL流程。本文通过完成ETL全负荷、增量负荷提取和原始数据层的ETL流程,提出了ETL架构模型。
Analisis Dynamic ETL Incremental Load untuk Data Integration Datawarehouse
Data integration is a combination of techniques and businesses that are used to collect data from different sources into useful and valuable information ETL process that includes extracting data from various data sources, transforming data to form and calculate data and load data on target storage, to support data warehouse need. Based on organizations and industries that have implemented data warehouse, the problem that generally arises regarding data load is the difficulty in integrating different data sources, how to form data from various data formats into uniform data, how to integrate data delta between data sources and target storage in an incremental load process so that this data synchronization process can be carried out continuously and relatively faster. ETL process requires a platform that can facilitate data integration needs, in order to run this process. SSIS (SQL Server Integration Service) is a Data Integration platform to build an enterprise-level data integration and solutions for data transformation. Integration Service can extract and change data (transform) from various sources such as XML data files, flat files, APIs, and relational data sources, and then load into one or several destination data. According to the problem related to data load, we will examine how the solution model uses SSIS for the ETL process. This paper proposed an ETL Architecture model by completing the ETL process for full & incremental load extraction and the original data layer.