A New Approach for Missing Data Imputation in Big Data Interface

IF 2 4区计算机科学 Q3 AUTOMATION & CONTROL SYSTEMS

Information Technology and Control Pub Date : 2020-12-19 DOI:10.5755/j01.itc.49.4.27386

Chunzhi Wang, N. Shakhovska, A. Sachenko, M. Komar

{"title":"A New Approach for Missing Data Imputation in Big Data Interface","authors":"Chunzhi Wang, N. Shakhovska, A. Sachenko, M. Komar","doi":"10.5755/j01.itc.49.4.27386","DOIUrl":null,"url":null,"abstract":"The three-stage approach for missing data imputation in Big data interface is proposed in the paper. The first stage includes designing the Big data model in the task of missing data recovery, which enables to process the structured and semistructured data. The next stage is developing the method of missing data recovery based on functional dependencies and association rules. The estimating the algorithm complexity for missing data recovery is provided at the last stage. The proposed method of missing data recovery creates additional data values using a based domain and functional dependencies and adds these values in available training data. The performing the analysis of data different types is possible too. The correctness of the imputed values is verified on the classifier built on the original dataset. The proposed method performs in 12% better than the EM and RF methods for 30% missing data and enables the parallel execution in distributed databases. The acceleration for m=41 attributes is larger in 12.5 times for 20 servers (processors) comparing with the non-parallel modeю","PeriodicalId":54982,"journal":{"name":"Information Technology and Control","volume":"129 1","pages":"541-555"},"PeriodicalIF":2.0000,"publicationDate":"2020-12-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"10","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Information Technology and Control","FirstCategoryId":"94","ListUrlMain":"https://doi.org/10.5755/j01.itc.49.4.27386","RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"AUTOMATION & CONTROL SYSTEMS","Score":null,"Total":0}

引用次数: 10

Abstract

The three-stage approach for missing data imputation in Big data interface is proposed in the paper. The first stage includes designing the Big data model in the task of missing data recovery, which enables to process the structured and semistructured data. The next stage is developing the method of missing data recovery based on functional dependencies and association rules. The estimating the algorithm complexity for missing data recovery is provided at the last stage. The proposed method of missing data recovery creates additional data values using a based domain and functional dependencies and adds these values in available training data. The performing the analysis of data different types is possible too. The correctness of the imputed values is verified on the classifier built on the original dataset. The proposed method performs in 12% better than the EM and RF methods for 30% missing data and enables the parallel execution in distributed databases. The acceleration for m=41 attributes is larger in 12.5 times for 20 servers (processors) comparing with the non-parallel modeю

查看原文本刊更多论文

大数据接口中缺失数据的一种新方法

提出了大数据接口中缺失数据补全的三阶段方法。第一阶段为缺失数据恢复任务中的大数据模型设计，实现对结构化和半结构化数据的处理。下一步是开发基于功能依赖和关联规则的缺失数据恢复方法。最后给出了缺失数据恢复算法复杂度的估计。所提出的缺失数据恢复方法使用基于域和功能依赖关系创建额外的数据值，并将这些值添加到可用的训练数据中。对不同类型的数据进行分析也是可能的。在原始数据集上建立的分类器上验证了输入值的正确性。对于30%的缺失数据，该方法的性能比EM和RF方法提高了12%，并且可以在分布式数据库中并行执行。对于20台服务器(处理器)，m=41属性的加速是非并行模型的12.5倍

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Information Technology and Control 工程技术-计算机：人工智能

CiteScore

2.70

自引率

9.10%

发文量

审稿时长

12 months

期刊介绍： Periodical journal covers a wide field of computer science and control systems related problems including: -Software and hardware engineering; -Management systems engineering; -Information systems and databases; -Embedded systems; -Physical systems modelling and application; -Computer networks and cloud computing; -Data visualization; -Human-computer interface; -Computer graphics, visual analytics, and multimedia systems.