Models for Internal Clustering Validation Indexes Based on Hadoop-MapReduce

Int. J. Distributed Syst. Technol. Pub Date : 2020-07-01 DOI:10.4018/ijdst.2020070103

Soumeya Zerabi, S. Meshoul, Samia Boucherkha

{"title":"Models for Internal Clustering Validation Indexes Based on Hadoop-MapReduce","authors":"Soumeya Zerabi, S. Meshoul, Samia Boucherkha","doi":"10.4018/ijdst.2020070103","DOIUrl":null,"url":null,"abstract":"Clustervalidationaimstobothevaluatetheresultsofclusteringalgorithmsandpredictthenumberof clusters.Itisusuallyachievedusingseveralindexes.Traditionalinternalclusteringvalidationindexes (CVIs)aremainlybasedincomputingpairwisedistanceswhichresultsinaquadraticcomplexity oftherelatedalgorithms.TheexistingCVIscannothandlelargedatasetsproperlyandneedtobe revisitedtotakeaccountoftheever-increasingdatasetvolume.Therefore,designofparalleland distributedsolutionstoimplementtheseindexesisrequired.Tocopewiththisissue,theauthors proposetwoparallelanddistributedmodelsforinternalCVIsnamelyforSilhouetteandDunnindexes usingMapReduceframeworkunderHadoop.TheproposedmodelstermedasMR_Silhouetteand MR_Dunnhavebeentestedtosolveboththeissueofevaluatingtheclusteringresultsandidentifying theoptimalnumberofclusters.Theresultsofexperimentalstudyareverypromisingandshowthat theproposedparallelanddistributedmodelsachievetheexpectedtaskssuccessfully. KeywoRDS Big Data, Clustering, Data Mining, Dunn Index, Hadoop, Internal Clustering Validation Indexes, MapReduce, Optimal Number Of Clusters, Silhouette Index","PeriodicalId":118536,"journal":{"name":"Int. J. Distributed Syst. Technol.","volume":"68 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2020-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"3","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Int. J. Distributed Syst. Technol.","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.4018/ijdst.2020070103","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}

引用次数: 3

Abstract

Clustervalidationaimstobothevaluatetheresultsofclusteringalgorithmsandpredictthenumberof clusters.Itisusuallyachievedusingseveralindexes.Traditionalinternalclusteringvalidationindexes (CVIs)aremainlybasedincomputingpairwisedistanceswhichresultsinaquadraticcomplexity oftherelatedalgorithms.TheexistingCVIscannothandlelargedatasetsproperlyandneedtobe revisitedtotakeaccountoftheever-increasingdatasetvolume.Therefore,designofparalleland distributedsolutionstoimplementtheseindexesisrequired.Tocopewiththisissue,theauthors proposetwoparallelanddistributedmodelsforinternalCVIsnamelyforSilhouetteandDunnindexes usingMapReduceframeworkunderHadoop.TheproposedmodelstermedasMR_Silhouetteand MR_Dunnhavebeentestedtosolveboththeissueofevaluatingtheclusteringresultsandidentifying theoptimalnumberofclusters.Theresultsofexperimentalstudyareverypromisingandshowthat theproposedparallelanddistributedmodelsachievetheexpectedtaskssuccessfully. KeywoRDS Big Data, Clustering, Data Mining, Dunn Index, Hadoop, Internal Clustering Validation Indexes, MapReduce, Optimal Number Of Clusters, Silhouette Index

查看原文本刊更多论文

基于Hadoop-MapReduce的内部聚类验证索引模型

Clustervalidationaimstobothevaluatetheresultsofclusteringalgorithmsandpredictthenumberof clusters.Itisusuallyachievedusingseveralindexes。Traditionalinternalclusteringvalidationindexes (CVIs)aremainlybasedincomputingpairwisedistanceswhichresultsinaquadraticcomplexity oftherelatedalgorithms。TheexistingCVIscannothandlelargedatasetsproperlyandneedtobe revisitedtotakeaccountoftheever-increasingdatasetvolume。Therefore，designofparalleland distributedsolutionstoimplementtheseindexesisrequired。Tocopewiththisissue，theauthors proposetwoparallelanddistributedmodelsforinternalCVIsnamelyforSilhouetteandDunnindexes usingMapReduceframeworkunderHadoop。TheproposedmodelstermedasMR_Silhouetteand MR_Dunnhavebeentestedtosolveboththeissueofevaluatingtheclusteringresultsandidentifying theoptimalnumberofclusters。Theresultsofexperimentalstudyareverypromisingandshowthat theproposedparallelanddistributedmodelsachievetheexpectedtaskssuccessfully。关键词:大数据，聚类，数据挖掘，Dunn索引，Hadoop，内部聚类验证索引，MapReduce，最优聚类数，Silhouette索引

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Int. J. Distributed Syst. Technol.

自引率

0.00%

发文量