Sophie Pavia, Nick Piraino, Kazi Islam, A. Pyayt, M. Gubanov
{"title":"大型结构化数据集的混合元数据分类","authors":"Sophie Pavia, Nick Piraino, Kazi Islam, A. Pyayt, M. Gubanov","doi":"10.26421/jdi3.4-4","DOIUrl":null,"url":null,"abstract":"Metadata location and classification is an important problem for large-scale structured datasets. For example, Web tables \\cite{wt_corpus} have hundreds of millions of tables, but often have missing or incorrect labels for rows (or columns) with attribute names. Such errors \\cite{wtitles} significantly complicate all data management tasks such as {\\em query processing, data integration, indexing}, etc. Different sources or authors position metadata rows/columns differently inside a table, which makes its reliable identification challenging.In this work we describe our scalable, hybrid two-layer Deep- and Machine-learning based ensemble, combining Long Short Term Memory (LSTM) and Naive Bayes Classifier to accurately identify Metadata-containing rows or columns in a table. We have performed an extensive evaluation on several datasets, including an ultra large-scale dataset containing more than 15 million tables coming from more than 26 thousands of sources to justify scalability and resistance to variety, stemming from a large number of sources. We observed superiority of this two-layer ensemble, compared to the recent previous approaches and report an impressive 95.73\\text{\\%} accuracy at scale with our ensemble model using regular LSTM.","PeriodicalId":232625,"journal":{"name":"J. Data Intell.","volume":"100 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2022-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"1","resultStr":"{\"title\":\"Hybrid Metadata Classification in Large-scale Structured Datasets\",\"authors\":\"Sophie Pavia, Nick Piraino, Kazi Islam, A. Pyayt, M. Gubanov\",\"doi\":\"10.26421/jdi3.4-4\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"Metadata location and classification is an important problem for large-scale structured datasets. For example, Web tables \\\\cite{wt_corpus} have hundreds of millions of tables, but often have missing or incorrect labels for rows (or columns) with attribute names. Such errors \\\\cite{wtitles} significantly complicate all data management tasks such as {\\\\em query processing, data integration, indexing}, etc. Different sources or authors position metadata rows/columns differently inside a table, which makes its reliable identification challenging.In this work we describe our scalable, hybrid two-layer Deep- and Machine-learning based ensemble, combining Long Short Term Memory (LSTM) and Naive Bayes Classifier to accurately identify Metadata-containing rows or columns in a table. We have performed an extensive evaluation on several datasets, including an ultra large-scale dataset containing more than 15 million tables coming from more than 26 thousands of sources to justify scalability and resistance to variety, stemming from a large number of sources. We observed superiority of this two-layer ensemble, compared to the recent previous approaches and report an impressive 95.73\\\\text{\\\\%} accuracy at scale with our ensemble model using regular LSTM.\",\"PeriodicalId\":232625,\"journal\":{\"name\":\"J. Data Intell.\",\"volume\":\"100 1\",\"pages\":\"0\"},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2022-11-01\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"1\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"J. Data Intell.\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.26421/jdi3.4-4\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"J. Data Intell.","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.26421/jdi3.4-4","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
Hybrid Metadata Classification in Large-scale Structured Datasets
Metadata location and classification is an important problem for large-scale structured datasets. For example, Web tables \cite{wt_corpus} have hundreds of millions of tables, but often have missing or incorrect labels for rows (or columns) with attribute names. Such errors \cite{wtitles} significantly complicate all data management tasks such as {\em query processing, data integration, indexing}, etc. Different sources or authors position metadata rows/columns differently inside a table, which makes its reliable identification challenging.In this work we describe our scalable, hybrid two-layer Deep- and Machine-learning based ensemble, combining Long Short Term Memory (LSTM) and Naive Bayes Classifier to accurately identify Metadata-containing rows or columns in a table. We have performed an extensive evaluation on several datasets, including an ultra large-scale dataset containing more than 15 million tables coming from more than 26 thousands of sources to justify scalability and resistance to variety, stemming from a large number of sources. We observed superiority of this two-layer ensemble, compared to the recent previous approaches and report an impressive 95.73\text{\%} accuracy at scale with our ensemble model using regular LSTM.