比较分析和复杂的使用方法的行和列数据库

T. Paromova, I. Solohubov, I. Zeleneva, T. Holub
{"title":"比较分析和复杂的使用方法的行和列数据库","authors":"T. Paromova, I. Solohubov, I. Zeleneva, T. Holub","doi":"10.31474/1996-1588-2021-2-33-40-49","DOIUrl":null,"url":null,"abstract":"\"At present, the volume of processed information increases extremely every year and can already reach hundreds of terabytes or several petabytes. This amount of data is always seen in the field of data analysis, modeling, testing, artificial intelligence, etc. Thus, the problem of saving and improving the performance of data processing systems becomes relevant. To solve this problem, many options for the internal organization of the database and DBMS were considered in this field. The main disadvantage of relational databases with row organization when processing large arrays is the irrational use of file system resources and RAM. One of the options for increasing the efficiency of processing large amounts of information is a columnar data organization model. This model proposes storing data in the form of several files corresponding to the data of each column, which in turn are stored in the form of a key-value. This data organization allows you to optimize the amount of information read from the database, as well as use compression, which has a positive effect on system performance. In this paper, the features of the columnar databases organization are experimentally studied, differences from the traditional row organization are considered, the main advantages and disadvantages of both organization options, their architectural features, which provide accelerated data processing, are analyzed. In the paper, a comparative analysis of the speed of information processing was carried out for various options for organizing a database using the example of a MySQL row database and a ClickHouse column database when executing queries of various types and complexity. Based on the experimental studies result, a system architecture with the integrated use of row and column databases was proposed to achieve universality and optimal performance in transactional systems such as OLTP, taking into account the growth in the volume of processed information. The advantages of the proposed complex database management system with different types of data organization is the achievement of a certain level of versatility and increased performance in transactional systems. The disadvantage of such a system may be its volume of data and the complexity of the organization, as well as problems with ensuring reliability. Consideration of reliability problems is a promising area of research. Theoretically, on the basis of the proposed complex system, it would be possible to create a separate type of database management system. It’s necessary to develop a certain external control level that organizes the operation of a complex of two different type databases, then to design a common interface and connect both databases in a modular way to test various combinations. This approach is quite possible, since some database management systems, such as ClickHouse, have several interfaces for interacting with others, such as MySQL, PostgreSQL.\"","PeriodicalId":104072,"journal":{"name":"Scientific papers of Donetsk National Technical University. Series: Informatics, Cybernetics and Computer Science","volume":"22 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"1900-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"COMPARATIVE ANALYSIS AND COMPLEX USING METHOD OF ROW AND COLUMN DATABASES\",\"authors\":\"T. Paromova, I. Solohubov, I. Zeleneva, T. Holub\",\"doi\":\"10.31474/1996-1588-2021-2-33-40-49\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"\\\"At present, the volume of processed information increases extremely every year and can already reach hundreds of terabytes or several petabytes. This amount of data is always seen in the field of data analysis, modeling, testing, artificial intelligence, etc. Thus, the problem of saving and improving the performance of data processing systems becomes relevant. To solve this problem, many options for the internal organization of the database and DBMS were considered in this field. The main disadvantage of relational databases with row organization when processing large arrays is the irrational use of file system resources and RAM. One of the options for increasing the efficiency of processing large amounts of information is a columnar data organization model. This model proposes storing data in the form of several files corresponding to the data of each column, which in turn are stored in the form of a key-value. This data organization allows you to optimize the amount of information read from the database, as well as use compression, which has a positive effect on system performance. In this paper, the features of the columnar databases organization are experimentally studied, differences from the traditional row organization are considered, the main advantages and disadvantages of both organization options, their architectural features, which provide accelerated data processing, are analyzed. In the paper, a comparative analysis of the speed of information processing was carried out for various options for organizing a database using the example of a MySQL row database and a ClickHouse column database when executing queries of various types and complexity. Based on the experimental studies result, a system architecture with the integrated use of row and column databases was proposed to achieve universality and optimal performance in transactional systems such as OLTP, taking into account the growth in the volume of processed information. The advantages of the proposed complex database management system with different types of data organization is the achievement of a certain level of versatility and increased performance in transactional systems. The disadvantage of such a system may be its volume of data and the complexity of the organization, as well as problems with ensuring reliability. Consideration of reliability problems is a promising area of research. Theoretically, on the basis of the proposed complex system, it would be possible to create a separate type of database management system. It’s necessary to develop a certain external control level that organizes the operation of a complex of two different type databases, then to design a common interface and connect both databases in a modular way to test various combinations. This approach is quite possible, since some database management systems, such as ClickHouse, have several interfaces for interacting with others, such as MySQL, PostgreSQL.\\\"\",\"PeriodicalId\":104072,\"journal\":{\"name\":\"Scientific papers of Donetsk National Technical University. Series: Informatics, Cybernetics and Computer Science\",\"volume\":\"22 1\",\"pages\":\"0\"},\"PeriodicalIF\":0.0000,\"publicationDate\":\"1900-01-01\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Scientific papers of Donetsk National Technical University. Series: Informatics, Cybernetics and Computer Science\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.31474/1996-1588-2021-2-33-40-49\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Scientific papers of Donetsk National Technical University. Series: Informatics, Cybernetics and Computer Science","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.31474/1996-1588-2021-2-33-40-49","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

摘要

“目前,处理的信息量每年都在急剧增加,已经可以达到数百tb或几pb。这种数据量在数据分析、建模、测试、人工智能等领域总是可以看到。因此,保存和提高数据处理系统性能的问题变得相关。为了解决这个问题,在这个领域中考虑了数据库和DBMS的内部组织的许多选项。在处理大型数组时,具有行组织的关系数据库的主要缺点是不合理地使用文件系统资源和RAM。提高处理大量信息的效率的选项之一是柱状数据组织模型。该模型建议以几个文件的形式存储数据,这些文件对应于每列的数据,而这些数据又以键值的形式存储。这种数据组织允许您优化从数据库读取的信息量,以及使用压缩,这对系统性能有积极的影响。本文通过实验研究了列式数据库组织的特点,考虑了与传统行组织的区别,分析了两种组织方式的主要优缺点,以及它们提供加速数据处理的体系结构特点。本文以MySQL行数据库和ClickHouse列数据库为例,在执行不同类型和复杂程度的查询时,对不同的数据库组织方式进行了信息处理速度的比较分析。在实验研究结果的基础上,考虑到处理信息量的增长,提出了一种综合使用行和列数据库的系统架构,以实现OLTP等事务系统的通用性和最优性能。所提出的具有不同类型数据组织的复杂数据库管理系统的优点是在事务系统中实现了一定程度的通用性并提高了性能。这种系统的缺点可能是它的数据量和组织的复杂性,以及在确保可靠性方面的问题。考虑可靠性问题是一个很有前途的研究领域。理论上,在所建议的复杂系统的基础上,可以创建一种单独类型的数据库管理系统。需要开发一定的外部控制层来组织两个不同类型数据库的综合体的操作,然后设计一个通用的接口,以模块化的方式将两个数据库连接起来,以测试各种组合。这种方法是非常可能的,因为一些数据库管理系统,如ClickHouse,有几个接口与其他数据库管理系统,如MySQL, PostgreSQL进行交互。”
本文章由计算机程序翻译,如有差异,请以英文原文为准。
COMPARATIVE ANALYSIS AND COMPLEX USING METHOD OF ROW AND COLUMN DATABASES
"At present, the volume of processed information increases extremely every year and can already reach hundreds of terabytes or several petabytes. This amount of data is always seen in the field of data analysis, modeling, testing, artificial intelligence, etc. Thus, the problem of saving and improving the performance of data processing systems becomes relevant. To solve this problem, many options for the internal organization of the database and DBMS were considered in this field. The main disadvantage of relational databases with row organization when processing large arrays is the irrational use of file system resources and RAM. One of the options for increasing the efficiency of processing large amounts of information is a columnar data organization model. This model proposes storing data in the form of several files corresponding to the data of each column, which in turn are stored in the form of a key-value. This data organization allows you to optimize the amount of information read from the database, as well as use compression, which has a positive effect on system performance. In this paper, the features of the columnar databases organization are experimentally studied, differences from the traditional row organization are considered, the main advantages and disadvantages of both organization options, their architectural features, which provide accelerated data processing, are analyzed. In the paper, a comparative analysis of the speed of information processing was carried out for various options for organizing a database using the example of a MySQL row database and a ClickHouse column database when executing queries of various types and complexity. Based on the experimental studies result, a system architecture with the integrated use of row and column databases was proposed to achieve universality and optimal performance in transactional systems such as OLTP, taking into account the growth in the volume of processed information. The advantages of the proposed complex database management system with different types of data organization is the achievement of a certain level of versatility and increased performance in transactional systems. The disadvantage of such a system may be its volume of data and the complexity of the organization, as well as problems with ensuring reliability. Consideration of reliability problems is a promising area of research. Theoretically, on the basis of the proposed complex system, it would be possible to create a separate type of database management system. It’s necessary to develop a certain external control level that organizes the operation of a complex of two different type databases, then to design a common interface and connect both databases in a modular way to test various combinations. This approach is quite possible, since some database management systems, such as ClickHouse, have several interfaces for interacting with others, such as MySQL, PostgreSQL."
求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信