通过分离键和值来优化LevelDB

Lei Wang, G. Ding, Yulong Zhao, Dingzeyu Wu, Chengrui He
{"title":"通过分离键和值来优化LevelDB","authors":"Lei Wang, G. Ding, Yulong Zhao, Dingzeyu Wu, Chengrui He","doi":"10.1109/PDCAT.2017.00074","DOIUrl":null,"url":null,"abstract":"LevelDB uses the merger mechanism for data integration during the writing process. During this process, the value will move together with the key, causing a lot of unnecessary data rewriting. This paper presents a structure that the key stored separately from the value. And the value is stored in a separate file (we call it Value File), with value offset in the file and length information stored in LevelDB. Test results show that the optimized LevelDBs sequential write performance is reduced by about 40%. But random write and overwrite performances improve more than 200%. And with the increase of the number of tests records, the improvement becomes more and more obvious. The amount of rewriting data and the number of merging files, depending on the length of every different record, reduce about 80% averagely, which significantly improves the performance of the original program.","PeriodicalId":119197,"journal":{"name":"2017 18th International Conference on Parallel and Distributed Computing, Applications and Technologies (PDCAT)","volume":"10 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2017-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"6","resultStr":"{\"title\":\"Optimization of LevelDB by Separating Key and Value\",\"authors\":\"Lei Wang, G. Ding, Yulong Zhao, Dingzeyu Wu, Chengrui He\",\"doi\":\"10.1109/PDCAT.2017.00074\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"LevelDB uses the merger mechanism for data integration during the writing process. During this process, the value will move together with the key, causing a lot of unnecessary data rewriting. This paper presents a structure that the key stored separately from the value. And the value is stored in a separate file (we call it Value File), with value offset in the file and length information stored in LevelDB. Test results show that the optimized LevelDBs sequential write performance is reduced by about 40%. But random write and overwrite performances improve more than 200%. And with the increase of the number of tests records, the improvement becomes more and more obvious. The amount of rewriting data and the number of merging files, depending on the length of every different record, reduce about 80% averagely, which significantly improves the performance of the original program.\",\"PeriodicalId\":119197,\"journal\":{\"name\":\"2017 18th International Conference on Parallel and Distributed Computing, Applications and Technologies (PDCAT)\",\"volume\":\"10 1\",\"pages\":\"0\"},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2017-12-01\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"6\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"2017 18th International Conference on Parallel and Distributed Computing, Applications and Technologies (PDCAT)\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.1109/PDCAT.2017.00074\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"2017 18th International Conference on Parallel and Distributed Computing, Applications and Technologies (PDCAT)","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/PDCAT.2017.00074","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 6

摘要

LevelDB在写入过程中使用合并机制进行数据集成。在这个过程中,值会随着键一起移动,导致大量不必要的数据重写。本文提出了一种键与值分开存储的结构。值存储在一个单独的文件中(我们称之为值文件),文件中的值偏移量和长度信息存储在LevelDB中。测试结果表明,优化后的leveldb顺序写性能降低了约40%。但是随机写入和覆盖性能提高了200%以上。并且随着检测记录数量的增加,这种改善也越来越明显。根据每条不同记录的长度,重写数据的数量和合并文件的数量平均减少了80%左右,这大大提高了原程序的性能。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
Optimization of LevelDB by Separating Key and Value
LevelDB uses the merger mechanism for data integration during the writing process. During this process, the value will move together with the key, causing a lot of unnecessary data rewriting. This paper presents a structure that the key stored separately from the value. And the value is stored in a separate file (we call it Value File), with value offset in the file and length information stored in LevelDB. Test results show that the optimized LevelDBs sequential write performance is reduced by about 40%. But random write and overwrite performances improve more than 200%. And with the increase of the number of tests records, the improvement becomes more and more obvious. The amount of rewriting data and the number of merging files, depending on the length of every different record, reduce about 80% averagely, which significantly improves the performance of the original program.
求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信