Zhijie Fan, Bo Yang, Jing Peng, Bingsen Pei, Changsong Zheng, Xin Li
{"title":"用于大规模统一日志数据收集的 VSS 中的动态自适应机制设计与实现","authors":"Zhijie Fan, Bo Yang, Jing Peng, Bingsen Pei, Changsong Zheng, Xin Li","doi":"10.4018/ijisp.349569","DOIUrl":null,"url":null,"abstract":"This paper studies the collection of large-scale log data of information system and puts forward a dynamic adaptive mechanism for large-scale unified log data collection. Furthermore, we design and implement our method for pan-government industry safety operation management platform. The data flow processing architecture based on message queue is adopted to realize the decoupling of log collection, log processing and log reporting. The traffic peak clipping technology of message queue is adopted to ensure the safety and reliability of log transmission. According to the characteristics of log traffic, a design mode supporting dynamic adjustment of consumption group is proposed to meet the high-performance requirements of the system. The whole system can meet the centralized analysis, security threat perception and intelligent analysis of various security data. Meanwhile, we analyzed and compared with the traditional open-source log collection technology, our proposed method and system has clear advantages.","PeriodicalId":44332,"journal":{"name":"International Journal of Information Security and Privacy","volume":null,"pages":null},"PeriodicalIF":0.5000,"publicationDate":"2024-08-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Dynamic Adaptive Mechanism Design and Implementation in VSS for Large-Scale Unified Log Data Collection\",\"authors\":\"Zhijie Fan, Bo Yang, Jing Peng, Bingsen Pei, Changsong Zheng, Xin Li\",\"doi\":\"10.4018/ijisp.349569\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"This paper studies the collection of large-scale log data of information system and puts forward a dynamic adaptive mechanism for large-scale unified log data collection. Furthermore, we design and implement our method for pan-government industry safety operation management platform. The data flow processing architecture based on message queue is adopted to realize the decoupling of log collection, log processing and log reporting. The traffic peak clipping technology of message queue is adopted to ensure the safety and reliability of log transmission. According to the characteristics of log traffic, a design mode supporting dynamic adjustment of consumption group is proposed to meet the high-performance requirements of the system. The whole system can meet the centralized analysis, security threat perception and intelligent analysis of various security data. Meanwhile, we analyzed and compared with the traditional open-source log collection technology, our proposed method and system has clear advantages.\",\"PeriodicalId\":44332,\"journal\":{\"name\":\"International Journal of Information Security and Privacy\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":0.5000,\"publicationDate\":\"2024-08-09\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"International Journal of Information Security and Privacy\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.4018/ijisp.349569\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q4\",\"JCRName\":\"COMPUTER SCIENCE, SOFTWARE ENGINEERING\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"International Journal of Information Security and Privacy","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.4018/ijisp.349569","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q4","JCRName":"COMPUTER SCIENCE, SOFTWARE ENGINEERING","Score":null,"Total":0}
Dynamic Adaptive Mechanism Design and Implementation in VSS for Large-Scale Unified Log Data Collection
This paper studies the collection of large-scale log data of information system and puts forward a dynamic adaptive mechanism for large-scale unified log data collection. Furthermore, we design and implement our method for pan-government industry safety operation management platform. The data flow processing architecture based on message queue is adopted to realize the decoupling of log collection, log processing and log reporting. The traffic peak clipping technology of message queue is adopted to ensure the safety and reliability of log transmission. According to the characteristics of log traffic, a design mode supporting dynamic adjustment of consumption group is proposed to meet the high-performance requirements of the system. The whole system can meet the centralized analysis, security threat perception and intelligent analysis of various security data. Meanwhile, we analyzed and compared with the traditional open-source log collection technology, our proposed method and system has clear advantages.
期刊介绍:
As information technology and the Internet become more and more ubiquitous and pervasive in our daily lives, there is an essential need for a more thorough understanding of information security and privacy issues and concerns. The International Journal of Information Security and Privacy (IJISP) creates and fosters a forum where research in the theory and practice of information security and privacy is advanced. IJISP publishes high quality papers dealing with a wide range of issues, ranging from technical, legal, regulatory, organizational, managerial, cultural, ethical and human aspects of information security and privacy, through a balanced mix of theoretical and empirical research articles, case studies, book reviews, tutorials, and editorials. This journal encourages submission of manuscripts that present research frameworks, methods, methodologies, theory development and validation, case studies, simulation results and analysis, technological architectures, infrastructure issues in design, and implementation and maintenance of secure and privacy preserving initiatives.