Frederic Schimmelpfennig, André Brinkmann, Hossein Asadi, Reza Salkhordeh
{"title":"跳过在 mmap 的","authors":"Frederic Schimmelpfennig, André Brinkmann, Hossein Asadi, Reza Salkhordeh","doi":"arxiv-2409.10946","DOIUrl":null,"url":null,"abstract":"Memory access efficiency is significantly enhanced by caching recent address\ntranslations in the CPUs' Translation Lookaside Buffers (TLBs). However, since\nthe operating system is not aware of which core is using a particular mapping,\nit flushes TLB entries across all cores where the application runs whenever\naddresses are unmapped, ensuring security and consistency. These TLB flushes,\nknown as TLB shootdowns, are costly and create a performance and scalability\nbottleneck. A key contributor to TLB shootdowns is memory-mapped I/O,\nparticularly during mmap-munmap cycles and page cache evictions. Often, the\nsame physical pages are reassigned to the same process post-eviction,\npresenting an opportunity for the operating system to reduce the frequency of\nTLB shootdowns. We demonstrate, that by slightly extending the mmap function,\nTLB shootdowns for these \"recycled pages\" can be avoided. Therefore we introduce and implement the \"fast page recycling\" (FPR) feature\nwithin the mmap system call. FPR-mmaps maintain security by only triggering TLB\nshootdowns when a page exits its recycling cycle and is allocated to a\ndifferent process. To ensure consistency when FPR-mmap pointers are used, we\nmade minor adjustments to virtual memory management to avoid the ABA problem.\nUnlike previous methods to mitigate shootdown effects, our approach does not\nrequire any hardware modifications and operates transparently within the\nexisting Linux virtual memory framework. Our evaluations across a variety of CPU, memory, and storage setups,\nincluding persistent memory and Optane SSDs, demonstrate that FPR delivers\nnotable performance gains, with improvements of up to 28% in real-world\napplications and 92% in micro-benchmarks. Additionally, we show that TLB\nshootdowns are a significant source of bottlenecks, previously misattributed to\nother components of the Linux kernel.","PeriodicalId":501333,"journal":{"name":"arXiv - CS - Operating Systems","volume":"14 1","pages":""},"PeriodicalIF":0.0000,"publicationDate":"2024-09-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Skip TLB flushes for reused pages within mmap's\",\"authors\":\"Frederic Schimmelpfennig, André Brinkmann, Hossein Asadi, Reza Salkhordeh\",\"doi\":\"arxiv-2409.10946\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"Memory access efficiency is significantly enhanced by caching recent address\\ntranslations in the CPUs' Translation Lookaside Buffers (TLBs). However, since\\nthe operating system is not aware of which core is using a particular mapping,\\nit flushes TLB entries across all cores where the application runs whenever\\naddresses are unmapped, ensuring security and consistency. These TLB flushes,\\nknown as TLB shootdowns, are costly and create a performance and scalability\\nbottleneck. A key contributor to TLB shootdowns is memory-mapped I/O,\\nparticularly during mmap-munmap cycles and page cache evictions. Often, the\\nsame physical pages are reassigned to the same process post-eviction,\\npresenting an opportunity for the operating system to reduce the frequency of\\nTLB shootdowns. We demonstrate, that by slightly extending the mmap function,\\nTLB shootdowns for these \\\"recycled pages\\\" can be avoided. Therefore we introduce and implement the \\\"fast page recycling\\\" (FPR) feature\\nwithin the mmap system call. FPR-mmaps maintain security by only triggering TLB\\nshootdowns when a page exits its recycling cycle and is allocated to a\\ndifferent process. To ensure consistency when FPR-mmap pointers are used, we\\nmade minor adjustments to virtual memory management to avoid the ABA problem.\\nUnlike previous methods to mitigate shootdown effects, our approach does not\\nrequire any hardware modifications and operates transparently within the\\nexisting Linux virtual memory framework. Our evaluations across a variety of CPU, memory, and storage setups,\\nincluding persistent memory and Optane SSDs, demonstrate that FPR delivers\\nnotable performance gains, with improvements of up to 28% in real-world\\napplications and 92% in micro-benchmarks. Additionally, we show that TLB\\nshootdowns are a significant source of bottlenecks, previously misattributed to\\nother components of the Linux kernel.\",\"PeriodicalId\":501333,\"journal\":{\"name\":\"arXiv - CS - Operating Systems\",\"volume\":\"14 1\",\"pages\":\"\"},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2024-09-17\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"arXiv - CS - Operating Systems\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/arxiv-2409.10946\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - CS - Operating Systems","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2409.10946","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
Memory access efficiency is significantly enhanced by caching recent address
translations in the CPUs' Translation Lookaside Buffers (TLBs). However, since
the operating system is not aware of which core is using a particular mapping,
it flushes TLB entries across all cores where the application runs whenever
addresses are unmapped, ensuring security and consistency. These TLB flushes,
known as TLB shootdowns, are costly and create a performance and scalability
bottleneck. A key contributor to TLB shootdowns is memory-mapped I/O,
particularly during mmap-munmap cycles and page cache evictions. Often, the
same physical pages are reassigned to the same process post-eviction,
presenting an opportunity for the operating system to reduce the frequency of
TLB shootdowns. We demonstrate, that by slightly extending the mmap function,
TLB shootdowns for these "recycled pages" can be avoided. Therefore we introduce and implement the "fast page recycling" (FPR) feature
within the mmap system call. FPR-mmaps maintain security by only triggering TLB
shootdowns when a page exits its recycling cycle and is allocated to a
different process. To ensure consistency when FPR-mmap pointers are used, we
made minor adjustments to virtual memory management to avoid the ABA problem.
Unlike previous methods to mitigate shootdown effects, our approach does not
require any hardware modifications and operates transparently within the
existing Linux virtual memory framework. Our evaluations across a variety of CPU, memory, and storage setups,
including persistent memory and Optane SSDs, demonstrate that FPR delivers
notable performance gains, with improvements of up to 28% in real-world
applications and 92% in micro-benchmarks. Additionally, we show that TLB
shootdowns are a significant source of bottlenecks, previously misattributed to
other components of the Linux kernel.