Evolution of file and memory management

Mahadev Satyanarayanan (Satya) presented his thoughts on "The Evolution of Memory and File Systems". He observed that over a 60-year period, there have been four drivers of progress: the quests for scale, performance, transparency, and robustness. At the dawn of computing, the quest for scale was dominant. Easing the memory limitations of early computers was crucial to the growth of computing and the creation of new applications, because memory was so scarce and so expensive. That quest has been phenomenally successful. On a cost per bit basis, volatile and persistent memory technologies have improved by nearly 13 orders of magnitude. The quest for performance has been dominated by the growing gap between processor performance and memory performance. This gap has been most apparent since the use of DRAM technology by the early 1980s, but it was already a serious issue 20 years before that in the era of core memory. Over time, memory hierarchies of increasing depth have improved average case performance by exploiting temporal and spatial locality. These have been crucial in overcoming the processor-memory performance gap, with clever prefetching and write-back techniques also playing important roles. For the first decade or so, the price of improving scale and performance was the need to rewrite software as computers were replaced by new ones. By the early 1960s, this cost was becoming significant. Over time, as people costs have increased relative to hardware costs, disruptive software changes have become unacceptable. This has led to the quest for transparency. In its System/360, IBM pioneered the concept of an invariant architecture with multiple implementations at different price/performance points. The principle of transparent management of data across levels of a memory hierarchy, which we broadly term "caching", was pioneered at the software level by the Atlas computer in the early 1960s. At the hardware level, it was demonstrated first in the IBM System 360 Model 85 in 1968. Since then, caching has been applied at virtually every system level and is today perhaps the most ubiquitous and powerful systems technique for achieving scale, performance and transparency. By the late 1960s, as computers began to used in mission-critical contexts, the negative impact of hardware and software failures escalated. This led to he emergence of techniques to improve robustness even at the possible cost of performance or storage efficiency. The concept of separate address spaces emerged partly because it isolated the consequences of buggy software. Improved resilience to buggy sofware has also been one of the reasons that memory and file systems have remained distinct, even though systems based on the single-level storage concept have been proposed and experimentally demonstrated. In addition, to cope with hardware, software and networking failures, technqiues such as RAID, software replication, and disconnected operation emerged. The quest for robustness continues to rise in importance as the cost of failures increases relative to memory and storage costs. In closing, Satya commented on recent predictions that the classic hierarchical file system will soon be extinct. He observed that such predictions are not new. Classic file systems may be overlaid by non-hierarchical interfaces that uses different abstractions (such as the Android interface for Java applications). However, they will continue to be important for unstructured data that must be preserved for very long periods of time. Satya observed that the deep reasons for the longevity of the hierarchical file system model were articulated in broad terms by Herb Simon in his 1962 work, "The Architecture of Complexity". Essentially, hierarchy arises due to the cognitive limitations of the human mind. File system implementations have evolved to be a good fit for these cognitive limitations. They are likely to be with us for a very long time.