Using available remote memory dynamically for parallel data mining application on ATM-connected PC cluster

Personal computer/Workstation (PC/WS) clusters are promising candidates for future high performance computers, because of their good scalability and cost performance ratio. Data intensive applications, such as data mining and ad hoc query processing in databases, are considered very important for massively parallel processors, as well as conventional scientific calculations. Thus, investigating the feasibility of data intensive applications on a PC cluster is meaningful. Association rule mining, one of the best-known problems in data mining, differs from conventional scientific calculations in its usage of main memory. It allocates many small data areas in main memory, and the number of those areas suddenly grows enormously during execution. As a result, the contents of memory must be swapped out if the requirement for memory space exceeds the real memory size. However, because the size of each data area is rather small and the elements are accessed almost at random, swapping out to a storage device must degrade the performance severely. In this paper, we investigate the feasibility of using available remote nodes' memory as a swap area when application execution nodes need to swap out their real memory contents during the execution of parallel data mining on PC clusters. We report our experiments in which application execution nodes acquire extra memory dynamically from several available remote nodes through an ATM network. A method of remote memory utilization with remote update operations is proposed and evaluated. The experimental results on our PC cluster show that the proposed method is expected to be considerably better than using hard disks as a swapping device. The dynamic decision mechanism for remote memory availability and the migration operations are also evaluated.